21PG2IT7 - DATA SCEINCE

Single Layer Perceptron

A perceptron is a neural network unit that does a precise computation to detect
features in the input data. Perceptron is mainly used to classify the data into two
parts. Therefore, it is also known as Linear Binary Classifier.

A regular neural network looks like this:

 21PG2IT7 - DATA SCEINCE

The perceptron consists of 4 parts.

 Input value or One input layer: The input layer of the perceptron is made
of artificial input neurons and takes the initial data into the system for
further processing.
 Weights and Bias:
Weight: It represents the dimension or strength of the connection between
units. If the weight to node 1 to node 2 has a higher quantity, then neuron
1 has a more considerable influence on the neuron.
Bias: It is the same as the intercept added in a linear equation. It is an
additional parameter which task is to modify the output along with the
weighted sum of the input to the other neuron.
 Net sum: It calculates the total sum.
 Activation Function: A neuron can be activated or not, is determined by an
activation function. The activation function calculates a weighted sum and
further adding bias with it to give the result.
 21PG2IT7 - DATA SCEINCE

A standard neural network looks like the below diagram.

How does it work?

The perceptron works on these simple steps which are given below:

a. In the first step, all the inputs x are multiplied with their weights w.
 21PG2IT7 - DATA SCEINCE

b. In this step, add all the increased values and call them the Weighted sum.

c. In our last step, apply the weighted sum to a correct Activation Function.
 21PG2IT7 - DATA SCEINCE

 In a single layer perceptron, the weights to each input node are assigned
randomly since there is no a priori knowledge associated with the nodes.
 Also, a threshold value is assigned randomly
 Now SLP sums all the weights which are inputted and if the sums are is
above the threshold then the network is activated.
 If the calculated value is matched with the desired value, then the model is
successful
 If it is not, then since there is no back-propagation technique involved in
this the error needs to be calculated using the below formula and the
weights need to be adjusted again.

The single layer perceptron (SLP) model is the simplest form of neural network
and the basis for the more advanced models that have been developed in deep
learning. Typically, we use SLP in classification problems where we need to give
the data observations labels (binary or multinomial) based on inputs. The values
in the input layer are directly sent to the output layer after they are multiplied by
weights and a bias is added to the cumulative sum. This cumulative sum is then
put into an activation function, which is simply a function that defines the output.
When that output is above or below a user-determined threshold, the final
output is determined.
 21PG2IT7 - DATA SCEINCE

Multilayer Perceptron Model

In a Multilayer Perceptron, the main intuition of using this method is when the
data is not linearly separable. Each node in a layer consists of a non-linear
activation function for processing. These functions are typically Sigmoid/Logistic
Function, tanh/Hyperbolic Tangent function, ReLU (Rectified Linear Unit),
Softmax. This neural network is fully connected and also has the capability to
learn by itself by changing the weights of connection after each data point is
processed and the amount of error it generates. Multi-Layer perceptron defines
the most complex architecture of artificial neural networks. It is substantially
formed from multiple layers of the perceptron. The pictorial representation of
multi-layer perceptron learning is as shown below-
 21PG2IT7 - DATA SCEINCE

MLP networks are used for supervised learning format. A typical learning
algorithm for MLP networks is also called back propagation's algorithm.

A multilayer perceptron (MLP) is a feed forward artificial neural network that

generates a set of outputs from a set of inputs. An MLP is characterized by several
layers of input nodes connected as a directed graph between the input nodes
connected as a directed graph between the input and output layers. MLP uses
backpropagation for training the network. MLP is a deep learning method.

Very similar to SLP, the multilayer perceptron (MLP) model features multiple
layers that are interconnected in such a way that they form a feed-forward neural
network. Each neuron in one layer has directed connections to the neurons of a
separate layer. One of the key distinguishing factors in this model and the single
layer perceptron model is the back-propagation algorithm, a common method of
training neural networks. Back-propagation passes the error calculated from the
output layer to the input layer such that we can see each layer’s contribution to
the error and alter the network accordingly. Here, we use a gradient descent
algorithm to determine the degree to which the weights should change upon
each iteration. Gradient descent—another popular machine learning/optimization
algorithm—is simply the derivative of a function such that we find a scalar (a
number with magnitude as its only property) value that points in the direction of
greatest momentum. By subtracting the gradient, this leads us to a solution that is
more optimal than the one we currently are at until we reach a global optimum
 21PG2IT7 - DATA SCEINCE

Convolutional Neural Network

Convolutional Neural Network, this type of neural network is an advanced version
of Multilayer Perceptron. In this type, there is one or more than one
convolutional layer. Now the basic question is what exactly is a convolutional
layer? Convolution is nothing but a simple filtering mechanism that enables
activation. When this filtering mechanism is repeated, it yields the location and
strength of a detected feature. As a result of this ability, these networks are
widely used in image processing, natural language processing, recommender
systems so as to yield effective results of the important feature detected.
 21PG2IT7 - DATA SCEINCE

A Convolutional Neural Network (ConvNet/CNN) is a Deep Learning algorithm

which can take in an input image, assign importance (learnable weights and
biases) to various aspects/objects in the image and be able to differentiate one
from the other. The pre-processing required in a ConvNet is much lower as
compared to other classification algorithms. While in primitive methods filters are
hand-engineered, with enough training, ConvNets have the ability to learn these
filters/characteristics.
 21PG2IT7 - DATA SCEINCE

How do convolutional neural networks work?

Convolutional neural networks are distinguished from other neural networks by

their superior performance with image, speech, or audio signal inputs. They have
three main types of layers, which are:

 Convolutional layer
 Pooling layer
 Fully-connected (FC) layer

The convolutional layer is the first layer of a convolutional network. While

convolutional layers can be followed by additional convolutional layers or pooling
layers, the fully-connected layer is the final layer. With each layer, the CNN
increases in its complexity, identifying greater portions of the image. Earlier layers
focus on simple features, such as colors and edges. As the image data progresses
through the layers of the CNN, it starts to recognize larger elements or shapes of
the object until it finally identifies the intended object.

Convolutional Layer

The convolutional layer is the core building block of a CNN, and it is where the
majority of computation occurs. It requires a few components, which are input
data, a filter, and a feature map.

Pooling Layer

Pooling layers, also known as downsampling, conducts dimensionality reduction,

reducing the number of parameters in the input. Similar to the convolutional
layer, the pooling operation sweeps a filter across the entire input, but the
difference is that this filter does not have any weights.

There are two main types of pooling:

 Max pooling: As the filter moves across the input, it selects the pixel with
the maximum value to send to the output array. As an aside, this approach
tends to be used more often compared to average pooling.
 Average pooling: As the filter moves across the input, it calculates the
average value within the receptive field to send to the output array.
 21PG2IT7 - DATA SCEINCE

Fully-Connected Layer

The name of the full-connected layer aptly describes itself. The pixel values of the
input image are not directly connected to the output layer in partially connected
layers. However, in the fully-connected layer, each node in the output layer
connects directly to a node in the previous layer.

This layer performs the task of classification based on the features extracted
through the previous layers and their different filters. While convolutional and
pooling layers tend to use ReLu functions, FC layers usually leverage a softmax
activation function to classify inputs appropriately, producing a probability from 0
to 1.

Recurrent Neural Networks

RNN works on the principle of saving the output of a particular layer and feeding
this back to the input in order to predict the output of the layer.

Below is how you can convert a Feed-Forward Neural Network into a Recurrent
Neural Network:

In a RNN output of the current layer is the input of the next layer. In traditional
neural networks, all the inputs and outputs are independent of each other, but in
cases like when it is required to predict the next word of a sentence, the previous
words are required and hence there is a need to remember the previous words.
 21PG2IT7 - DATA SCEINCE

It uses Long Short Term Memory (LSTM) to remember the previous output. In a
RNN same weights and bias are used for input of every layer. Because it performs
the same task on all the inputs of hidden layers to produce the output. The main
and most important feature of RNN is Hidden state, which remembers some
information about a sequence By LSTM. A RNN is the best use for sequential
model data like predicting the next word of a sentence or next position of a
running ball. The Recurrent Neural Network consists of multiple fixed activation
function units, one for each time step. Each unit has an internal state which is
called the hidden state of the unit. This hidden state signifies the past knowledge
that that the network currently holds at a given time step. This hidden state is
updated at every time step to signify the change in the knowledge of the network
about the past.

How Does Recurrent Neural Networks Work?

In Recurrent Neural networks, the information cycles through a loop to the

middle hidden layer. The input layer ‘x’ takes in the input to the neural network
and processes it and passes it onto the middle layer.

The middle layer ‘h’ can consist of multiple hidden layers, each with its own
activation functions and weights and biases. If you have a neural network where
the various parameters of different hidden layers are not affected by the previous
layer, ie: the neural network does not have memory, then you can use a recurrent
neural network.

The Recurrent Neural Network will standardize the different activation functions
and weights and biases so that each hidden layer has the same parameters. Then,
instead of creating multiple hidden layers, it will create one and loop over it as
many times as required.

Feed-Forward Neural Networks vs Recurrent Neural Networks

A feed-forward neural network allows information to flow only in the forward

direction, from the input nodes, through the hidden layers, and to the output
nodes. There are no cycles or loops in the network.
 21PG2IT7 - DATA SCEINCE

Applications of Recurrent Neural Networks

Image Captioning

RNNs are used to caption an image by analyzing the activities present.

Time Series Prediction

Any time series problem, like predicting the prices of stocks in a particular month,
can be solved using an RNN.

Natural Language Processing

Text mining and Sentiment analysis can be carried out using an RNN for Natural
Language Processing (NLP).

Types of Recurrent Neural Networks

There are four types of Recurrent Neural Networks:

1. One to One
2. One to Many
3. Many to One
4. Many to Many

One to One RNN

This type of neural network is known as the Vanilla Neural Network. It's used for
general machine learning problems, which has a single input and a single output.
 21PG2IT7 - DATA SCEINCE

One to Many RNN

This type of neural network has a single input and multiple outputs. An example
of this is the image caption.

Many to One RNN

This RNN takes a sequence of inputs and generates a single output. Sentiment
analysis is a good example of this kind of network where a given sentence can be
classified as expressing positive or negative sentiments.
 21PG2IT7 - DATA SCEINCE

Many to Many RNN

This RNN takes a sequence of inputs and generates a sequence of outputs.

Machine translation is one of the examples.

Recurrent neural networks (RNNs) are models of Artificial neural networks

(ANNs) where the connections between units form a directed cycle. Specifically, a
 21PG2IT7 - DATA SCEINCE

directed cycle is a sequence where the walk along the vertices and edges is
completely determined by the set of edges used and therefore has some
semblance of a specific order. RNNs are often specifically used for speech and
handwriting recognition