PANIMALAR ENGINEERING COLLEGE

DEPARTMENT OF CSE-ME

23CS2902- DEEP LEARNING - UNIT 1

Syllabus: Basic concept of neurons – Perceptron algorithm – Feed forward and Back
propagation networks.

Deep learning is based on the branch of machine learning, which is a subset of artificial
intelligence. Since neural networks imitate the human brain and so deep learning will do. In
deep learning, nothing is programmed explicitly. Basically, it is a machine learning class that
makes use of numerous nonlinear processing units so as to perform feature extraction as well
as transformation. The output from each preceding layer is taken as input by each one of the
successive layers.
Deep learning is implemented with the help of Neural Networks, and the idea behind the
motivation of Neural Network is the biological neurons, which is nothing but a brain cell.
Deep learning is a collection of statistical techniques of machine learning for learning feature
hierarchies that are actually based on artificial neural networks.
So basically, deep learning is implemented by the help of deep networks, which are nothing
but neural networks with multiple hidden layers.
Example of Deep Learning

In the example given above, we provide the raw data of images to the first layer of the input
layer. After then, these input layer will determine the patterns of local contrast that means it
will differentiate on the basis of colors, luminosity, etc. Then the 1st hidden layer will
determine the face feature, i.e., it will fixate on eyes, nose, and lips, etc. And then, it will
fixate those face features on the correct face template. So, in the 2nd hidden layer, it will
actually determine the correct face here as it can be seen in the above image, after which it
will be sent to the output layer. Likewise, more hidden layers can be added to solve more
complex problems, for example, if you want to find out a particular kind of face having large
or light complexions. So, as and when the hidden layers increase, we are able to solve
complex problems.

1
 PANIMALAR ENGINEERING COLLEGE
DEPARTMENT OF CSE-ME

ARCHITECTURES

Deep Neural Networks

It is a neural network that incorporates the complexity of a certain level, which means
several numbers of hidden layers are encompassed in between the input and output layers.
They are highly proficient on model and process non-linear associations.

Deep Belief Networks

A deep belief network is a class of Deep Neural Network that comprises of multi-layer belief
networks.

Steps to perform DBN:

With the help of the Contrastive Divergence algorithm, a layer of features is learned from
perceptible units.
Next, the formerly trained features are treated as visible units, which perform learning of
features.
Lastly, when the learning of the final hidden layer is accomplished, then the whole DBN is
trained.

Recurrent Neural Networks

It permits parallel as well as sequential computation, and it is exactly similar to that of the
human brain (large feedback network of connected neurons). Since they are capable enough
to reminisce all of the imperative things related to the input they have received, so they are
more precise.

TYPES OF DEEP LEARNING NETWORKS

1. Feed Forward Neural Network

A feed-forward neural network is none other than an Artificial Neural Network,
which ensures that the nodes do not form a cycle. In this kind of neural network, all the
perceptrons are organized within layers, such that the input layer takes the input, and the
output layer generates the output. Since the hidden layers do not link with the outside world,
it is named as hidden layers. Each of the perceptrons contained in one single layer is
associated with each node in the subsequent layer. It can be concluded that all of the nodes
are fully connected. It does not contain any visible or invisible connection between the nodes
in the same layer. There are no back-loops in the feed-forward network. To minimize the
prediction error, the backpropagation algorithm can be used to update the weight values.

Applications:
Data Compression, Pattern Recognition, Computer Vision, Sonar Target Recognition, Speech
Recognition, Handwritten Characters Recognition

2
 PANIMALAR ENGINEERING COLLEGE
DEPARTMENT OF CSE-ME

2. Recurrent Neural Network

They are yet another variation of feed-forward networks. Here each of the neurons present in
the hidden layers receives an input with a specific delay in time. The Recurrent neural
network mainly accesses the preceding info of existing iterations. For example, to guess the
succeeding word in any sentence, one must have knowledge about the words that were
previously used. It not only processes the inputs but also shares the length as well as weights
crossways time. It does not let the size of the model to increase with the increase in the input
size. However, the only problem with this recurrent neural network is that it has slow
computational speed as well as it does not contemplate any future input for the current state.
It has a problem with reminiscing prior information.
Applications:
Machine Translation, Robot Control, Time Series Prediction, Speech Recognition, Speech
Synthesis, Time Series Anomaly Detection, Rhythm Learning, Music Composition

3. Convolutional Neural Network

They are a special kind of neural network mainly used for image classification, clustering of
images and object recognition. DNNs enable unsupervised construction of hierarchical
image representations. To achieve the best accuracy, deep convolutional neural networks are
preferred more than any other neural network.
Applications:
Identify Faces, Street Signs, Tumors, Image Recognition, Video Analysis, NLP, Anomaly
Detection, Drug Discovery, Checkers Game, Time Series Forecasting.

4. Restricted Boltzmann Machine - RBMs

They are yet another variant of Boltzmann Machines. Here the neurons present in the input
layer and the hidden layer encompasses symmetric connections amid them. However, there is
no internal association within the respective layer. But in contrast to RBM, Boltzmann
machines do encompass internal connections inside the hidden layer. These restrictions in
BMs helps the model to train efficiently.
Applications:
Filtering, Feature Learning, Classification, Risk Detection, Business and Economic analysis.

5. Autoencoders
An autoencoder neural network is another kind of unsupervised machine learning algorithm.
Here the number of hidden cells is merely small than that of the input cells. But the number
of input cells is equivalent to the number of output cells. An autoencoder network is trained
to display the output similar to the fed input to force AEs to find common patterns and
generalize the data. The autoencoders are mainly used for the smaller representation of the
input. It helps in the reconstruction of the original data from compressed data. This algorithm
is comparatively simple as it only necessitates the output identical to the input.
Encoder: Convert input data in lower dimensions.
Decoder: Reconstruct the compressed data.
Applications:
Classification, Clustering, Feature Compression, Deep learning applications

3
 PANIMALAR ENGINEERING COLLEGE
DEPARTMENT OF CSE-ME

Self-Driving Cars
In self-driven cars, it is able to capture the images around it by processing a huge amount of
data, and then it will decide which actions should be incorporated to take a left or right or
should it stop. So, accordingly, it will decide what actions it should take, which will further
reduce the accidents that happen every year.

Voice Controlled Assistance

When we talk about voice control assistance, then Siri is the one thing that comes into our
mind. So, you can tell Siri whatever you want it to do it for you, and it will search it for you
and display it for you.

Automatic Image Caption Generation

Whatever image that you upload, the algorithm will work in such a way that it will generate
caption accordingly. If you say blue colored eye, it will display a blue-colored eye with a
caption at the bottom of the image.

Automatic Machine Translation

With the help of automatic machine translation, we are able to convert one language into
another with the help of deep learning.

Limitations
It only learns through the observations.
It comprises of biases issues.

Advantages
It lessens the need for feature engineering.
It eradicates all those costs that are
needless. It easily identifies difficult
defects.
It results in the best-in-class performance on problems.

Disadvantages
It requires an ample amount of data.
It is quite expensive to train.
It does not have strong theoretical groundwork.

What is a Neuron in Deep Learning?

Neurons in deep learning models are nodes through which data and computations flow.

Neurons work like this:

 They receive one or more input signals. These input signals can come from either the raw
data set or from neurons positioned at a previous layer of the neural net.
 They perform some calculations.
4
 PANIMALAR ENGINEERING COLLEGE
DEPARTMENT OF CSE-ME

 They send some output signals to neurons deeper in the neural net through a synapse.
Here is a diagram of the functionality of a neuron in a deep learning neural net:

Let’s walk through this diagram step-by-step.

As you can see, neurons in a deep learning model are capable of having synapses that
connect to more than one neuron in the preceding layer. Each synapse has an associated
weight, which impacts the preceding neuron’s importance in the overall neural network.
Weights are a very important topic in the field of deep learning because adjusting a model’s
weights is the primary way through which deep learning models are trained. You’ll see this in
practice later on when we build our first neural networks from scratch.

Once a neuron receives its inputs from the neurons in the preceding layer of the model, it
adds up each signal multiplied by its corresponding weight and passes them on to an
activation function, like this:

The activation function calculates the output value for the neuron. This output value is then
passed on to the next layer of the neural network through another synapse.

5
 PANIMALAR ENGINEERING COLLEGE
DEPARTMENT OF CSE-ME

This serves as a broad overview of deep learning neurons. Do not worry if it was a lot to take
in – we’ll learn much more about neurons in the rest of this tutorial. For now, it’s sufficient
for you to have a high-level understanding of how they are structured in a deep learning
model.

Deep Learning Activation Functions

Activation functions are a core concept to understand in deep learning.
They are what allows neurons in a neural network to communicate with each other through
their synapses.

In this section, you will learn to understand the importance and functionality of activation
functions in deep learning.

What is an Activation Function?

An activation function is a non-linear function applied by a neuron to introduce non-linear

properties in the network.

A relationship is linear if a change in the first variable corresponds to a constant change in the
second variable. A non-linear relationship means that a change in the first variable doesn’t
necessarily correspond with a constant change in the second. However, they may impact each
other but it appears to be unpredictable.

A quick visual example, by introducing non-linearity we can better capture the patterns in
this data

Best fit linear and non-linear models:

Linear Activation Function

6
 PANIMALAR ENGINEERING COLLEGE
DEPARTMENT OF CSE-ME

 A straight line function: a is a constant value.

 Values can get very large.
 The linear function alone doesn’t capture complex patterns.

Sigmoid Activation Function

 A non-linear function so can capture more complex patterns.

 Output values are bounded so don’t get too large.
 Can suffer from “vanishing gradient”.

Hyperbolic Tangent Activation Function

 A non-linear function so can capture more complex patterns.

 Output values are bounded so don’t get too large.
 Can suffer from “vanishing gradient”.

Rectified Linear Unit (ReLU) Activation Function

 A non-linear function so can capture more complex patterns.

 Values can get very large.
 As it does not allow for negative values, certain patterns may not be captured.
 Gradient can go towards 0 so weights are not updated: “dying ReLU problem”.

7
 PANIMALAR ENGINEERING COLLEGE
DEPARTMENT OF CSE-ME

Leaky ReLU Activation Function

 A non-linear function so can capture more complex patterns.

 Attempts to solve the “dying ReLU problem”.
 Values can get very large.
 Alternatively, instead of using 0.01, that can also be a parameter, α, which is then
learned during training alongside the weights.
 This is referred to as Parametric ReLU (PReLU):

Softmax Activation Function

 Each value ranges between 0 and 1 and the sum of all values is 1 so can be used to
model probability distributions.
 Only used in the output layer rather than throughout the network.

Perceptron in Machine Learning

In Machine Learning and Artificial Intelligence, Perceptron is the most commonly used term for all
folks. It is the primary step to learn Machine Learning and Deep Learning technologies, which
consists of a set of weights, input values or scores, and a threshold. Perceptron is a building block of
an Artificial Neural Network. Initially, in the mid of 19th century, Mr. Frank Rosenblatt invented
the Perceptron for performing certain calculations to detect input data capabilities or business
intelligence. Perceptron is a linear Machine Learning algorithm used for supervised learning for

8
 PANIMALAR ENGINEERING COLLEGE
DEPARTMENT OF CSE-ME

various binary classifiers. This algorithm enables neurons to learn elements and processes them one
by one during preparation. In this tutorial, "Perceptron in Machine Learning," we will discuss in-depth
knowledge of Perceptron and its basic functions in brief. Let's start with the basic introduction of
Perceptron.

What is the Perceptron model in Machine Learning?

Perceptron is Machine Learning algorithm for supervised learning of various binary classification
tasks. Further, Perceptron is also understood as an Artificial Neuron or neural network unit that
helps to detect certain input data computations in business intelligence.

Perceptron model is also treated as one of the best and simplest types of Artificial Neural networks.
However, it is a supervised learning algorithm of binary classifiers. Hence, we can consider it as a
single-layer neural network with four main parameters, i.e., input values, weights and Bias, net sum,
and an activation function.

What is Binary classifier in Machine Learning?

In Machine Learning, binary classifiers are defined as the function that helps in deciding whether
input data can be represented as vectors of numbers and belongs to some specific class.

Binary classifiers can be considered as linear classifiers. In simple words, we can understand it as
a classification algorithm that can predict linear predictor function in terms of weight and feature
vectors.

Basic Components of Perceptron

Mr. Frank Rosenblatt invented the perceptron model as a binary classifier which contains three main
components. These are as follows:

o Input Nodes or Input Layer:

This is the primary component of Perceptron which accepts the initial data into the system for further
processing. Each input node contains a real numerical value.

9
 PANIMALAR ENGINEERING COLLEGE
DEPARTMENT OF CSE-ME

o Wight and Bias:

Weight parameter represents the strength of the connection between units. This is another most
important parameter of Perceptron components. Weight is directly proportional to the strength of the
associated input neuron in deciding the output. Further, Bias can be considered as the line of intercept
in a linear equation.

o Activation Function:

These are the final and important components that help to determine whether the neuron will fire or
not. Activation Function can be considered primarily as a step function.

Types of Activation functions:

o Sign function
o Step function, and
o Sigmoid function

The data scientist uses the activation function to take a subjective decision based on various problem
statements and forms the desired outputs. Activation function may differ (e.g., Sign, Step, and
Sigmoid) in perceptron models by checking whether the learning process is slow or has vanishing or
exploding gradients.

How does Perceptron work?

In Machine Learning, Perceptron is considered as a single-layer neural network that consists of four
main parameters named input values (Input nodes), weights and Bias, net sum, and an activation
function. The perceptron model begins with the multiplication of all input values and their weights,
then adds these values together to create the weighted sum. Then this weighted sum is applied to the
activation function 'f' to obtain the desired output. This activation function is also known as the step
function and is represented by 'f'.

10
 PANIMALAR ENGINEERING COLLEGE
DEPARTMENT OF CSE-ME

This step function or Activation function plays a vital role in ensuring that output is mapped between
required values (0,1) or (-1,1). It is important to note that the weight of input is indicative of the
strength of a node. Similarly, an input's bias value gives the ability to shift the activation function
curve up or down.

Perceptron model works in two important steps as follows:

Step-1

In the first step first, multiply all input values with corresponding weight values and then add them to
determine the weighted sum. Mathematically, we can calculate the weighted sum as follows:

∑wi*xi = x1*w1 + x2*w2 +…wn*xn

Add a special term called bias 'b' to this weighted sum to improve the model's performance.

∑wi*xi + b

Step-2

In the second step, an activation function is applied with the above-mentioned weighted sum, which
gives us output either in binary form or a continuous value as follows:

Y = f(∑wi*xi + b)

11
 PANIMALAR ENGINEERING COLLEGE
DEPARTMENT OF CSE-ME

Types of Perceptron Models

Based on the layers, Perceptron models are divided into two types. These are as follows:

1. Single-layer Perceptron Model

2. Multi-layer Perceptron model

Single Layer Perceptron Model:

This is one of the easiest Artificial neural networks (ANN) types. A single-layered perceptron model
consists feed-forward network and also includes a threshold transfer function inside the model. The
main objective of the single-layer perceptron model is to analyze the linearly separable objects with
binary outcomes.

In a single layer perceptron model, its algorithms do not contain recorded data, so it begins with
inconstantly allocated input for weight parameters. Further, it sums up all inputs (weight). After
adding all inputs, if the total sum of all inputs is more than a pre-determined value, the model gets
activated and shows the output value as +1.

If the outcome is same as pre-determined or threshold value, then the performance of this model is
stated as satisfied, and weight demand does not change. However, this model consists of a few
discrepancies triggered when multiple weight inputs values are fed into the model. Hence, to find
desired output and minimize errors, some changes should be necessary for the weights input.

12
 PANIMALAR ENGINEERING COLLEGE
DEPARTMENT OF CSE-ME

"Single-layer perceptron can learn only linearly separable patterns."

Multi-Layered Perceptron Model:

Like a single-layer perceptron model, a multi-layer perceptron model also has the same model
structure but has a greater number of hidden layers.

The multi-layer perceptron model is also known as the Backpropagation algorithm, which executes in
two stages as follows:

o Forward Stage: Activation functions start from the input layer in the forward stage and
terminate on the output layer.
o Backward Stage: In the backward stage, weight and bias values are modified as per the
model's requirement. In this stage, the error between actual output and demanded originated
backward on the output layer and ended on the input layer.

Hence, a multi-layered perceptron model has considered as multiple artificial neural networks having
various layers in which activation function does not remain linear, similar to a single layer perceptron
model. Instead of linear, activation function can be executed as sigmoid, TanH, ReLU, etc., for
deployment.

A multi-layer perceptron model has greater processing power and can process linear and non-linear
patterns. Further, it can also implement logic gates such as AND, OR, XOR, NAND, NOT, XNOR,
NOR.

Advantages of Multi-Layer Perceptron:

o A multi-layered perceptron model can be used to solve complex non-linear problems.

o It works well with both small and large input data.
o It helps us to obtain quick predictions after the training.
o It helps to obtain the same accuracy ratio with large as well as small data.

Disadvantages of Multi-Layer Perceptron:

o In Multi-layer perceptron, computations are difficult and time-consuming.

o In multi-layer Perceptron, it is difficult to predict how much the dependent variable affects
each independent variable.

o The model functioning depends on the quality of the training.

Perceptron Function
Perceptron function ''f(x)'' can be achieved as output by multiplying the input 'x' with the learned
weight coefficient 'w'.

Mathematically, we can express it as follows:

13
 PANIMALAR ENGINEERING COLLEGE
DEPARTMENT OF CSE-ME

f(x)=1; if w.x+b>0

otherwise, f(x)=0

o 'w' represents real-valued weights vector

o 'b' represents the bias
o 'x' represents a vector of input x values.

Characteristics of Perceptron
The perceptron model has the following characteristics.

1. Perceptron is a machine learning algorithm for supervised learning of binary classifiers.

2. In Perceptron, the weight coefficient is automatically learned.
3. Initially, weights are multiplied with input features, and the decision is made whether the
neuron is fired or not.
4. The activation function applies a step rule to check whether the weight function is greater
than zero.
5. The linear decision boundary is drawn, enabling the distinction between the two linearly
separable classes +1 and -1.
6. If the added sum of all input values is more than the threshold value, it must have an output
signal; otherwise, no output will be shown.

Limitations of Perceptron Model

A perceptron model has limitations as follows:

o The output of a perceptron can only be a binary number (0 or 1) due to the hard limit transfer
function.
o Perceptron can only be used to classify the linearly separable sets of input vectors. If input
vectors are non-linear, it is not easy to classify them properly.

14