SOFT COMPUTING

Subject Code:PCP7H010
Credit:3-0-0
7th Semester
Branch:ETC
By
Dr. Sakuntala Mahapatra
Dean (R &D), Professor and HOD
Dept. of Electronics & Telecommunication Engg.
TRIDENT ACADEMY OF TECHNOLOGY
BHUBANESWAR, ODISHA

@ Dr. Sakuntala Mahapatra 9/12/2020

Contents

 Structure of ANN
 Learning in Neural Networks
 Single Layer Perceptron
 Activation Functions

@ Dr. Sakuntala Mahapatra 9/12/2020 Neural Networks 2

 PPT CONTENT
NEURAL NETWORKS
CLASS-27
(MODULE-1II)
DATE:9/12/2020
TIME:12.15 PM-1.15 PM

@ Dr. Sakuntala Mahapatra 9/12/2020 Neural Networks 3

Artificial Neural Networks(ANN)
 Artificial Neural Networks (ANNs), usually
simply called Neural Networks (NNs), are
computing systems vaguely inspired by
the Biological Neural Networks that
constitute Human Brain.
 An ANN is based on a collection of connected
units or nodes called Artificial Neurons,
which loosely model the Neurons in a biological
brain.

@ Dr. Sakuntala Mahapatra 9/12/2020 Neural Networks 4

Artificial Neural Networks(ANN)
 An Artificial Neural Network (ANN) is the piece
of a computing system designed to simulate the way
the human brain analyzes and processes
information.
 It is the foundation of Artificial Intelligence (AI)
and solves problems that would prove impossible or
difficult by human or statistical standards.
 ANNs have self-learning capabilities that enable them
to produce better results as more data becomes
available.
 Artificial Neural Networks are built like the human
brain, with neuron nodes interconnected like a web.

@ Dr. Sakuntala Mahapatra 9/12/2020 Neural Networks 5

Structure of Artificial Neural
Networks (ANN)
 Artificial Neural Networks is the
information processing system the
mechanism of which is inspired with the
functionality of biological neural circuits.
 An Artificial Neural Network possesses
many processing units connected to each
other. The schematic in the next slide
shows representation of Artificial Neural
Network.
@ Dr. Sakuntala Mahapatra 9/12/2020 Neural Networks 6
Structure of Artificial Neural
Networks (ANN)

@ Dr. Sakuntala Mahapatra 9/12/2020 Neural Networks 7

Structure of Artificial Neural
Networks (ANN)

@ Dr. Sakuntala Mahapatra 9/12/2020 Neural Networks 8

Components of Artificial
Neural Networks
 ANNs are comprised of three core layers or
phases – an Input layer, Hidden layer/s, and an
Output layer.
 Input Layer: The first layer is fed with the input,
that is, raw data. It conveys the information
from the outside world to the network. In this
layer, no computation is performed – the nodes
merely pass on the information to the hidden
layer.

@ Dr. Sakuntala Mahapatra 9/12/2020 Neural Networks 9

Components of Artificial
Neural Networks
 Hidden Layer: In this layer, the nodes lie hidden
behind the input layer – they comprise the
abstraction part in every neural network. All the
computations on the features entered through
the input layer occur in the hidden layer/s, and
then, it transfers the result to the output layer.
 Output Layer: This layer depicts the results of
the computations performed by the network to
the outer world.

@ Dr. Sakuntala Mahapatra 9/12/2020 Neural Networks 10

Structure of Artificial Neural
Networks (ANN)
 The diagram shows that the hidden units
communicate with the external layer. While the
input and output units communicate only through
the hidden layer of the network.
 The pattern of connection with nodes, the total
number of layers and level of nodes between
inputs and outputs with the number of neurons
per layer define the architecture of a neural
network.
 There are two types of architecture in ANN.
1. Single Layer Perceptron
2. Multi Layer Perceptron

@ Dr. Sakuntala Mahapatra 9/12/2020 Neural Networks 11

Structure of Artificial Neural
Networks (ANN)
 Mathematically, ANNs can be represented as
weighted directed graphs. The most common
ANN architectures are:
 Single-Layer Feed-Forward NNs: One input
layer and one output layer of processing units. No
feedback connections (e.g. a Perceptron)
 Multi-Layer Feed-Forward NNs: One input
layer, one output layer, and one or more hidden
layers of processing units. No feedback
connections (e.g. a Multi-Layer Perceptron).
 Recurrent NNs: Any network with at least one
feedback connection. It may, or may not, have
hidden units.
@ Dr. Sakuntala Mahapatra 9/12/2020 Neural Networks 12
Structure of Artificial Neural
Networks (ANN)

@ Dr. Sakuntala Mahapatra 9/12/2020 Neural Networks 13

Structure of ANN
Single Layer Neural Network

@ Dr. Sakuntala Mahapatra 9/12/2020 Neural Networks 14

Structure of ANN
Single Layer Neural Network

@ Dr. Sakuntala Mahapatra 9/12/2020 Neural Networks 15

Structure of ANN
Single Layer Neural Network

@ Dr. Sakuntala Mahapatra 9/12/2020 Neural Networks 16

Structure of ANN
Comparison between a Single Layer and Multi
Layer Neural Network

@ Dr. Sakuntala Mahapatra 9/12/2020 Neural Networks 17

Structure of ANN
Multi Layer Neural Network

@ Dr. Sakuntala Mahapatra 9/12/2020 Neural Networks 18

Structure of ANN
Multi Layer Neural Network

@ Dr. Sakuntala Mahapatra 9/12/2020 Neural Networks 19

Structure of ANN
Multi Layer Neural Network
(2:3:1 Neural Network)

@ Dr. Sakuntala Mahapatra 9/12/2020 Neural Networks 20

Structure of ANN
Multi Layer Neural Network
(4:3:1 Neural Network)

@ Dr. Sakuntala Mahapatra 9/12/2020 Neural Networks 21

Structure of ANN
Multi Layer Neural Network
(3:4:4:1 Neural Network)

@ Dr. Sakuntala Mahapatra 9/12/2020 Neural Networks 22

Structure of ANN
Multi Layer Neural Network

@ Dr. Sakuntala Mahapatra 9/12/2020 Neural Networks 23

 PPT CONTENT
NEURAL NETWORKS
CLASS-28
(MODULE-1II)
DATE:10/12/2020
TIME:12.15 PM-1.15 PM

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 24

Learning in Neural Networks
Learning
 An artificial Neural Network's Learning Rule or Learning
Process is a method, mathematical logic or algorithm which
improves the network's performance and/or training time.
 Usually, this rule is applied repeatedly over the network. It is
done by updating the weights and bias levels of a network
when a network is simulated in a specific data environment.
 A Learning Rule may accept existing conditions (weights and
biases) of the network and will compare the expected result
and actual result of the network to give new and improved
values for weights and bias.
 There are three types of Learning in Neural Networks.
1. Supervised Learning
2. Unsupervised Learning
3. Reinforcement Learning

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 25

Supervised Learning
 Supervised Learning as the name indicates the
presence of a supervisor as a teacher.
 Basically Supervised Learning is a learning in
which we teach or train the machine using data
which is well labeled that means some data is
already tagged with the correct answer.
 After that, the machine is provided with a new set
of examples(data) so that supervised learning
algorithm analyses the training data(set of training
examples) and produces a correct outcome from
labeled data.

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 26

Supervised Learning
 A supervised learning algorithm learns from
labeled training data, helps us to predict
outcomes for unforeseen data.
 Successfully building, scaling, and deploying
accurate supervised machine learning data
science model takes time and technical
expertise from a team of highly skilled data
scientists.

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 27

Supervised Learning
 Supervised learning allows us to collect data or
produce a data output from the previous
experience.
 It helps to optimize performance criteria using
experience.
 Supervised machine learning helps to solve
various types of real-world computation
problems.

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 28

Types of Supervised Machine Learning
Techniques
 Supervised learning classified into two categories
of algorithms.
1. Classification
 Classification means to group the output inside a
class. If the algorithm tries to label input into two
distinct classes, it is called binary classification.
Selecting between more than two classes is
referred to as multiclass classification.
 A classification problem is when the output
variable is a category, such as “Red” or “Blue” or
“disease” and “no disease”.
 Example: Determining whether or not someone
will be a defaulter of the loan.
@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 29
Types of Supervised Machine Learning
Techniques
2. Regression
 Regression technique predicts a single output
value using training data.
 A regression problem is when the output
variable is a real value, such as “dollars” or
“weight”.
 Example: Regression can be used to predict
the house price from training data. The input
variables will be locality, size of a house, etc.

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 30

Types of Supervised Machine Learning
Techniques

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 31

Types of Supervised Machine Learning
Techniques

 Regression
 Logistic Regression
 Classification
 Naïve Bayes Classifiers
 Decision Trees
 Support Vector Machine

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 32

Advantages of Supervised Learning

 Supervised Learning allows collecting data and

produce data output from the previous
experiences.
 It helps to optimize performance criteria with the
help of experience.
 Supervised machine learning helps to solve
various types of real-world computation
problems.
 Outputs always have a probabilistic interpretation,
and the algorithm can be regularized to avoid
overfitting.

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 33

Disadvantages of Supervised Learning

 Classifying big data can be challenging.

 Training for Supervised Learning needs a lot
of computation time. So, it requires a lot of
time.
 Logistic regression may underperform when
there are multiple or non-linear decision
boundaries.

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 34

Unsupervised Learning
 Unsupervised Learning is a machine learning
technique, where we do not need to supervise
the model. Instead, we need to allow the model
to work on its own to discover information. It
mainly deals with the unlabelled data.
 Unsupervised Learning is the training of machine
using information that is neither classified nor
labeled and allowing the algorithm to act on that
information without guidance.
 Here the task of machine is to group unsorted
information according to similarities, patterns and
differences without any prior training of data.

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 35

Unsupervised Learning
 Unlike Supervised Learning, no teacher is
provided that means no training will be given to
the machine.
 Therefore machine is restricted to find the
hidden structure in unlabeled data by itself.
 For instance, suppose it is given an image having
both dogs and cats which have not seen ever.

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 36

Advantages of Unsupervised Learning
 Unsupervised machine Learning finds all kind of
unknown patterns in data.
 Unsupervised methods help us to find features
which can be useful for categorization.
 It is taken place in real time, so all the input data
to be analyzed and labeled in the presence of
learners.
 It is easier to get unlabeled data from a
computer than labeled data, which needs
manual intervention.

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 37

Types of Unsupervised Machine
Learning Techniques
 Unsupervised Learning classified into two categories
of algorithms.
1. Clustering
 Clustering is an important concept when it comes to
unsupervised learning.
 It mainly deals with finding a structure or pattern in a
collection of uncategorized data.
 Clustering algorithms will process the data and find
natural clusters(groups) if they exist in the data.
 We can also modify how many clusters the
algorithms should identify.
 It allows to adjust the granularity of these groups.

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 38

Types of Unsupervised Machine
Learning Techniques

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 39

Types of Unsupervised Machine
Learning Techniques
2. Association
 Association rules allow to establish associations
amongst data objects inside large databases. This
unsupervised technique is about discovering
exciting relationships between variables in large
databases.
 An association rule learning problem is where
we want to discover rules that describe large
portions of data, such as people that buy X also
tend to buy Y.

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 40

Types of Unsupervised Machine
Learning Techniques
Examples
 For example, people that buy a new home most
likely to buy new furniture.
 A subgroup of cancer patients grouped by their
gene expression measurements
 Groups of shopper based on their browsing and
purchasing histories
 Movie group by the rating given by movies
viewers.

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 41

Types of Unsupervised Machine
Learning Techniques
 Clustering
 Exclusive (partitioning)
 Agglomerative
 Overlapping
 Probabilistic
 Clustering Types
 Hierarchical clustering
 K-means clustering
 K-NN (k nearest neighbors)
 Principal Component Analysis
 Singular Value Decomposition
 Independent Component Analysis

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 42

Supervised vs. Unsupervised Learning

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 43

Supervised vs. Unsupervised Learning

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 44

Summary
 In Supervised learning, you train the machine using data
which is well "labeled."
 Unsupervised learning is a machine learning technique,
where you do not need to supervise the model.
 Supervised learning allows you to collect data or
produce a data output from the previous experience.
 Unsupervised machine learning helps you to finds all
kind of unknown patterns in data.
 For example, you will able to determine the time taken
to reach back come base on weather condition, Times
of the day and holiday.
 For example, Baby can identify other dogs based on past
supervised learning.

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 45

Reinforcement Learning (RL)
 Reinforcement Learning is a feedback-based Machine
learning technique in which an agent learns to behave in an
environment by performing the actions and seeing the
results of actions. For each good action, the agent gets
positive feedback, and for each bad action, the agent gets
negative feedback or penalty.
 In Reinforcement Learning, the agent learns automatically
using feedbacks without any labeled data, unlike supervised
learning.
 Since there is no labeled data, so the agent is bound to
learn by its experience only.
 RL solves a specific type of problem where decision
making is sequential, and the goal is long-term, such
as game-playing, robotics, etc.

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 46

Reinforcement Learning (RL)
 The agent interacts with the environment and explores it by
itself. The primary goal of an agent in reinforcement learning
is to improve the performance by getting the maximum
positive rewards.
 The agent learns with the process of hit and trial, and based
on the experience, it learns to perform the task in a better
way. Hence, we can say that "Reinforcement learning is a
type of machine learning method where an
intelligent agent (computer program) interacts with
the environment and learns to act within that." How
a Robotic dog learns the movement of his arms is an
example of Reinforcement learning.
 It is a core part of Artificial intelligence, and all AI
agent works on the concept of reinforcement learning. Here
we do not need to pre-program the agent, as it learns from
its own experience without any human intervention.

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 47

Reinforcement Learning (RL)
 Example: Suppose there is an AI agent present within
a maze environment, and his goal is to find the diamond.
The agent interacts with the environment by
performing some actions, and based on those actions,
the state of the agent gets changed, and it also receives
a reward or penalty as feedback.
 The agent continues doing these three things (take
action, change state/remain in the same state,
and get feedback), and by doing these actions, he
learns and explores the environment.
 The agent learns that what actions lead to positive
feedback or rewards and what actions lead to negative
feedback penalty. As a positive reward, the agent gets a
positive point, and as a penalty, it gets a negative point.
@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 48
Reinforcement Learning (RL)

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 49

Reinforcement Learning (RL)
 Reinforcement learning enables the learning
of optimal behavior in tasks that require the
selection of sequential actions.
 This method of learning is based on
interactions between an agent and its
environment.
 Through repeated interactions with the
environment, and the receipt of rewards, the
agent learns which actions are associated
with the greatest cumulative reward.

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 50

Reinforcement Learning (RL)
 Reinforcement Learning is an area of Machine Learning. It is
about taking suitable action to maximize reward in a
particular situation.
 It is employed by various software and machines to find the
best possible behavior or path it should take in a specific
situation.
 Reinforcement learning differs from the supervised learning
in a way that in supervised learning the training data has the
answer key with it so the model is trained with the correct
answer itself.
 Whereas in Reinforcement Learning, there is no answer but
the reinforcement agent decides what to do to perform the
given task. In the absence of a training dataset, it is bound to
learn from its experience.

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 51

Reinforcement Learning (RL)
 Reinforcement Learning (RL) is a technique useful in
solving control optimization problems.
 By control optimization, we mean the problem of
recognizing the best action in every state visited by
the system so as to optimize some objective function,
e.g., the average reward per unit time and the total
discounted reward over a given time horizon.
 Typically, RL is used when the system has a very large
number of states (>> 1000) and has complex
stochastic structure, which is not amenable to closed
form analysis.
 When problems have a relative small number of
states and the underlying random structure is
relatively simple, one can use dynamic programming

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 52

Reinforcement Learning (RL)
Main points in Reinforcement Learning
 Input: The input should be an initial state from
which the model will start.
 Output: There are many possible output as there
are variety of solution to a particular problem.
 Training: The training is based upon the input, the
model will return a state and the user will decide
to reward or punish the model based on its
output.
 The model keeps continues to learn.
 The best solution is decided based on the
maximum reward.

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 53

Reinforcement Learning (RL)
Types of Reinforcement: There are two types of
Reinforcement Learning.

Positive
Positive Reinforcement is defined as when an event,
occurs due to a particular behavior, increases the
strength and the frequency of the behavior. In other
words, it has a positive effect on behavior.
 Advantages of reinforcement learning are:
◦ Maximizes Performance
◦ Sustain Change for a long period of time
 Disadvantages of reinforcement learning:
◦ Too much Reinforcement can lead to overload of
states which can diminish the results

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 54

Reinforcement Learning (RL)
Negative
Negative Reinforcement is defined as
strengthening of a behavior because a
negative condition is stopped or avoided.
 Advantages of reinforcement learning:
◦ Increases Behavior
◦ Provide defiance to minimum standard of
performance
 Disadvantages of reinforcement learning:
◦ It Only provides enough to meet up the
minimum behavior

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 55

Difference between Reinforcement
Learning and Supervised Learning

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 56

McCulloch-Pitts Neuron
 This vastly simplified model of real neurons is also
known as a Threshold Logic Unit.
1. A set of synapses (i.e. connections) brings in activations
from other neurons
2. A processing unit sums the inputs, and then applies a
non-linear activation function
3. An output line transmits the result to other neurons.

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 57

Networks of McCulloch-Pitts
Neurons
 One neuron can’t do much on its own. Usually we will
have many neurons labelled by indices k, i, j and
activation flows between via synapses with strengths
wki, wij.

@ Dr. Sakuntala Mahapatra 10/12/2020 Neural Networks 58

 PPT CONTENT
NEURAL NETWORKS
CLASS-29
(MODULE-1II)
DATE:14/12/2020
TIME:12.15 PM-1.15 PM

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 59

Single Layer Perceptron (SLP)
 The perceptron is a single processing unit of any
neural network. Frank Rosenblatt first
proposed in 1958 is a simple neuron which is
used to classify its input into one or two
categories. Perceptron is a linear classifier, and is
used in supervised learning. It helps to organize
the given input data.
 A perceptron is a neural network unit that does a
precise computation to detect features in the
input data. Perceptron is mainly used to classify
the data into two parts. Therefore, it is also
known as Linear Binary Classifier.

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 60

Single Layer Perceptron(SLP)
Activation Function of Perceptrons
 The activation function is used to map the input
between the required value like (0, 1) or (-1, 1).

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 61

Single Layer Perceptron(SLP)
Signum Activation Function

 The Single Layer Perceptron( McCulloch-Pitts Neurons)

with signum activation function is represented as shown
in the Figure below.

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 62

Single Layer Perceptron
Signum Activation Function
 The Single Layer Perceptron is calculated by
calculating the sum of the input vector
multiplied by the corresponding element of the
vector, with each increasing the amount of the
corresponding component of the vector by
weight.
 The value that is displayed in the output is the
input of an activation function.

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 63

Single Layer Perceptron(SLP)
Signum Activation Function

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 64

Single Layer Perceptron(SLP)
Signum Activation Function

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 65

Single Layer Perceptron(SLP)
Unit Step (Threshold) Activation Function
 The Single Layer Perceptron with unit step activation
function is represented as shown in the Figure of next
slide.

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 66

Single Layer Perceptron(SLP)
Unit Step (Threshold) Activation Function

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 67

Single Layer Perceptron(SLP)
Unit Step (Threshold) Activation Function

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 68

Single Layer Perceptron(SLP)
The perceptron consists of 4 parts.
1. Input value or One input layer: The input
layer of the perceptron is made of artificial input
neurons and takes the initial data into the system
for further processing.
2. Weights and Bias
Weight: It represents the dimension or strength
of the connection between units. If the weight to
node 1 to node 2 has a higher quantity, then
neuron 1 has a more considerable influence on
the neuron.
Bias: It is the same as the intercept added in a
linear equation. It is an additional parameter
which task is to modify the output along with the
weighted sum of the input to the other neuron.
@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 69
Single Layer Perceptron(SLP)
3. Net Sum: It calculates the total sum.
4. Activation Function: A neuron can be activated or
not, is determined by an activation function. The
activation function calculates a weighted sum and
further adding bias with it to give the output/result.

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 70

Single Layer Perceptron
 The perceptron works on these simple steps as
described below.
a. In the first step, all the inputs x are multiplied
with their weights w.

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 71

Single Layer Perceptron
b. In this step, add all the increased values and call them
the Weighted sum.

C. In the last step, apply the weighted sum to a

correct Activation Function.

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 72

Perceptron Learning Rule

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 73

Perceptron: Classification

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 74

Implementation of Logical
NOT, AND, and OR
 We can use McCulloch-Pitts neurons to
implement the basic logic gates (e.g. AND, OR,
NOT).
 It is well known from logic that we can construct
any logical function from these three basic logic
gates.
 All we need to do is find the appropriate
connection weights and neuron thresholds to
produce the right outputs for each set of inputs.
 We shall see explicitly how one can construct
simple networks that perform NOT, AND, and
OR.
@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 75
Implementation of Logical AND using
Single Layer Perceptron

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 76

Implementation of Logical AND using
Single Layer Perceptron

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 77

Implementation of Logical AND using
Single Layer Perceptron

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 78

Implementation of Logical OR using Single
Layer Perceptron

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 79

Implementation of Logical OR using Single
Layer Perceptron

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 80

Implementation of Logical
NOT, AND, and OR

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 81

Implementation of Logical
NOT, AND, and OR

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 82

Decision Boundaries for AND
and OR

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 83

Decision Boundary for XOR

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 84

Activation Functions

 An Artificial Neuron simply calculates a “weighted sum”

of its input, adds a bias and then decides whether it
should be “fired” or not.

 Now, the value of Y can be anything ranging from -inf to

+inf.
 The Neuron really doesn’t know the bounds of the
value. So how do we decide whether the neuron should
fire or not.

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 85

Activation Functions

 The “Activation Functions” are used for this purpose. To

check the Y value produced by a neuron and decide
whether outside connections should consider this
neuron as “fired” or not Or “activated” or not.
 Activation functions are mathematical equations that
determine the output of a neural network.
 The Activation function is attached to each neuron in
the network, and determines whether it should be
activated (“fired”) or not, based on whether each
neuron’s input is relevant for the model’s prediction.
 Activation functions also help normalize the output of
each neuron to a range between 1 and 0 or between -1
and 1.

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 86

Activation Functions

 The Activation Functions can be basically

divided into 2 types;
 Linear Activation Function
 Non-linear Activation Functions

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 87

Activation Functions
Linear or Identity Activation Function
 The function is a line or linear as shown in Figure below.
 Therefore, the output of the functions will not be
confined between any range.

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 88

Activation Functions
Linear or Identity Activation Function
 Equation : y=f(x) = x
 Range : (-infinity to infinity)
 It doesn’t help with the complexity or
various parameters of usual data that is
fed to the Neural Networks.

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 89

Activation Functions
Nonlinear Activation Functions
 The Nonlinear Activation Functions are
the most used activation functions.
 It makes it easy for the model to
generalize or adapt with variety of data
and to differentiate between the output.
 The Nonlinear Activation Functions are
mainly divided on the basis of their range
or curves.

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 90

Activation Functions
Step ( Threshold) Function
 Step Function is one of the simplest kind of activation
functions. In this, we consider a threshold value and if
the value of net input say y is greater than the threshold
then the neuron is activated.
 This activation function very basic and it comes to mind
every time if we try to bound output. It is basically a
threshold base classifier, in this, a threshold value is
taken to decide output that neuron should be activated
or deactivated.

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 91

Activation Functions
Step ( Threshold) Function
 Mathematically it is represented as;

 Its output is 1 ( activated) when value > 0 (threshold)

and outputs a 0 ( not activated) otherwise.

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 92

Activation Functions
Piecewise Linear Function

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 93

 Activation Functions
Sigmoid or Logistic Activation Function
 The Sigmoid Function curve looks like a S-shape.

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 94

Activation Functions
Sigmoid or Logistic Activation Function
 This is a smooth function and is continuously
differentiable. The biggest advantage that it has over step
and linear function is that it is non-linear. This is an
incredibly cool feature of the sigmoid function. This
essentially means that when multiple neurons having
sigmoid function as their activation function – the
output is non linear as well. The function ranges from 0-
1 having an S shape.

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 95

Activation Functions
Sigmoid or Logistic Activation Function
 The main reason why we use sigmoid function is
because it exists between (0 to 1). Therefore, it is
especially used for models where we have to predict
the probability as an output.
 Since probability of anything exists only between the
range of 0 and 1, sigmoid is the right choice.
 The function is differentiable.That means, we can find
the slope of the sigmoid curve at any two points.
 The function is monotonic but function’s derivative is
not.

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 96

Activation Functions
Sigmoid or Logistic Activation Function

 More threshold-like
 We can make this more and more threshold-like, or step-like, by
increasing the weights on the links, and so increasing the summed
input.

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 97

Activation Functions
Sigmoid or Logistic Activation Function

 More linear

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 98

Activation Functions
Sigmoid or Logistic Activation Function

 More linear

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 99

Activation Functions
Sigmoid or Logistic Activation Function

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 100

Activation Functions
Sigmoid or Logistic Activation Function

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 101

Activation Functions
Sigmoid or Logistic Activation Function

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 102

Activation Functions
Tanh or hyperbolic tangent Activation
Function
 tanh is also like logistic sigmoid but better. The range of
the tanh function is from (-1 to 1). tanh is also sigmoidal
(s - shaped).

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 103

Activation Functions
Tanh or hyperbolic tangent Activation
Function

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 104

Activation Functions
Tanh or hyperbolic tangent Activation
Function
 The advantage is that the negative inputs will be
mapped strongly negative and the zero inputs will
be mapped near zero in the tanh graph.
 The function is differentiable.
 The function is monotonic while its derivative is
not monotonic.
 The tanh function is mainly used classification
between two classes.
 Both tanh and logistic sigmoid activation functions
are used in feed-forward networks.

@ Dr. Sakuntala Mahapatra 14/12/2020 Neural Networks 105

 PPT CONTENT
NEURAL NETWORKS
CLASS-30
(MODULE-1II)
DATE:15/12/2020
TIME:12.15 PM-1.15 PM

@ Dr. Sakuntala Mahapatra 15/12/2020 Neural Networks 106

Activation Functions
Sigmoid or Logistic Activation Function

 The logistic sigmoid function can cause a neural

network to get stuck at the training time.
 The sigmoid function causes a problem mainly
termed as vanishing gradient problem which
occurs because we convert large input in
between the range of 0 to 1 and therefore their
derivatives become much smaller which does
not give satisfactory output.
 To solve this problem another activation
function such as ReLU is used where we do not
have a small derivative problem.

@ Dr. Sakuntala Mahapatra 15/12/2020 Neural Networks 107

Activation Function
ReLU (Rectified Linear Unit)
 The ReLU is the most used activation function in the
world right now. This is because it is used in almost all
the Convolutional Neural Networks or Deep Learning.

@ Dr. Sakuntala Mahapatra 15/12/2020 Neural Networks 108

Activation Function
ReLU (Rectified Linear Unit)
 The ReLu function gives an output x, if x is positive and
0 otherwise.
 ReLu is less computationally expensive than tanh and
sigmoid because it involves simpler mathematical
operations. That is a good point to consider when we
are designing Deep Neural Networks.

@ Dr. Sakuntala Mahapatra 15/12/2020 Neural Networks 109

Activation Function
ReLU (Rectified Linear Unit)
 The main advantage of using the ReLU function over
other activation functions is that it does not activate all
the neurons at the same time.
 In the ReLU function, if the input is negative it will
convert it to zero and the neuron does not get
activated.
 The ReLU is half rectified (from bottom). f(x) is zero
when x is less than zero and f(x) is equal to x when x is
above or equal to zero.
 Range: [ 0 to infinity)
 The function and its derivative both are monotonic.

@ Dr. Sakuntala Mahapatra 15/12/2020 Neural Networks 110

Activation Function
ReLU (Rectified Linear Unit)
 But the issue is that all the negative values
become zero immediately which decreases the
ability of the model to fit or train from the data
properly.
 That means any negative input given to the
ReLU activation function turns the value into
zero immediately in the graph, which in turns
affects the resulting graph by not mapping the
negative values appropriately.

@ Dr. Sakuntala Mahapatra 15/12/2020 Neural Networks 111

Activation Function
Leaky ReLU
 It is an attempt to solve the dying ReLU
problem.

@ Dr. Sakuntala Mahapatra 15/12/2020 Neural Networks 112

Activation Function
Leaky ReLU
 Leaky ReLU function is nothing but an improved version
of the ReLU function.
 Instead of defining the Relu function as 0 for x less than
0, we define it as a small linear component of x. It can
be defined as:

@ Dr. Sakuntala Mahapatra 15/12/2020 Neural Networks 113

Activation Function
Leaky ReLU
 The leak helps to increase the range of the ReLU
function. Usually, the value of a is 0.01 or so.
 When a is not 0.01 then it is called Randomized
ReLU.
 Therefore the range of the Leaky ReLU is (-infinity to
infinity).
 Both Leaky and Randomized ReLU functions are
monotonic in nature and their derivatives also
monotonic in nature.

@ Dr. Sakuntala Mahapatra 15/12/2020 Neural Networks 114

Activation Function
Radial Basis Function

@ Dr. Sakuntala Mahapatra 15/12/2020 Neural Networks 115

Activation Function
Radial Basis Function

@ Dr. Sakuntala Mahapatra 15/12/2020 Neural Networks 116