Course of B.

E (EIC)
UEI-401
Artificial Intelligent Techniques &
Applications
Lecture 1 Chapter 2: Artificial Neural Networks

Dr . Gagandeep Kaur
Associate Professor,
Department of Electrical & Instrumentation Engineering,
Thapar Institute of Engineering & Technology, Patiala
 Chapter 2 : ARTIFICIAL NEURAL NETWORKS

1.Structure and Functions of a Single Neuron

2.Artificial Neuron Models,
3.Types of Activation Functions
4.Neural Network Architectures
5.Neural Learning
6.Evaluation of Networks
7.Supervised Learning
8.Back Propagation Algorithm
9.Unsupervised Learning
10.Winner Take all Networks
11.Applications of neural networks for classification ,
clustering , pattern associations , function approximation ,
forecasting
 Neural Network
• Neural network is a processing device , either
an algorithm or actual hardware whose design
and functioning is inspired by human /animal
brains
• Computing algorithms has provided solutions
to many problems with this kind of thought
process implementation and is known as
Artificial Neural Network , ANN
• Before discussing ANN Lets see how human brain
works
• In human brain there is a specific type of cell
called “neuron” which helps the human
processor to work ,perform and take decisions
• There are approx 100 million neurons in human
brain
• Each neuron can connect with 200000 other
neurons for processing the signals
• Human brain power lies in these neurons and
their interconnections
• This biological neuron s the basic building
block of Artificial Neural Network , ANN
• To understand the ANN the functioning of
biological neuron is very important
Structure of Biological Neuron
 Parts and Functions of Biological
Neuron
• The biological neuron shown has three
important parts
1. SOMA or cell body : Here the nucleus of cell
is located
2. DENDRITES : where the nerve is connected to
cell body
3. AXON : which carries the impulse of the
neuron
Terminology relationship between
Biological Neuron and Artificial Neuron
Biological neuron Artificial neuron
1.Cell 1.Neuron
2.Dendrites 2. Weights or
interconnection
3.Soma 3. Net Input
4.Axon 4.Output
Artificial Neuron Network structure
Functions of parts of Artificial Neuron
• X ,X are the inputs
1 2

• B is the bias signal

• W , W are weights which help in taking
1 2

decision
• Y is output
The Processing element processes the inputs
and weights and provides output accordingly
Another representation of Artificial
Neural Network model
 OUTPUT of ANN
x and x are inputs
1 2

w and w are weights

1 2

Y = x w +x w +....+x w = ∑ x w
in 1 1 2 2 n n i i

Output = Function(net input calculated)

Y=f(y ) in

Y is output
 Brain Vs Computer
• Comparison between biological neuron and
artificial neuron
1. Speed
2. Processing
3. Size and complexity
4. Storage capacity
5. Tolerance
6. Control mechanism
 Advantages of Neural Networks(NN)
• Neural network is having ability to derive
meaning from complicated or imprecise data
• Neural networks can extract patterns and
detect trends from imprecise data which other
computer techniques may not
• Neural networks can answer few uncertainties
of the kind “what if”
 Advantages of Artificial neural
network(ANN)
• Adaptive learning : ANN have ability to learn
how to do task from given data
• Self organization : ANN can create its own
organisation or representation of information
it receives during learning time
• Real time operation : ANN can do real time
operations
• Fault tolerance : ANN havegreat fault
tolerance
 Applications of ANN
• Air traffic control
• Appraisal and valuation of property , building , automobiles , machinery
• weather forecast
• Traffic flaws
• Voice recognition
• Image analysis
• Handwriting and typewriting analysis
• river water level
• Fraud detection
• Data mining ,cleaning and validation
• Complex physical and chemical process
• Echo patterns from SONAR and RADAR
• Econometric modelling for statistical analysis
• Betting in share market , horse race , sporting events
• Music composition
• Scheduling of air planes , buses and trains
• Strategies for gamming business and war
• Medical diagnosis and medical research etc
 Evolution of neural networks
• It all started in 1943 with McCullock and Pitts neuron by
McCullock and Pitts
• Then in 1949 Hebb’s network
• 1958 it is Perceptron
• 1960 it is Adaline
• 1972 it is Kohonen self organising maps
• 1982 is Hop field network
• 1986 is Back propagation network
• 1988 is Counter propagation network
• 1987-1990 is Adaptive Resonance theory(ART)
• 1988 is Radial bias function
• 1988 is Neo cognition
 References for Lecture
1. Principles of Soft Computing , Second edition ,
S.N. Sivanandam , S.N. Deepa , Wiley , India
2. Artificial Intelligence, E.Rich , K.Knight , TMH
3. Artificial Intelligence, D.Khemani , Mcgraw Hill
4. Introduction to Artificial Intelligence and Expert
System, D.W.Patterson , PHI
5. Fuzzy logic with engineering applications,
T.J.Ross, second edition, Wiley India
6. Google.com
Thanks
 Course of B.E (EIC)
UEI-401
Artificial Intelligent Techniques &
Applications
Lecture 2: Chapter :2 Artificial Neural Networks

Dr.Gagandeep Kaur
Associate Professor,
Department of Electrical & Instrumentation Engineering,
Thapar Institute of Engineering & Technology, Patiala
 CHAPTER 2 : ARTIFICIAL NEURAL
NETWORKS

1.STRUCTURE AND FUNCTIONS OF A SINGLE NEURON

2.ARTIFICIAL NEURON MODELS
3.TYPES OF ACTIVATION FUNCTIONS
4.NEURAL NETWORK ARCHITECTURES
5.NEURAL LEARNING
6.EVALUATION OF NETWORKS
7.SUPERVISED LEARNING &UNSUPERVISED LEARNING
8.BACK PROPAGATION ALGORITHM
9.WINNER TAKE ALL NETWORKS
10.APPLICATIONS OF NEURAL NETWORKS FOR CLASSIFICATION,
CLUSTERING ,PATTERN ASSOCIATIONS, FUNCTION
APPROXIMATION , FORECASTING
 Neural Networks are..............
• Neural networks are those information
processing systems which are constructed and
implemented to model the human brain
• Main objective of NN is to develop a
computational device for modelling the brain to
perform faster than traditional system and of
course they are more accurate also
• ANN is efficient information processing system
which resembles in characteristics with biological
neural network which exists in human brain
 ANN........
• ANN possess large number of interconnected
processing elements called “nodes” or “units” or
“neurons”
• Neurons operate in parallel and are configured in an
architecture which is specific and called as ANN
architect
• Each neuron is connected with other by connecting link
• Each connecting link is associated with a weight which
contains information about the information signal
• The information is used by neuron net (output)to solve
a particular problem
 ANN.......
• ANN’s collective behaviour is characterised by
their ability to learn, recall and generalise
training patterns or data similar to human
brain
• This ANN processing element is called
“neurons” or “artificial neurons”
Architecture of simple neuron net
 Notations in ANN
• X is the input
• Y is the output
• W is the weight
Biological neural network
 Notations in Biological neuron
• Cell body or SOMA : where nucleus is located
• Dendrite : where the nerve connected to cell body(tree
like network made of nerve fibre connected to cell
body)
• Axon : which carries impulse of the neuron(it is single
long connection extending from the cell body and
carries signal from neuron)
• The end of Axon splits into fine strands and each strand
end in bulb like structure called synapse
• Synapse: through synapse the neuron introduces the
signal to other neuron . The synapse’s receiving end is
connected both to dendrites and cell body
 Notations in Biological neuron
• Chemically the information is processed in biological
cell/neuron
• The processing of chemical increases or decreases the
electric bio-potential inside the receiving cell
• If electric potential reaches the threshold then the
receiving cell fires a pulse or “action potential” of fixed
strength and duration is sent out through the axon to
the synaptic junctions of other cells
• After firing a cell has to weight for some time called
“refractory period” before it can fire again
 Notations in Biological neuron
• Synapse is said to be “inhibitory” if they
hinder the firing impulses
• Synapse is said to be “excitatory” if it causes
the firing of receiving cell
 Artificial Neural Network
• Different models of artificial neural networks
are available which are based on the basic
mathematical model of neuron
Operation of Neural Net
 Layers of neurons
• Neurons can be visualised for their arrangement
in layers for different applications
• Neural networks consists of interconnected
processing elements i.e neurons in such a way
that output is connected through weights to the
other processing elements or itself
• The geometry and arrangements are very
important
• The point where the connections originate and
terminate should be specified
 Models of Neural Networks
(NN or ANN)
The basic models of ANN are specified by
following basic entities
(i)Connections
(ii)Training/Learning
(iii)Activation functions
Arrangements of neurons to form layers and
connection patterns within and between
layers is called network architecture
 Basic models of Artificial Neural
Network(ANN)
1. Single layered feed forward network
2. Multilayer feed forward network
3. Single node with its own feed back
4. Single layer recurrent network
5. Multilayer recurrent network
Single layered feed forward network
 Single layered feed forward network
• When a layer of processing nodes is formed ,
the input can be connected to these nodes
with various weights , resulting in series of
outputs , one per node
• This kind of connectivity is called single
layered feed forward network
• Simple single/one input and one output layer
is there in this connection
Multilayer feed forward network
 Multilayer feed forward network
• Multilayer feed forward network is formed by
interconnection of several layers
• Input layer receives the input and the function of this layer
is to buffers the input signal
• Output layer generates the output of network
• Layer in between the input layer and output layer is called
Hidden layer
• Hidden layer is internal to the network and no direct
connection with external environment
• More number of hidden layers more complex is the
network
• The network shown is fully connected multilayer feed
forward network .There is NO feedback signal in this model
Single node with its own feed back
 Single node with its own feed back

• In Single node with its own feed back model

the output of processing element is directed
back to the processing elements in the same
layer than it is called lateral feedback
Single layer recurrent network
Multilayer recurrent network
Different neural networks
 learning
1. Supervised learning
2. Unsupervised learning
3. Reinforced learning
 References for Lecture
1. Principles of Soft Computing , S.N Sivanandan
,S.N Deepa ; Wiley India (UEI-401 Lecture 2)
2. Artificial Intelligence, E.Rich , K.Knight , TMH
3. Artificial Intelligence, D.Khemani,Mcgraw Hill
4. Introduction to Artificial Intelligence and Expert
System, D.W.Patterson,PHI
5. Oxford dictionary
6. Google.com
THANKS
 Course of B.E (EIC)
UEI-401
Artificial Intelligent Techniques &
Applications
Lecture 3 Chapter 2: Artificial Neural Networks

Dr . Gagandeep Kaur
Associate Professor,
Department of Electrical & Instrumentation Engineering,
Thapar Institute of Engineering & Technology, Patiala
 Chapter 2 : ARTIFICIAL NEURAL NETWORKS

1.Structure and Functions of a Single Neuron

2.Artificial Neuron Models,
3.Types of Activation Functions
4.Neural Network Architectures
5.Neural Learning
6.Evaluation of Networks
7.Supervised Learning
8.Back Propagation Algorithm
9.Unsupervised Learning
10.Winner Take all Networks
11.Applications of neural networks for classification ,
clustering , pattern associations , function approximation ,
forecasting
 Learning
i. The main property or feature on ANN is its
capability to learn
ii. Learning or Training is a process by which
the NN or ANN adapts itself to the stimulus
or inputs by making adjustments within its
own network to achieve the target output
iii. It can be classified in two types
(a)Parameter learning
(b)Structure learning
 Parameter Learning & Structure
Learning
• Parameter learning updates the connecting
weights in the neural network
• Structure learning focuses on the change in
network structure i.e. the number of
processing elements and their connection
type
• BOTH THESE LEARNING ARE PERFORMED
SIMULTANEOUSLY OR SEPARATELYY
Another classification of learning....
1. Supervised learning: where you need
teacher or trainer
2. Unsupervised learning: where you do not
need teacher or trainer
3. Reinforcement learning : Similar to
supervised learning but here the data or
information is less or incomplete or surety is
less
 Supervised Learning

In Supervised learning or training

• Learning is performed with the help of a teacher or
trainer
• ANN following the supervised learning ,each input
vector requires a corresponding target vector which
represents the desired output
• Input vector along with the target vector is called
“training pair”
• Network is informed precisely about what should be
emitted as output
• During training the input vector is presented in the
network , which results in output network
 Supervised Learning
• Error is used for adjustment of weights till both desired & actual
output matches

• Correct target output values are known for every input pattern
 Unsupervised Learning
• In ANNs following unsupervised learning, the input
vectors of similar type are grouped without the use
of training data to specify how a member of each
group looks or to which group a number belongs.

• In the training process, the network receives the

input patterns and organizes these patterns to form
clusters.
 Unsupervised Learning
• When a new input pattern is applied, the neural
network gives an output response indicating the
class to which the input pattern belongs.

• If for an input, a pattern class cannot be found then a

new class is generated .

• The block diagram of unsupervised learning is shown

in Figure
Unsupervised Learning
 Reinforcement Learning
• This learning process is similar to supervised
learning.

• In the case of supervised learning, the correct target

output values are known for each input pattern. But,
in some cases, less information might be available.
 Reinforcement Learning
• For example, the network might be told that its actual
output is only "50% correct" or so.

• Thus, here only critic information is available, not the

exact information.

• The learning based on this critic information is called

reinforcement learning and the feedback sent is called
reinforcement signal.
 Reinforcement Learning
• The block diagram of reinforcement learning is
shown in figure
 Reinforcement Learning
• The reinforcement learning is a form of supervised
learning because the network receives some
feedback from its environment. However, the
feedback obtained here is only evaluative and not
instructive
• The external reinforcement signals are processed in
the critic signal generator, and the obtained critic
signals are sent to the ANN for adjustment of
weights properly so as to get better critic feedback in
future.
 Reinforcement Learning
• The reinforcement learning is also called
learning with a critic as opposed to learning
with a teacher, which indicates supervised
learning.
 Activation Functions

• To obtain exact output, some force or

activation may be given.

• The activation function is applied over the net

input to calculate the output of an ANN.
 Activation Functions
• The information processing of a processing
element can be viewed as consisting of two
major parts: input and output.

• This function serves to combine activation,

information or evidence from an external
source or other processing elements into a net
input to the processing element.
 Types of Activation Functions

1. Identity function
2. Binary step function
3. Bipolar step function
4. Sigmoidal functions
5. Ramp function
 Types of Activation Functions
1. Identity function: It is a linear function and
can be defined as
f(x) = x for all x
The output here remains the same as input.
The input layer uses the identity activation
function.
Types of Activation Functions
 Types of Activation Functions
2. Binary step function: This function can be
defined as
f(x) = { 1 if x ≥ θ
{ 0 if x< θ
where θ represents the threshold value. This
function is most widely used in single-layer
nets to convert the net input to an output that
is a binary (1 or 0).
Types of Activation Functions
 Types of Activation Functions
3. Bipolar step function: This function can be
defined as
f(x) = { 1 if x ≥ θ
{ - 1 if x< θ

where θ represents the threshold value. This

function is also used in single-layer nets to
convert the net input to an output that is
bipolar(+ 1 or -1).
Types of Activation Functions
 Types of Activation Functions
4. Sigmoidal functions-. The sigmoidal functions
are widely used in back-propagation nets
because of the relationship between the value
of the functions of a point and the value of
the derivative at that point which reduces the
computational burden during training.
• Sigmoidal functions- are of two types:
– Binary sigmoid function
– Bipolar sigmoid function
 Types of Activation Functions
(i) Binary sigmoid function: It is also termed as logistic
sigmoid function or unipolar sigmoid function.
It can be defined as

where A is the steepness parameter. The derivative

of this function is

Here the range of the sigmoid function is from 0 to 1.

Types of Activation Functions
 Types of Activation Functions
(ii) Bipolar sigmoid function: This function is defined as

where λ is the steepness parameter and the sigmoid

function range is between -1 and+ 1. The derivative
of the function can be

• The graphical representations of the functions is

Types of Activation Functions
 Types of Activation Functions
• The bipolar sigmoidal function is closely related to
hyperbolic tangent function, which is written as

• The derivative of the hyperbolic tangent function is

• If the network uses a binary data, it is better to

convert it to bipolar form and use the bipolar
sigmoidal activation function or hyperbolic tangent
function.
 Types of Activation Functions
5. Ramp function: The ramp function is defined as

The graphical representations of the functions is

Types of Activation Functions
 Other important parameters associated
with ANNs
• Weights
• Bias
• Threshold
• Learning rate (α)
• Momentum Factor
• Vigilance Parameter
• Notations
 References for Lecture
1. Principles of soft computing, S.N. Sivanandam , S.N.
Deeepa ; Wiley India(Chapter 4 ANN of syllabus of
UEI-401)
2. Artificial Intelligence, E.Rich , K.Knight , , TMH
3. Artificial Intelligence, D.Khemani , Mcgraw Hill
4. Introduction to Artificial Intelligence and Expert
System, D. W . Patterson , PHI
5. Fuzzy Logic with Engineering Applications , T.J Ross ;
Wiley India
6. Oxford dictionary
7. Google.com
 Course of B.E (EIC)
UEI-401
Artificial Intelligent Techniques &
Applications
Lecture 4 Chapter 2: Artificial Neural Networks

Dr . Gagandeep Kaur
Associate Professor,
Department of Electrical & Instrumentation Engineering,
Thapar Institute of Engineering & Technology, Patiala
 Chapter 2 : ARTIFICIAL NEURAL NETWORKS

1.Structure and Functions of a Single Neuron

2.Artificial Neuron Models,
3.Types of Activation Functions
4.Neural Network Architectures
5.Neural Learning
6.Evaluation of Networks
7.Supervised Learning
8.Back Propagation Algorithm
9.Unsupervised Learning
10.Winner Take all Networks
11.Applications of neural networks for classification ,
clustering , pattern associations , function approximation ,
forecasting
 Important parameters associated with ANNs

• Weights
• Bias
• Threshold
• Learning rate (α)
• Momentum Factor
• Vigilance Parameter
• Notations
Other important parameters associated with ANNs
Weights
• In the architecture of an ANN, each neuron is
connected to other neurons by means of directed
communication links, and each communication link is
associated with weights.
• The weights contain information about the input
signal which is used by the net to solve a problem.
Other important parameters associated with ANNs

• It can be represent in terms of matrix called

connection matrix
Other important parameters associated with ANNs
• where wi = [wi1, wi1 , ... , win ] Ƭ, i = 1,2, ... , n, is the
weight vector of processing dement and wij is the
weight from processing element“ i.e. source node to
processing element "j'' i.e. destination node.
• If the weight matrix W contains all the adaptive
elements of an ANN, then the set of all W matrices
will determine the set of all possible information
processing configurations for this ANN. The ANN can
be realized by finding an appropriate matrix W.
• The weights encode long-term memory (LTM)
• The activation states of neurons encode short-term
memory (STM) in a neural network.
Other important parameters associated with ANNs

Bias : It has impact in calculating the net input in the

network.
• It is included by adding a component x0 = 1 to the
input vector X. So X becomes

X= (1,X1, ... ,Xi, ... ,Xn)

Other important parameters associated with ANNs
• The bias is considered like another weight i.e.
W0j = bj as shown below for simple neural network
Other important parameters associated with ANNs

• Now the net input to the output neuron Yj is

calculated as
Other important parameters associated with ANNs
• The activation function is applied over this net input
to calculate the output.
• The bias can also be explained as follows
• Consider an equation of straight line,

where x is the input, m is the weight, c is the bias and

y is the output.
Other important parameters associated with ANNs

• The equation of the straight line can also be

represented as a block diagram shown below

• Bias plays a major role in determining the output of

the network.
Other important parameters associated with ANNs

• Bias is of two types

1. Positive Bias : It helps in increasing the net input
of network.
2. Negative Bias : It helps in decreasing the net input
of network.

So bias effect can cause changes in the output of the

network
Other important parameters associated with ANNs

• Threshold : It is a set value based upon which the

final output of the network may be calculated.
• The threshold value is used in the activation function.
• A comparison is made between the calculated net
input and the threshold to obtain the network
output.
• For each and every application there is threshold
limit.
Other important parameters associated with ANNs

• The activation function using threshold can be

defined as

• Where θ is the fixed threshold value.

Other important parameters associated with ANNs

• Learning Rate : It is denoted by "α."

• It is used to control the amount of weight adjustment

at each step of training.

• The learning rate, ranging from 0 -to 1

• It also determines the rate of learning at each time
step.
Other important parameters associated with ANNs
• Momentum Factor : Convergence is made faster if a
momentum factor is added to the weight updation
process.
• This is generally done in the back propagation
network. If momentum has to be used, the weights
from one or more previous training patterns must be
saved.
• Momentum helps the net in reasonably large weight
adjustments until the corrections are in the same
general direction for several patterns.
Other important parameters associated with ANNs

• Vigilance Parameter (ρ ): It is used in adaptive

resonance theory (ART) network to control the
degree of similarity required for patterns to be
assigned to the same cluster unit.

• The choice of it ranges between approximately from

0.7 to 1 to perform useful work in controlling the
number of clusters.
 References for Lecture
1. Principles of soft computing, S.N. Sivanandam , S.N.
Deeepa ; Wiley India(Chapter 2 ANN of syllabus of
UEI-401)
2. Artificial Intelligence, E.Rich , K.Knight , , TMH
3. Artificial Intelligence, D.Khemani , Mcgraw Hill
4. Introduction to Artificial Intelligence and Expert
System, D. W . Patterson , PHI
5. Fuzzy Logic with Engineering Applications , T.J Ross ;
Wiley India
6. Oxford dictionary
7. Google.com
 Course of B.E (EIC)
UEI-401
Artificial Intelligent Techniques &
Applications
Chapter 2: Artificial Neural Networks

Lecture 5

Dr . Gagandeep Kaur
Associate Professor,
Department of Electrical & Instrumentation Engineering,
Thapar Institute of Engineering & Technology, Patiala
1
 Chapter 2 : ARTIFICIAL NEURAL NETWORKS

1.Structure and Functions of a Single Neuron

2.Artificial Neuron Models,
3.Types of Activation Functions
4.Neural Network Architectures
5.Neural Learning
6.Evaluation of Networks
7.Supervised Learning
8.Back Propagation Algorithm
9.Unsupervised Learning
10.Winner Take all Networks
11.Applications of neural networks for classification ,
clustering , pattern associations , function approximation ,
forecasting

2
 McCULLOCH–PITTS (MP)NEURON

• The McCulloch-Pitts neuron was the earliest neural

network discovered in 1943 also called as M-P
neuron.
• The M-P neurons are connected by directed
weighted paths.
• The activation of a M-P neuron is binary, that is, at
any time step the neuron may fire or may not fire.

3
 McCULLOCH–PITTS NEURON

• The weights associated with the communication links

may be excitatory (weight is positive) or inhibitory
(weight is negative).

• All the excitatory connected weights entering into a

particular neuron will have same weights.

• The threshold plays a major role in M-P neuron

4
 McCULLOCH–PITTS NEURON

• There is a fixed threshold for each neuron, and if the

net input to the neuron is greater than the threshold
then the neuron fires.
• Also, nonzero inhibitory input would prevent the
neuron from firing.
• The M-P neurons are most widely used in the case
of logic functions.

5
 McCULLOCH–PITTS NEURON
ARCHITECTURE
• It is excitatory with weight (w > 0) or inhibitory with
weight -p(p < 0).

• Inputs from x1 to xn possess excitatory weighted

connections and inputs from xn+ 1 to xn+m possess
inhibitory weighted interconnections.

• The output neuron is based upon the threshold, the

activation function here is defined as

6
7
 McCULLOCH–PITTS NEURON
ARCHITECTURE

• For inhibition to be absolute, the threshold with the

activation function should satisfy the following
condition:

• The output will fire if it receives says “k” or more

excitatory inputs but no inhibitory inputs, where
8
 McCULLOCH–PITTS NEURON
ARCHITECTURE

• It has no particular training algorithm.

• The analysis has to be performed to determine the values of

the weights and the threshold.

• The weights of the neuron are set along with the threshold to
make the neuron "perform a simple logic function.

9
 McCULLOCH–PITTS NEURON
ARCHITECTURE
• It is used as building blocks on which function or
phenomenon, which represented as a logic function
can be modeled.

10
 LINEAR SEPARABILITY
 Linear separability is the concept wherein the separation of the
input space into regions is based on whether the network response
is positive or negative.
 Consider a network having
positive response in the first
quadrant and negative response
in all other quadrants (AND
function) with either binary or
bipolar data, then the decision
line is drawn separating the
positive response region from
the negative response region.

11
HEBB NETWORK
Donald Hebb stated in 1949 that in the brain, the learning is
performed by the change in the synaptic gap. Hebb explained it:

“When an axon of cell A is near enough to excite cell B, and

repeatedly or permanently takes place in firing it, some growth
process or metabolic change takes place in one or both the cells
such that A’s efficiency, as one of the cells firing B, is increased.”

According to the Hebb rule, the weight vector is found to

increase proportionately to the product of the input and the
learning signal. Here the learning signal is equal to the neuron's
output.

12
HEBB NETWORK
• In Hebb learning, if two interconnected neurons are 'on'
simultaneously then the weights associated with these
neurons can be increased by the modification made in their
synaptic gap (strength). The weight update in Hebb rule is
given by

• The Hebb rule is more suited for bipolar data than binary
data. If binary data is used, the above weight updation
formula cannot distinguish two conditions namely:

13
HEBB NETWORK
1. A training pair in which an input unit is "on" and target value
is "off."
2. A training pair in which both the input unit and the target
value are "off.“

Thus, there are limitations in Hebb rule application over

binary data. Hence, the representation using bipolar data is
advantageous.

14
 HEBB NETWORK
Flow Chart of Training Algorithm
• The training algorithm is used for the calculation and
adjustment of weights.

• The flowchart for the training algorithm of Hebb network is

given as

15
16
 HEBB NETWORK
Training Algorithm
Step 0: First initialize the weights. Basically in this network they
may be set to zero, i.e., wi = 0 for i= 1 to n where "n" may
be the total number of input neurons.
Step 1: Steps 2-4 have to be performed for each input training
vector and target output pair, s: i.

Step 2: Input units activations are set. Generally, the activation

function of input layer is identity function:
xi = si for i= 1 to n

Step 3: Output units activations are set: y = t.

17
 HEBB NETWORK
Training Algorithm
Step 4: Weight adjustments and bias adjustments are performed:

• The above five steps complete the algorithmic process.

• The Hebb rule can be used for pattern association, pattern

categorization, pattern classification and over a range of other
areas.

18
 Problems
1. For the network shown, we will find the net input to the output
neuron.

Solution: The given neural net consists of three input

neurons and one output neuron. The inputs and weights are

19
• The net input can be calculated as

= 0.3 X 0.2+0.5 X 0.1 + 0.6 X (-0.3)

= 0.06 + 0.05-0.18 = -0.07

20
 Problems
2. Obtain the output of the neuron Y for the network shown in
figure using activation functions as: (i) binary sigmoidal and
(ii) bipolar sigmoidal.

• Solution: The given network has three input neurons with bias
and one output neuron. These form a single-layer network.

21
 Problems
The inputs are given as [x1 , x2, x3] = [0.8,0.6,0.4] and the weights
are [w1 , w2 , w3 ] = [0.1, 0.3, -0.2] with bias b = 0.35
(its input is always 1).
The net input to the output neuron is

22
 Problems
(i) For binary sigmoidal activation function.

(ii) For bipolar sigmoidal activation function.

23
24
 References for Lecture
1. Principles of Soft Computing , S.N. Sivanandan, SN
Deepa ; Wiley India (Chapter 2: ANN of course of
UEI-401)
2. Fuzzy logic with Engineering Applications, T.J Ross ,
Second edition ,Wiley India (Chapter 3: Fuzzy Logic of
course of UEI-401)
3. Artificial Intelligence, E . Rich , K . Knight ,
TMH(Chapter 1: Overview of Artificial Intelligence)
4. Artificial Intelligence, D. Khemani , Mcgraw Hill
5. Introduction to Artificial Intelligence and Expert
System, D.W. Patterson , PHI
6. Google.com
 Course of B.E (EIC)
UEI-401
Artificial Intelligent Techniques &
Applications
Chapter 2: Artificial Neural Networks

Lecture 6

Dr . Gagandeep Kaur
Associate Professor,
Department of Electrical & Instrumentation Engineering,
Thapar Institute of Engineering & Technology, Patiala
1
 Chapter 2 : ARTIFICIAL NEURAL NETWORKS

1.Structure and Functions of a Single Neuron

2.Artificial Neuron Models,
3.Types of Activation Functions
4.Neural Network Architectures
5.Neural Learning
6.Evaluation of Networks
7.Supervised Learning
8.Back Propagation Algorithm
9.Unsupervised Learning
10.Winner Take all Networks
11.Applications of neural networks for classification ,
clustering , pattern associations , function approximation ,
forecasting

2
 Supervised Learning Network

Supervised learning networks are listed below

1. Simple perceptrons
2. Adaline : Single–layer feed-forward network
3. Madaline : Multilayer feed forward network
4. Back Propagation network

3
 Supervised Learning Network
Perceptron Networks

The key points to be noted in a perceptron network

are:
1. The perceptron network consists of three units,
namely, sensory unit (input unit), associator unit
(hidden unit), response unit (output unit).
2. The sensory units are connected to associator units
with fixed weights having values 1, 0 or -1, which are
assigned at random.

4
 Supervised Learning Network
Perceptron Networks
3. The binary activation function is used in sensory unit
and associator unit.
4.. The response unit has an activation of 1, 0 or -1. The
binary step with fixed threshold θ is used as
activation for associator. The output signals that are
sent from the associator unit to the response unit are
only binary.

5
 Supervised Learning Network
Perceptron Networks

5. The output of the perceptron network is given by

y = f(yin)

where f(yin) is activation function and is defined as

1 if yin> θ
f(yin) = 0 if - θ ≤ yin ≤ θ
1 if yin< - θ

6
 Supervised Learning Network
Perceptron Networks
6. The perceptron learning rule is used in the weight
updation between the associator unit and the
response unit. For each training input, the net will
calculate the response and it will determine whether
or not an error has occurred.
7. The error calculation is based on the comparison of
the targets with those of the calculated outputs.

7
 Supervised Learning Network
Perceptron Networks
9. The weights will be adjusted on the basis of the
learning rule if an error has occurred for a particular
training pattern i.e,

• If no error occurs, there is no weight updation and

hence the training process may be stopped.
• the target value "t" is + 1 or -1 and α is the learning
rate.
8
 Supervised Learning Network
Perceptron Networks
• these learning rules begin with an initial guess at the
weight values and then successive adjustments are
made on the basis of the evaluation of an objective
function.
• Eventually, the learning rules reach near-optimal or
optimal solution in a finite number of steps.

9
10
 Supervised Learning Network
Original Perceptron Networks
1 Sensory Unit
2. Associator Unit
3. Response Unit

11
 Supervised Learning Network
Perceptron Learning Rule
• Learning signal is the difference between the desired and
actual response of neuron.
The perceptron learning rule is explained as follows :
• Consider a finite "n" number of input training vectors, with
their associated target(desired) values x(n) and t{n), where
"n" ranges from 1 to N.

12
 Supervised Learning Network
Perceptron Networks
The weight updation in case of perceptron learning is as shown.
• lf y ≠ t, then
w(new) = w (old) + α tx {α - learning rate)
else, we have
w (new) = w (old)
• If there is a weight vector W, such that
f(x(n) W) = t(n) for all n

• Then for any starting vector w1, the perceptron learning rule
will converge to a weight vector that gives the correct
response for all training patterns and this learning takes place
within a finite number of steps provided that solution exists.
13
 Supervised Learning Network
Perceptron Networks

14
 Supervised Learning Network
Perceptron Networks
• The flowchart depicted here
presents the flow of the training
process
• First the basic initialization
required for the training process
is performed.
• The entire loop of the training
process continues until the
training input pair is presented
to the network.
• The training {weight updation) is
done on the basis of the
comparison between the
calculated and desired output.
• The loop is terminated if there is
no change in weight. 15
 Supervised Learning Network
Perceptron Training Algorithm

Step 0: Initialize the weights and the bias (for

calculation they can be set to zero).Also initialize
the learning race α(0 < α ≤ 1). For simplicity α is
set to 1.
Step 1: Perform Steps 2-6 until the final stopping
condition is false.
Step 2: Perform Steps 3-5 for each training pair
indicated by s:t.

16
 Supervised Learning Network
Perceptron Training Algorithm

Step 3: The input layer containing input units is applied

with identity activation functions:
xi =si
Step 4: Calculate the output of the network. To do so, first
obtain the net input:

• where "n" is the number of input neurons in the input layer.

Then apply activations over the net input calculated to obtain
the output:

17
 Supervised Learning Network
Perceptron Training Algorithm

Step 5: Weight and bias adjustment: Compare the value of the

actual (calculated) output and desired (target) output.
lf y ≠ t, then
wi (new) = wi (old) + α tx
b (new) = b (old) + α t
else, we have
wi (new) = wi (old)
b (new) = b (old)
Step 6: Train the network until there is no weight change. This is
the stopping condition for the network. If this condition is
not met, then start again from Step 2.

18
19
20
21
22
23
24
25
26
• Adaline , Madaline and Back propagation
network are the supervised learning based
neural network algorithms
• The problems we will be solving in the tutorial
classes

27
 References for Lecture
1. Principles of Soft Computing , S.N. Sivanandan, SN Deepa
; Wiley India (Chapter 2 ANN and 3: Fuzzy Logic of
course of UEI-401)
2. Fuzzy logic with engineering applications, T.J Ross ,
Second edition ,Wiley India (Chapter 3: Fuzzy Logic of
course of UEI-401)
3. Artificial Intelligence, E . Rich , K . Knight ,
TMH(Chapter1:Overview of AI of the course of UEI-401)
4. Artificial Intelligence, D. Khemani , Mcgraw Hill
5. Introduction to Artificial Intelligence and Expert System,
D.W. Patterson , PHI
6. Google.com
 Course of B.E (EIC)
UEI-401
Artificial Intelligent Techniques &
Applications
Chapter 2: Artificial Neural Networks

Lecture 7

Dr . Gagandeep Kaur
Associate Professor,
Department of Electrical & Instrumentation Engineering,
Thapar Institute of Engineering & Technology, Patiala
1
 Chapter 2 : ARTIFICIAL NEURAL NETWORKS

1.Structure and Functions of a Single Neuron

2.Artificial Neuron Models,
3.Types of Activation Functions
4.Neural Network Architectures
5.Neural Learning
6.Evaluation of Networks
7.Supervised Learning
8.Back Propagation Algorithm
9.Unsupervised Learning
10.Winner Take all Networks
11.Applications of neural networks for classification ,
clustering , pattern associations , function approximation ,
forecasting

2
 Back-Propagation Network

• This network has reawakened the scientific and

engineering community to the model in and
processing of numerous quantitative phenomena
using neural networks.
• It is applied to multilayer feed-forward network
consisting of processing elements with continuous
differentiable activation functions.

3
 Back-Propagation Network

• The aim of the neural network is to train the

net to achieve a balance between the net's
ability to respond (memorization) and its
ability to give reasonable responses to the
input that is similar but not identical to the
one that is used in training(generalization).

4
Back-Propagation Network

5
 Back-Propagation Network
Flowchart for training Process
• The flowchart for the training process using a BPN is shown
The terminologies used in the flowchart and in the training
algorithm are as follows:
• x = input training vecro.r (x1, ... , xi;, ... , xn)
• t = target output vector (t1, ... , tk, ... , tm)
• α = learning rate parameter
• xi = input unit i. (Since the input layer uses identity activation
function, the input and output signals here are same.)
• ν0j = bias on jth hidden unit
• ωok = bias on kth output unit
• Z j=hidden unit j. The net input to Zj is

6
 Back-Propagation Network
Flowchart for training Process

and the output is

yk = output unit k. The net input to yk is

7
 Back-Propagation Network
Flowchart for training Process
and the output is

8
Back-Propagation Network
Flowchart for training Process

9
 Back-Propagation Network

Step 0: Initialize weights and learning rate (take

some small random values).
Step 1: Perform Steps 2-9 when stopping condition is
false.
Step 2: Perform Steps 3-8 for each training pair.
Step 3: Each input unit receives input signal x; and
sends it to the hidden unit (i = 1 to n}.
Step 4: Each hidden unit Zj(j = 1 to p) sums irs Weighted
input signals to calculate net input:

10
 Back-Propagation Network

Calculate output of the hidden uilit by applying its

activation functions over Zinj (binary or bipolar
sigmoidal activation function):

and send the output signal from the hidden unit to

the input of output layer units.
Step 5: For each output unit yk (k = 1 to m), calculate
the net input:

11
 Back-Propagation Network
and apply the activation function to compute output signal

Back Propogation of error(Phase II)

Step6: Each output unit yk (k = 1 to m), receives a target pattern
corresponding to the input training pattern and computes the
error correction term:

On the basis of the calculated error correction term, update the

change in weights and bias:

Also, send δk to the hidden layer backwards.

12
 Back-Propagation Network
Step 7: Each hidden unit (zinj = 1 to p) sums its delta
inputs from the output units:

The term δinj gets multiplied with the derivative of

f(zinj) to calculate the error term:

On the basis of the calculated δj , update the change in

weights and bias:

13
 Back-Propagation Network
Weight and bias updation (Phase Ill)
Step 8: Each output unit (yk, k = 1 to m) updates the
bias and weights:

Each hidden unit (zj = 1 top) updates its bias and weights:

Step 9: Check for the stopping condition. The stopping

condition may be certain number of epochs reached or
when the actual output equals the target output.

14
15
16
17
18
19
20
21
22
23
24
25
 References
1. Principles of Soft Computing , S.N. Sivanandan, SN Deepa
; Wiley India (Chapter 2 ANN and 3: Fuzzy Logic of
course of UEI-401)
2. Fuzzy logic with engineering applications, T.J Ross ,
Second edition ,Wiley India (Chapter 3: Fuzzy Logic of
course of UEI-401)
3. Artificial Intelligence, E . Rich , K . Knight ,
TMH(Chapter1:Overview of AI of the course of UEI-401)
4. Artificial Intelligence, D. Khemani , Mcgraw Hill
5. Introduction to Artificial Intelligence and Expert System,
D.W. Patterson , PHI
6. Google.com
 Course of B.E (EIC)
UEI-401
Artificial Intelligent Techniques &
Applications
Chapter 2: Artificial Neural Networks

Lecture 8

Dr . Gagandeep Kaur
Associate Professor,
Department of Electrical & Instrumentation Engineering,
Thapar Institute of Engineering & Technology, Patiala
1
 Chapter 2 : ARTIFICIAL NEURAL NETWORKS

1.Structure and Functions of a Single Neuron

2.Artificial Neuron Models,
3.Types of Activation Functions
4.Neural Network Architectures
5.Neural Learning
6.Evaluation of Networks
7.Supervised Learning
8.Back Propagation Algorithm
9.Unsupervised Learning
10.Winner Take all Networks
11.Applications of neural networks for classification ,
clustering , pattern associations , function approximation ,
forecasting

2
ADALINE
 Course of B.E (EIC)
UEI-401
Artificial Intelligent Techniques &
Applications
Chapter 2: Artificial Neural Networks

Lecture 8

Dr . Gagandeep Kaur
Associate Professor,
Department of Electrical & Instrumentation Engineering,
Thapar Institute of Engineering & Technology, Patiala
1
 Chapter 2 : ARTIFICIAL NEURAL NETWORKS

1.Structure and Functions of a Single Neuron

2.Artificial Neuron Models,
3.Types of Activation Functions
4.Neural Network Architectures
5.Neural Learning
6.Evaluation of Networks
7.Supervised Learning
8.Back Propagation Algorithm
9.Unsupervised Learning
10.Winner Take all Networks
11.Applications of neural networks for classification ,
clustering , pattern associations , function approximation ,
forecasting

2
ADALINE
 References
1. Principles of Soft Computing , S.N. Sivanandan, SN Deepa
; Wiley India (Chapter 2 ANN and 3: Fuzzy Logic of
course of UEI-401)
2. Fuzzy logic with engineering applications, T.J Ross ,
Second edition ,Wiley India (Chapter 3: Fuzzy Logic of
course of UEI-401)
3. Artificial Intelligence, E . Rich , K . Knight ,
TMH(Chapter1:Overview of AI of the course of UEI-401)
4. Artificial Intelligence, D. Khemani , Mcgraw Hill
5. Introduction to Artificial Intelligence and Expert System,
D.W. Patterson , PHI
6. Google.com
Thanks