Chapter 1 Neurophysiology and

Learning Algorithms
Dr. Gaur Sanjay B.C.
Introduction
• The human brain consists of an estimated 1011 (100 billion)
nerve cells or neurons, an example of which is shown in
Figure 1.1.

Figure 1.1 Essential components of a neuron shown in stylized form

 Intro…
• Neurons communicate via electrical signals that are short-lived
impulses or "spikes" in the voltage of the cell wall or
membrane.

Figure: Action-potential propagation.

Intro…
• The inter-neuron connections are mediated by
electrochemical junctions called synapses, which are located
on branches of the cell referred to as dendrites.

Fig. 1.2 Synapses and dendrites.

Intro…
• Each neuron typically receives many thousands of connections
from other neurons and is therefore constantly receiving a
multitude of incoming signals, which eventually reach the cell
body.
• Here, they are integrated or summed together in some way.
• If the resulting signal exceeds some threshold then the neuron
will "fire" or generate a voltage impulse in response.
• This is then transmitted to other neurons via a branching fibre
known as the axon.

Fig. 1.3 Axon.

Biological model V/S ANN

Fig. 1.4 Equivalent ANN model.

 Artificial neurons: the TLU
• Artificial Neuron is known as a Threshold Logic Unit (TLU)
and was originally proposed by McCulloch and Pitts
(McCulloch & Pitts 1943). It is highly simplified
computational model.

Figure 1.5 Simple artificial neuron Figure 1.6 Activation Function

 Assumptions (TLU)
Their "neurons" operated under the following assumptions:
1. TLUs are binary devices (Vi = [0,1])
2. Each neuron has a fixed threshold, theta
3. The neuron receives inputs from excitatory synapses, all
having identical weights. (It receive multiple inputs from
the same source, so the excitatory weights are effectively
positive integers.)
4. Inhibitory inputs have an absolute veto power over any
excitatory inputs. They decide alone output of neuron.
5. Excitatory connection have positive weight and inhibitory
connection have negative weight.
 Assumptions
6. At each time step the neurons are simultaneously
(synchronously) updated by summing the weighted
excitatory inputs.
7. and setting the output (y)
1 if a  
y=
0 if a  
if the neuron receives no inhibitory input.
 Inhibitory and Excitatory
• ‘excite’ and ‘inhibit’ means whether the target cell is made
more (excitation) or less (inhibition)

• If output is governed by one input or it depends on only one

input irrespective of what are the other inputs, it is called as
inhibitory.
x y output
0 0 0
0 1 1
1 0 0
1 1 1
Two-input patterns in pattern space

Figure 1.7 Two-input patterns in pattern space

The output
y=w1*x1+w2*x2
Where x1and x2 are input of neuron (shown on both axis)
The linear separable two variable
classes

Figure 1.8: Linearly separable, two variable

classes
Nonlinear plane required to separate
both classes

Figure 1.9: Not Linearly separable

 Example: The linear separation of
classes OR GATE
• Truth table of Two input OR GATE

x1 x2 Output
0 0 0
0 1 1
1 0 1
1 1 1

❑ The output y is =1, if at least one i/p is =1;

v = w1 x1 + w2 x2
1. x1 = 0, x2 = 0, w1 = 1, w2 = 1
v = 1 0 + 1 0 = 0
2. x1 = 0, x2 = 1, w1 = 1, w2 = 1
v = 1 0 + 1 1 = 1
3. x1 = 1, x2 = 0, w1 = 1, w2 = 1
v = 1 1 + 1 0 = 1
4. x1 = 1, x2 = 1, w1 = 1, w2 = 1
v = 1 1 + 1 1 = 2
What we get through above What expected Output y (OR)
calculations v
x1 x2 Output x1 x2 Output
0 0 0 0 0 0
0 1 1 0 1 1
1 0 1 1 0 1
1 1 2 1 1 1
1 if v  1
y=
0 if v  1
Threshold to be used to
convert v into y
Example: The linear separation of
classes OR GATE

Decision Line
x1+x2=θ=1

Figure 1.10 Decision line for OR gate

❑To make the decision similar to truth table selection

of θ is to be =1. the line x1+x2=θ=1, can separate
two patterns as follows:
1 if v  1
y=
0 if v  1
❑ As shown in figure output is one if y>=1;
❑ The pattern is linearly separable by the line x1+x2=θ=1
❑ First pattern (y=0) is below the above line and other one
is either on the line or above the line. Hence, this problem
is linearly separable.
Example : Two input AND GATE
• Truth table

x1 x2 Output
0 0 0
0 1 0
1 0 0
1 1 1
 AND GATE…
Decision Line
x1+x2=θ=2

1. v = 1 0 + 1 0 = 0
2. v = 1 0 + 11 = 1
3. v = 11 + 1 0 = 1 Figure 1.11 : Decision line for OR gate

4. v = 11 + 11 = 2

To make AND gate like decision, activation function

should be θ=2
AND GATE…

❑ The output y is =1, if both i/p’s are =1;

❑ The decision line x1+x2=θ=2 can separate two patterns as
follows:
1 if v  2
y=
0 if v  2
 Example : Tautology (Always On)
• Truth table

x1 x2 Output
0 0 1
0 1 1
1 0 1
1 1 1
 Tautology …

Decision Line
x1+x2=θ=2
1. v = 1 0 + 1 0 = 0
2. v = 1 0 + 11 = 1
3. v = 11 + 1 0 = 1 Figure : Decision line for OR gate

4. v = 11 + 11 = 2

To make AND gate like decision, activation function

should be θ=0
Tautology …

❑ The output y is =1, if i/p’s is 0 or 1;

❑ The decision line x1+x2=θ=0 can separate two patterns as
follows:
1 if v  0
y=
0 if v  0
Example: Generate ANDNOT function
using McCulloch-Pitt Neuron
• Truth table

x1 x2 Output • Output= (x1 ANDNOT x2)

0 0 0
0 1 0
1 0 1
1 1 0
v = w1 x1 + w2 x2
1. x1 = 0, x2 = 0, w1 = 1, w2 = −1
v = 1 0 − 1 0 = 0
2. x1 = 0, x2 = 1, w1 = 1, w2 = −1
v = 1 0 − 11 = −1
3. x1 = 1, x2 = 0, w1 = 1, w2 = −1
v = 1 1 − 1 0 = 1
4. x1 = 1, x2 = 1, w1 = 1, w2 = −1
v = 1 1 − 1 1 = 0
What we get through above What is expected Output y
calculations v (ANDNOT)
x1 x2 Output x1 x2 Output
0 0 0 0 0 0
0 1 -1 0 1 0
1 0 1 1 0 1
1 1 0 1 1 0
1 if v  1
y=
0 if v  1
Threshold to be used to
convert v into y
 Figure : Decision line for XOR gate

❑ The output y is =1, if x1 is =0 or x2 is=1;

❑ The decision line x1-x2=θ=1 can separate two patterns as
follows:
1 if v  1
y=
0 if v  1
• There is no way to arrange the position of a line x1+x2=θ, so
that the correct two points lies on either side of the line. (try
it).
• The simple linear threshold unit cannot correctly perform the
XOR problem.
 Example : XOR Problem
• Truth table

x1 x2 Output • Output= (x1 ANDNOT x2)

0 0 0 OR
0 1 1 (x2 ANDNOT x1)
1 0 1
1 1 0
XOR …

w1 = 1, w2 = 1
1. v = 1 0 + 1 0 = 0
2. v = 1 0 + 1 1 = 1 Figure : Decision line for XOR gate
3. v = 1 1 + 1 0 = 1
4. v = 1 1 + 1 1 = 2
To make XOR gate like decision, activation function should
be θ=0.5 and 1.5, it is not possible to solve it by single layer
perception.
• There is no way to arrange the position of a line x1+x2=θ so
that the correct two points lies on either side of the line. (try
it).
• The simple linear threshold unit cannot correctly perform the
XOR problem.
 Solution of XOR problem
• The graph suggests that we could partition the space correctly
if we had three regions.
• One region belong to one output class, and the other two
belong to the other second output class.
• The fig. below show a network of linear threshold that
performs the proper partitioning.

Line2
x1+x2=θ=1.5

Line1
x1+x2=θ=0.5
• The addition of two hidden layer, or middle layer unit, give the
network, flexibility to solve the problem.
• In fact, the existence of this hidden layer gives us ability to
construct networks that can solve complex problem.
 Approach for XOR (S.N. Deepa)
Example: Realize the XOR function using McCulloch Pitt Neuron.
We know that one layer neuron alone is not able to predict the
value of threshold for the neuron to fire, hence, another layer is
introduced.
• First part

• Output= (x1 ANDNOT x2)

v = w1 x1 + w2 x2
1. x1 = 0, x2 = 0, w1 = 1, w2 = −1
v = 1 0 − 1 0 = 0
2. x1 = 0, x2 = 1, w1 = 1, w2 = −1
v = 1 0 − 11 = −1
3. x1 = 1, x2 = 0, w1 = 1, w2 = −1
v = 1 1 − 1 0 = 1
4. x1 = 1, x2 = 1, w1 = 1, w2 = −1
v = 1 1 − 1 1 = 0
What we get through above What expected Output y
calculations v (ANDNOT)
x1 x2 Output v x1 x2 Output z1
0 0 0 0 0 0
0 1 -1 0 1 0
1 0 1 1 0 1
1 1 0 1 1 0
1 if v  1
y=
0 if v  1
Threshold to be used to
convert v into y
• Second part

• Output= (x2 ANDNOT x1)

v = w1 x1 + w2 x2
1. x1 = 0, x2 = 0, w1 = −1, w2 = 1
v = −1 0 + 1 0 = 0
2. x1 = 0, x2 = 1, w1 = −1, w2 = 1
v = −1 0 + 11 = 1
3. x1 = 1, x2 = 0, w1 = −1, w2 = 1
v = −11 + 1 0 = −1
4. x1 = 1, x2 = 1, w1 = −1, w2 = 1
v = −11 + 11 = 0
What we get through above What expected Output y
calculations v (ANDNOT)
x1 x2 Output v x1 x2 Output z2
0 0 0 0 0 0
0 1 1 0 1 1
1 0 -1 1 0 0
1 1 0 1 1 0
1 if v  1
z2 = 
0 if v  1
Threshold to be used to
convert v into z2
• Third part

• Output= (z1 OR z2)

x1 x2 Output z1 x1 x2 Output z2
0 0 0 0 0 0
0 1 0 0 1 1
1 0 1 1 0 0
1 1 0 1 1 0

Patterns for input to third section

x1 x2 Output Output
z1 z2
0 0 0 0
0 1 0 1
1 0 1 0
1 1 0 0
 Output Output
z1 z2
0 0
0 1
1 0
0 0

v = w1 z1 + w2 z 2
1. z1 = 0, z 2 = 0, w1 = 1, w2 = 1
v = 1 0 + 1 0 = 0
2. z1 = 0, z 2 = 1, w1 = 1, w2 = 1
v = 1 0 + 1 1 = 1
3. z1 = 1, z 2 = 0, w1 = 1, w2 = 1
v = 1 1 + 1 0 = 1
What we get through above What is expected Output y (OR)
calculations v
z1 z2 Output v z1 z2 Output y
0 0 0 0 0 0
0 1 1 0 1 1
1 0 1 1 0 1
0 0 0 0 0 0
1 if v  1
y=
0 if v  1
Threshold to be used to
convert v into z2
Example: Consider the NN (McCulloch-Pitt) as shown in figure.
Each neurons (other than the input Neurons N1 and N2) has a
threshold of 2.
a) Define the response of Neuron N5 at time t in terms of activation
function input Neurons, N1 and N2 at the appropriate time.
b) Show that the activation of each Neuron that results from an
input signal of N1=1 and N2=0 at t=0.

Wrong Solution in Sivannadan,

Sumathi and Deepa
To define the response of neuron N5 at time t
(i). Response of N3: N 3−in = w1 N1 + w2 N 2
Given , w1=1, w2=1 and θ=2
N1 + N 2
N1 N2 N3_in N3 1 if v  2
N3 = 
0 0 0 0 0 if v  2
0 1 1 0
1 0 1 0
1 1 2 1

(ii). Response of N4:

Given , w1= 1, w2= 1 and θ=2 N 4−in = w1 N1 + w2 N 2
N1 N2 N4_in N4
= N1 + N 2
0 0 0 0 1 if v  2
N4 = 
0 if v  2
0 1 1 0
1 0 1 0
1 1 2 1
 (iii). Response of N5:
Given , w1=2, w2=2 and θ=2 N 5−in = w1 N1 + w2 N 2
N3 N4 N5_in N5 2 N1 + 2 N 2
1 if v  2
0 0 0 0
N5 = 
0 0 0 0 0 if v  2
0 0 0 0
1 1 4 1

(a). Response of N5 at time =t:

Finally,
N 5 = N1 AND N 2
At time, t
N 5 (t ) = N1 (t − 2 ) AND N 2 (t − 2 )
b). Show that the activation of each Neuron that results from an input signal of
N1=1 and N2=0 at t=0.
Step-I:

Step-II:

N1 N2 N3_in N3

0 0 0 0
0 1 1 0
1 0 1 0
1 1 2 1
Step-III:

N1 N2 N4_in N4

0 0 0 0
0 1 1 0
1 0 1 0
1 1 2 1

N3 N4 N5_in N5
Hence, N5=N3+N4
=0+0=0 0 0 0 0
1 1 4 1
 What are the Learning Rules in Neural Network?

Learning rule or Learning process is a method or a

mathematical logic, to improve the performance of Artificial
Neural Network’s.
• Learning rules updates the weights and bias levels of a
network, when a network simulates in a specific data
environment.
• Applying learning rule is an iterative process. It helps a neural
network to learn from the existing conditions and improve its
performance. (Rote learning)
 What are the types of Learning
Rules in Neural Network?
Let us see different learning rules in the Neural network:
❑ Hebbian learning rule
❑ Perceptron learning rule
❑ Delta learning
❑ Correlation learning rule
❑ Competitive Learning Rule
❑ Boltzmann Learning
 Hebbian Learning Rule
The Hebbian rule was the first learning rule. In 1949 Donald
Hebb developed it as learning algorithm of the unsupervised
neural network.
We can use it to identify how to improve the weights of nodes of
a network.
The Hebb learning rule assumes that – 1). If two neighbor
neurons activated and deactivated at the same time. Then the
weight connecting these neurons should increase.
2). For neurons operating in the opposite phase, the
weight between them should decrease.
3). If there is no signal correlation, the weight should not
change.
• Two neurons are active simultaneously, the strength of the
connection between them should increase.

• When inputs of both the nodes xi and oj are either positive or

negative, then a strong positive weight exists between the
nodes.
• If the input of a node first xi is positive and second input oj is
negative, a strong negative weight exists between the nodes.
 Algorithm…
• At the start, values of all weights are set to zero. This
learning rule can be used for both soft- and hard-
activation functions.
• The Hebbian learning rule (activity product rule)
describes the formula as follows:
Δ𝑤𝑖𝑗 = 𝑥𝑖 ∗ 𝑜𝑗
𝑥𝑖 = signal on the ith input
𝑜𝑗 = is the output signal
Limitation : not useful for binary data classification (data should be Bipolar)
 Perceptron Learning Rule
• Frank Rosenblatt (1958,1959,1962) has given this learning
rule, which is more powerful than Hebbian rule.
• For example: in supervised learning, the network starts its
learning by assigning a random value to each weight.
• The network is trained by set of input and output dataset. The
network then compares the calculated output value with the
expected value. It calculates an error function ∈, as follows:
wij = c xi  o j
c = is the learning constant
xi = signal on the i th input
o j = is the output signal
Delta Learning Rule
• Developed by Widrow and Hoff (1960), the delta rule, is one of
the most common learning rules. (supervised learning).
• This rule states that the modification in sympatric weight of a
node is equal to the multiplication of error and the input.
• In Mathematical form the delta rule is as follows:

w =  (t − y )xi where is learning rate

For a given input vector xi, compare the output vector y with the
correct answer (= t).
Then, the difference (t-y) is zero, it means no learning takes place;
otherwise, adjusts its weights to reduce this difference.
 Correlation Learning Rule
• The correlation learning rule based on a similar principle as
the Hebbian learning rule.
• It assumes that weights between responding neurons (with
same reaction) should be more positive, and weights between
neurons with opposite reaction should be more negative.
• Differing to the Hebbian rule, the correlation rule is the
supervised learning. Instead of the response, yj, the desired
response, dj, uses for the weight-change calculation.
• In Mathematical form the correlation learning rule is as
follows: w = x d
ij i j
 Competitive Learning Rule
• The competitive learning rule is unsupervised in nature and as
the name says, is based on the principle of competition
amongst the nodes. That is why it is also known as the
‘Winner takes it all’ rule.
• In this rule, all the output nodes represent the input pattern and
the best one, meaning, the one having the most number of
outputs is the winner. This winner node is then assigned the
value 1 and the others that lose, remain at 0. Naturally, only
one neuron remains active at once.
•∆wij is the changes in weight between the ith input neuron
Δ𝑤𝑖𝑗 = 𝜂(𝑥𝑖 −𝑤𝑖𝑗 )
and jth output neuron
•η is the learning rate, xi is the input vector
•wij is the weight between the ith input neuron and jth
output neuron
 Boltzmann Learning
• The learning is stochastic learning (having a random
probability distribution or pattern).
• This learning is useful for Recurrent structure and they work
in binary form.
• This learning is characterized by an energy function, E, as
−1
E=   wij xi x j ( for i  j )
2 i j
i ≠ j, means no neuron has self feedback. Feedback is
taken from randomly selected neurons.
• The neurons of this learning are divided into two groups:
Visible and Hidden
 Visible and Hidden Neurons

• Visible Neurons: there is interface between network and the

environment in which it operates. They are also clamped to
onto specific states determined by the environment.
• Hidden Neurons: they operates independent of the
environment. And there is free running condition, in which all
the neurons are allowed to operate freely.