ARTIFICIAL NEURAL

NETWORKS
PERCEPTRON AND OTHER BASIC ELEMENTS
Neural Network

A Neural Network is an information processing

system which is;
Non-Algorithmic
Non-Digital
Parallel
Consists of very simple processes called
“NEURODES”
“Neurode” are analogs of Biological Neural Cells,
called Neurons
Neurodes

Neurode is a simple device/machine in its

construction
It receives input from other neurodes or stimuli
and transmits an output response
Definition : “Synapse” is the place where input
signal meets the neurodes
Neurodes are connected with the weighted links
Neurode receive different inputs but produce
only one kind of output response that is
transferred to the other neurode or outside world
Neuron Model
NN Architecture
Output Response

Outer Layer

Hidden Layer

Input Layer

Input Signal
Translation Operation
The mathematical expression that describes the translation of
input stimulus pattern to output response signal is called the
transfer function of the neurode.

How a Neurode Works

First Step:
First of all Neurode calculates the net weighted input coming
towards it using following simple summation,
n
I i  wij x j
j 1
Where Ii is the net input received by the ith neurode from a
total of n neurodes and xj is the incoming signal from Jth
neurode.
Once net weighted input is calculated the individual
information about the strength of the input signals is lost
Translation Operation
Second Step:
In this stage the net weighted input is converted into an
activation level.
This activation level simulates the level of excitement of a
biological neuron.
Mostly sigmoid functions are used as an activation function.
The most commonly used sigmoid function is
1
f (I )  I
1 e
Function should be monotonically increasing and should be
bounded by upper and lower limits.
Range of above function is from 0 to 1 as value of the net
weighted input changes from negative infinity to positive
infinity.
Translation Operation
Third Step:
Transfer function converts the neurodes activation level to an
output signal using following expression,

 f ( I ), if f ( I )  T 
yi  
 0, Otherwise 

Where T is threshold value

Transfer function used in a neurode is same for a particular

network
Some researchers uses hyperbolic tangent function which is
symmetric around zero
NN Working
Output Response
Each Neurode
sees a
different input Outer Layer
pattern,
resulting in
different
output
response Hidden Layer

Input Layer

Input Signal
Neural Network

NN can be of just one layer and may have several

layers
Number of layers depends on the problem to
solve and the designer/user but all networks work
on the principle of “spreading activation”
Note: NN are basically very crud form of their
biological counterparts i.e. neurons.
Learning and Training

Learning and Training is fundamental

to all Neural Networks
Learning is achieved by changing the
weights of the neurodes
Training is the procedure by which
the network learns where as
learning is the end result of training
Training is done by examples
therefore Training is an external
process and learning is an internal
process
Types of Training

There are differebnt types of training

procedures
Supervised Training
Network is provided with input and corresponding desired output patterns
It works by calculating the error between the actual output and desired output –
i.e. learning law
Unsupervised Training – Self Organization
No output pattern is provided for training purposes
Conclusion

A neural network consists of four main parts:

1. Processing units
2. Weighted interconnections between the various
processing units which determine how the
activation of one unit leads to input for another
unit.
3. An activation rule which acts on the set of input
signals at a unit to produce a new output signal, or
activation.
4. Optionally, a learning rule that specifies how to
adjust the weights for a given input/output pair.
Perceptron

Key Neural Network Problem is separating patterns into two categories

First ever neurode applied for such problem was proposed by Waren S.
McCulloch and Walter Pitts in 1943
Output
y

Neurode

w1 w2

Input Input
x1 x2
Perceptron

This simple neurode is still the basis of modern neural networks

Idea of this simple neurode is to calculate the weighted sum and compares it
with a threshold value T.
If weighted sum>T then output of the neurode y will be +1 and
If weighted sum<T then output of the neurode y will be -1

Note: McCulloch-Pitts neurode is virtually indistinguishable from the neurodes

used in perceptrons and adaline.
So, n
I  wi xi
i 1

  1, if I  T 
y  
  1, if I  T 
Perceptron
McCulloch and Pitts Neurode model is the basis of NN
Rosenlastt’s perceptron uses the same neurode and proposed a training
law for seperating patterns.

Rosenblatt’s training law is

wnew = wold + yx
Where,
b =+1, if the perceptron’s answer is correct, and
=-1, if the perceptron’s answer is wrong
y= perceptron’s answer
Perceptron Algorithm’s Pseudocode
Perceptron Training Algorithm
For each pattern in the training set
{
apply the next pattern to the perceptron
record the perceptron’s response (either +1 or -1)
if the perceptron’s answer is correct
{
and the answer was +1, then
the new weight vector = the old weight vector + the input pattern vector
and the answer was -1, then
the new weight vector = the old weight vector - the input pattern vector
}
if the perceptron’s answer is incorrect
{
and the answer was +1, then
the new weight vector = the old weight vector - the input pattern vector
and the answer was -1, then
the new weight vector = the old weight vector + the input pattern vector
}

} /* end for each pattern in the training set */

Classification using Vector
Training
Classify the two data categories A &
B.
There are two data points of A
Category with +1 output, and two
data points with B Category with -1
output on the next slide
Take Initial Weight Vector as (-0.6,
0.8)
Threshold is assumed to be 0
Apply A1, B1, A2 and B2 to the
perceptron and check if it can
Classification using Vector
Training
Classification using Vector
Training

Weight Vectors
Initial W0 = (-
0.6, 0.8)
After A1, W1 = (-0.3,1.5)
After B1, W2 = (0.3, 1.2)
After A2, W3 = (2.0, 1.5)
After B2, W4 = (2.2, 2.3)
Classification using Vector
Training

[email protected] © M. Shahbaz – 2016

Observations

What is happening to the weight

vector?
What will happen if we introduce some
more data points?
Do you think perceptron can classify
all types of data points?
Is there any thing we can do to reduce
the length of the weight vector?
Write a simple routine to simulate
perceptron (keeping above
implications in mind)
[email protected] © M. Shahbaz – 2016
 Adaline

Adaline is the first ever practical supervised learning scheme

Proposed by Bernard Widrow and Ted Hoff in 1960
Adaline is a kind of perceptron
Adaline = ADAptive LINear Element

Characteristics:
Bipolar Output of +1 (say category A ) if weighted input is greater
than 0 &
-1 (say category B ) if weighted input is less than or
equal to 0

n
Input can be calculated as,
I w x
i 1
i i

[email protected] © M. Shahbaz – 2016

Adaline

x x Desired
1 2 Answer

w1 w1

Comparer
Weight
Update
Info

Category (+1 or -
1)

[email protected] © M. Shahbaz – 2016

Widrow-Hoff Law

Error = Desired output – Actual output

Error = +2, 0 , -2
old Ex
w w  2
Where
x
 =learning constant varies from 0 to 1
E = Error (Desired output – Actual output )
x = input vector

Above weight change law is called widrow-Hoff rule, LMS (Least

Mean Squared) Rule and most commonly Delta Rule

Adaline Pseudocode is here

[email protected] © M. Shahbaz – 2016

Pseudocode for the Adaline
For (each pattern) in the training set do
{
apply the pattern to the adaline
compute the adaline’s net input, I
if( I > 0) output +1; else output -1
compute error, E, as desired output – actual output
if (E ? 0) do
{
for (this pattern) back thru all previously trained patterns, do
/*if adjust the weights for this pattern, must reconfirm that all previous
patterns are still ok*/
{
apply the pattern to the adaline
compute the adaline’s net input, I
if ( I > 0 ) output +1; else output -1
computer error, E, as desired output – actual output
while (E ? 0) do
/*adjust the weights using delta rule till this pattern is correct*/
{
compute square of length of input vector (L = x12+x22+…)
for each element of weight vector, wi do
{
change in wi =  Exi / L
wi = wi + <change in wi >
}/*end of each element of weight vector*/

compute the adaline’s net input, I, for this same input vector (to see
if done)
if ( I > 0 ) output +1; else output -1
compute error, E, as above
}/* end while error not equal to zero */
}/* end of all previously trained patterns */
}/* end of original error was not zero */
}/* end of all patterns in training set */
[email protected] © M. Shahbaz – 2016
Adalin
e
If the new pattern cause a weight change
then all the previous patterns should be
checked again for their correct
classification.

[email protected] © M. Shahbaz – 2016

Classification Example using
Adaline
Data Points Class
A1 = ( 0.3, 0.7) +1 Assume Learning
constant  = 0.5
B1 = (-0.6, 0.3) -1
A2 = ( 0.4, 0.9) +1
B2 = (-0.4, -0.2) -1
A3 = ( 0.5, 0.5) +1
B3 = ( 0.3, -0.4) -1
A4 = ( 0.7, 0.3) +1
B4 = (-0.2, -0.8) -1

Weight Vectors

W0 = (-0.6, 0.8)

[email protected] © M. Shahbaz – 2016

[email protected] © M. Shahbaz – 2016
Classification Example using
Adaline
Apply A1:
I = -0.18 + 0.56 = 0.3
Since I > 0 therefore output = +1
Error = 1 – 1 = 0
No weight change is required

Apply B1:
I = 0.36+ 0.24 = 0.60
Since I > 0, therefore output = +1
Error = -1 -1 = -2
Weight needs update
Square of input pattern B1 = 0.36 + 0.09 = 0.45
old Ex
Change in w1 = (0.5)(-2)(-0.6) / (0.45) = 1.3 w w  2
Change in w2 = (0.5)(-2)(0.3) / (0.45) = -0.7 x
[email protected] © M. Shahbaz – 2016
Classification Example using
Adaline
Delta Vector = (1.3, 0.7)
New weight Vector w1 = (0.7, 0.1)

Confirm this new weight vector correctly classify Pattern B1

I = -0.42 + 0.03 = -0.3
Since I < 0 therefore output = -1
E = -1 – (-1) = 0

Since weight is changed therefore check all the previous patterns if they are
still correctly classifiable with the new weight vector
Apply Pattern A1
I = 0.21+ 0.07 = 0.28
Since I > 0 therefore output = +1
E=1–1=0
[email protected] © M. Shahbaz – 2016
Delta Rule

A
2
A1
A3
B1

B2
B3

[email protected] © M. Shahbaz – 2006

Adaline Example
W3 = (0.8, 1.1)
Train the Adaline on another point B5(0.6, -.02)

[email protected] © M. Shahbaz – 2006

Delta Rule

A2
A1
A3
B1
A4

B2 B5
B3

B4

[email protected] © M. Shahbaz – 2006

Delta Rule
Net input of Adaline
I = w1x1+w2x2

Dot Product in Cartesian Coordinate System

w  x  w * x cos 
Where  is the angle between input vector and
weight vector

Sign of the above expression depends on angle

[email protected] © M. Shahbaz – 2006

Delta Rule

S A

T C
 will decide the sign as length of the vector will always be
positive
Cos() = Positive when = 90
[email protected] © M. Shahbaz – 2006
Delta Rule
Dot Product of two vectors will be positive if the angle between them
is less than 90o.

Adaline returns +1 if angle between weight and input vector is less

than 90o
Adaline returns -1 if angle between weight and input vector is
greater than 90o
x
w

Dot Product = length of projection of one vector on other

[email protected] © M. Shahbaz – 2006
Delta Rule

+1
output

-1 output

Input patters in the shaded area cause weight vector to give +1

output
Input patters in the unshaded area cause weight vector to give -1
output
[email protected] © M. Shahbaz – 2006
Delta Rule

A2
A1
A3
B1 +1
A4
output
B2 B5

-1 outputB3
B4

[email protected] © M. Shahbaz – 2006