fo

.in
rs
de
ea
yr
Back Propagation Network : Soft Computing Course Lecture 15 – 20, notes, slides

.m
w
w
www.myreaders.info/ , RC Chakraborty, e-mail [email protected] , Aug. 10, 2010
,w
ty

http://www.myreaders.info/html/soft_computing.html
or
ab

www.myreaders.info
kr
ha
C
C
R

Return to Website

Back Propagation Network

Soft Computing
Back-Propagation Network, topics : Background, what is back-prop
network ? learning AND function, simple learning machines - Error
measure , Perceptron learning rule, Hidden Layer, XOR problem.
Back-Propagation Learning : learning by example, multi-layer
feed-forward back-propagation network, computation in input,
hidden and output layers, error calculation. Back-propagation
algorithm for training network - basic loop structure, step-by-step
procedure, numerical example.
 fo
.in
rs
de
ea
Back Propagation Network

yr
.m
w
w
,w

Soft Computing
ty
or
ab
kr

Topics
ha
C
C

(Lectures 15, 16, 17, 18, 19, 20 6 hours) Slides

R

1. Back-Propagation Network - Background 03-11

What is back-prop network ?; Learning : AND function; Simple

learning machines - Error measure , Perceptron learning rule ; Hidden
Layer , XOR problem.

2. Back-Propagation Learning - learning by example 12-16

Multi-layer Feed-forward Back-propagation network; Computation of

Input, Hidden and Output layers ; Calculation of Error.

3. Back-Propagation Algorithm 17-32

Algorithm for training Network - Basic loop structure, Step-by-step

procedure; Example: Training Back-prop network, Numerical example.

4. References 33

02
 fo
.in
rs
de
ea
yr
Back-Propagation Network
.m
w
w
,w
ty

What is BPN ?
or
ab
kr
ha

• A single-layer neural network has many restrictions. This network

C
C
R

can accomplish very limited classes of tasks.

Minsky and Papert (1969) showed that a two layer feed-forward

network can overcome many restrictions, but they did not present
a solution to the problem as "how to adjust the weights from input
to hidden layer" ?

• An answer to this question was presented by Rumelhart, Hinton

and Williams in 1986. The central idea behind this solution is that
the errors for the units of the hidden layer are determined by
back-propagating the errors of the units of the output layer.

This method is often called the Back-propagation learning rule.

Back-propagation can also be considered as a generalization of the

delta rule for non-linear activation functions and multi-layer networks.

• Back-propagation is a systematic method of training multi-layer

artificial neural networks.

03
 fo
.in
rs
de
SC - NN - BPN – Background

ea
1. Back-Propagation Network – Background

yr
.m
w
w
Real world is faced with a situations where data is incomplete or noisy.
,w
ty
or

To make reasonable predictions about what is missing from the information

ab
kr

available is a difficult task when there is no a good theory available

ha
C
C

that may to help reconstruct the missing data. It is in such situations the
R

Back-propagation (Back-Prop) networks may provide some answers.

• A BackProp network consists of at least three layers of units :

- an input layer,

- at least one intermediate hidden layer, and

- an output layer.

• Typically, units are connected in a feed-forward fashion with input

units fully connected to units in the hidden layer and hidden units
fully connected to units in the output layer.

• When a BackProp network is cycled, an input pattern is propagated

forward to the output units through the intervening input-to-hidden
and hidden-to-output weights.

• The output of a BackProp network is interpreted as a classification

decision.

[Continued in next slide]

04
 fo
.in
rs
de
SC - NN - BPN –Background

ea
[Continued from previous slide]

yr
.m
w • With BackProp networks, learning occurs during a training phase.
w
,w
ty

The steps followed during learning are :

or
ab
kr
ha

− each input pattern in a training set is applied to the input units and
C
C
R

then propagated forward.

− the pattern of activation arriving at the output layer is compared

with the correct (associated) output pattern to calculate an error signal.

− the error signal for each such target output pattern is then
back-propagated from the outputs to the inputs in order to
appropriately adjust the weights in each layer of the network.

− after a BackProp network has learned the correct classification for

a set of inputs, it can be tested on a second set of inputs to see

how well it classifies untrained patterns.

• An important consideration in applying BackProp learning is how

well the network generalizes.
05
 fo
.in
rs
de
SC - NN - BPN – Background

ea
1.1 Learning :

yr
.m
w
w
AND function
,w
ty
or

Implementation of AND function in the neural network.

ab
kr
ha
C
C

AND W1
R

X1 X2 Y Input I1
0 0 0 A
0 1 0 Output O
W2 C
1 0 0
1 1 1 Input I2
B

AND function implementation

− there are 4 inequalities in the AND function and they must be

satisfied.
w10 + w2 0 < θ , w1 0 + w2 1 < θ ,
w11 + w2 0 < θ , w1 1 + w2 1 > θ
− one possible solution :

if both weights are set to 1 and the threshold is set to 1.5, then
(1)(0) + (1)(0) < 1.5 assign 0 , (1)(0) + (1)(1) < 1.5 assign 0
(1)(1) + (1)(0) < 1.5 assign 0 , (1)(1) + (1)(1) > 1.5 assign 1

Although it is straightforward to explicitly calculate a solution to the

AND function problem, but the question is "how the network can
learn such a solution". That is, given random values for the weights
can we define an incremental procedure which will cover a set of
weights which implements AND function.

06
 fo
.in
rs
de
SC - NN - BPN – Background

ea
• Example 1

yr
.m
w
w
AND Problem
,w
ty
or
ab

Consider a simple neural network made up of two inputs connected

kr
ha

to a single output unit.

C
C
R

AND W1
X1 X2 Y Input I1
0 0 0 A
0 1 0 Output O
W2 C
1 0 0
1 1 1 Input I2
B

Fig A simple two-layer network applied to the AND problem

− the output of the network is determined by calculating a weighted

sum of its two inputs and comparing this value with a threshold θ.
− if the net input (net) is greater than the threshold, then the output

is 1, else it is 0.
− mathematically, the computation performed by the output unit is

net = w1 I1 + w2 I2 if net > θ then O = 1, otherwise O = 0.

• Example 2
Marital status and occupation
In the above example 1
− the input characteristics may be : marital Status (single or married)

and their occupation (pusher or bookie).

− this information is presented to the network as a 2-D binary input vector

where 1st element indicates marital status (single = 0, married = 1)

and 2nd element indicates occupation ( pusher = 0, bookie = 1 ).
− the output, comprise "class 0" and "class 1".
− by applying the AND operator to the inputs, we classify an
individual as a member of the "class 0" only if they are both
married and a bookie; that is the output is 1 only when both of the
inputs are 1.
07
 fo
.in
rs
de
SC - NN - BPN – Background

ea
1.2 Simple Learning Machines

yr
.m
w
w
Rosenblatt (late 1950's) proposed learning networks called Perceptron.
,w
ty
or

The task was to discover a set of connection weights which correctly

ab
kr

classified a set of binary input vectors. The basic architecture of the

ha
C
C

perceptron is similar to the simple AND network in the previous example.

R

A perceptron consists of a set of input units and a single output unit.

As in the AND network, the output of the perceptron is calculated
n
by comparing the net input net = Σ wi Ii and a threshold θ.
i=1
If the net input is greater than the threshold θ , then the output unit is
turned on , otherwise it is turned off.

To address the learning question, Rosenblatt solved two problems.

− first, defined a cost function which measured error.

− second, defined a procedure or a rule which reduced that error by

appropriately adjusting each of the weights in the network.

However, the procedure (or learning rule) required to assesses the
relative contribution of each weight to the total error.

The learning rule that Roseblatt developed, is based on determining

the difference between the actual output of the network with the
target output (0 or 1), called "error measure" which is explained
in the next slide.

08
 fo
.in
rs
de
SC - NN - BPN – Background

ea
• Error Measure ( learning rule )

yr
.m
w Mentioned in the previous slide, the error measure is the difference
w
,w
ty

between actual output of the network with the target output (0 or 1).
or
ab
kr
ha

― If the input vector is correctly classified (i.e., zero error),

C
C
R

then the weights are left unchanged, and

the next input vector is presented.

― If the input vector is incorrectly classified (i.e., not zero error),

then there are two cases to consider :

Case 1 : If the output unit is 1 but need to be 0 then

◊ the threshold is incremented by 1 (to make it less likely that the

output unit would be turned on if the same input vector was
presented again).
◊ If the input Ii is 0, then the corresponding weight Wi is left
unchanged.
◊ If the input Ii is 1, then the corresponding weight Wi is

decreased by 1.
Case 2 : If output unit is 0 but need to be 1 then the opposite
changes are made.
09
 fo
.in
rs
de
SC - NN – BPN – Background

ea
• Perceptron Learning Rule : Equations

yr
.m
w
w
The perceptron learning rules are govern by two equations,
,w
ty
or

− one that defines the change in the threshold and

ab
kr

− the other that defines change in the weights,

ha
C
C
R

The change in the threshold is given by

∆ θ = - (tp - op) = - dp

where p specifies the presented input pattern,

op actual output of the input pattern Ipi
tp specifies the correct classification of the input pattern ie target,
dp is the difference between the target and actual outputs.

The change in the weights are given by

∆ wi = (tp - op) Ipi = - dp Ipi

10
 fo
.in
rs
de
SC - NN - BPN – Background

ea
1.3 Hidden Layer

yr
.m
w
w
Back-propagation is simply a way to determine the error values in
,w
ty
or

hidden layers. This needs be done in order to update the weights.

ab
kr
ha
C

The best example to explain where back-propagation can be used is

C
R

the XOR problem.

Consider a simple graph shown below.
− all points on the right side of the line are +ve, therefore the output of

the neuron should be +ve.

− all points on the left side of the line are –ve, therefore the output of
the neuron should be –ve.

With this graph, one can make a simple table of

X2
+ + inputs and outputs as shown below.
― + + +
― ― + + + AND Training a network to operate
― ― + + X1 X2 Y
―
― X1
1 1 1 as an AND switch can be
― ― ― ― ― 1 0 0
― ― ― ― 0 1 0 done easily through only one
― 0 0 0
neuron (see previous slides)

But a XOR problem can't be solved using only one neuron.

If we want to train an XOR, we need 3 neurons, fully-connected in a
feed-forward network as shown below.

XOR X1
A
X1 X2 Y X2
1 1 0 C Y
1 0 1
0 1 1 X2
B
0 0 0 X1

11
 fo
.in
rs
de
SC - NN – Back Propagation Network

ea
2. Back Propagation Network

yr
.m
w
w
Learning By Example
,w
ty
or
ab

Consider the Multi-layer feed-forward back-propagation network below.

kr
ha

The subscripts I, H, O denotes input, hidden and output neurons.

C
C
R

th th
The weight of the arc between i input neuron to j hidden layer is Vij .

The weight of the arc between i th hidden neuron to j th out layer is Wij

II1 OI1 V11 IH1 OH1 W11 IO1 OO1

1 1 1
V21 W21
II2 OI2 IH2 OH2 IO2 OO2
2 2 2
Vl1 Wm1
IIℓ OIℓ IHm OHm IOn OOn
ℓ m n
Vij Wij
Input Layer Hidden Layer Output Layer
i - nodes m- nodes n - nodes

Fig Multi-layer feed-forward back-propagation network

The table below indicates an 'nset' of input and out put data.
It shows ℓ inputs and the corresponding n output data.
Table : 'nset' of input and output data
No Input Ouput
I1 I2 .... Iℓ O1 O2 .... On

1 0.3 0.4 .... 0.8 0.1 0.56 .... 0.82

2
:
nset

In this section, over a three layer network the computation in the input,
hidden and output layers are explained while the step-by-step
implementation of the BPN algorithm by solving an example is illustrated
in the next section.

12
 fo
.in
rs
de
SC - NN – Back Propagation Network

ea
2.1 Computation of Input, Hidden and Output Layers

yr
.m
w (Ref.Previous slide, Fig. Multi-layer feed-forward back-propagation network)
w
,w
ty
or
ab

• Input Layer Computation

kr
ha
C

Consider linear activation function.

C
R

If the output of the input layer is the input of the input layer and
the transfer function is 1, then
{ O }I = { I }I
ℓx1 ℓx1 (denotes matrix row, column size)
The hidden neurons are connected by synapses to the input neurons.
th
- Let Vij be the weight of the arc between i input neuron to
th
j hidden layer.
- The input to the hidden neuron is the weighted sum of the outputs

of the input neurons. Thus the equation

IHp = V1p OI1 + V2p OI2 + . . . . + V1p OIℓ where (p =1, 2, 3 . . , m)
denotes weight matrix or connectivity matrix between input neurons
and a hidden neurons as [ V ].
we can get an input to the hidden neuron as ℓ x m
T
{ I }H = [ V ] { O }I
mx1 mxℓ ℓx1 (denotes matrix row, column size)

13
 fo
.in
rs
de
SC - NN – Back Propagation Network

ea
• Hidden Layer Computation

yr
.m
w Shown below the pth neuron of the hidden layer. It has input from the
w
,w
ty

output of the input neurons layers. If we consider transfer function as

or
ab
kr

sigmoidal function then the output of the pth hidden neuron is given by
ha
C

1
C
R

OHp = -λ (IHP – θHP)

(1+e )

where OHp is the output of the pth hidden neuron,

IHp is the input of the pth hidden neuron, and
θHP is the threshold of the pth neuron;

Note : a non zero threshold neuron, is computationally equivalent to an input

that is always held at -1 and the non-zero threshold becomes the connecting
weight value as shown in Fig. below.
II1 OI1
1 –
–
II2 OI2 V1p 1
2 V2p
{ O }H = -λ (IHP – θHP)
(1+e )
II3 OI3 V3p
3 p –
Vℓp –
IIℓ OIℓ
ℓ
Note : the threshold is not treated as
θHP
shown in the Fig (left); the outputs of
O
IIO = -1 OIO = -1 the hidden neuron are given by the
above equation.
Fig. Example of Treating threshold
in hidden layer

Treating each component of the input of the hidden neuron separately,

we get the outputs of the hidden neuron as given by above equation .
The input to the output neuron is the weighted sum of the outputs of
the hidden neurons. Accordingly, Ioq the input to the qth output neuron
is given by the equation
Ioq = W1q OH1 + W2q OH2 + . . . . + Wmq OHm , where (q =1, 2, 3 . . , n)

It denotes weight matrix or connectivity matrix between hidden neurons

and output neurons as [ W ], we can get input to output neuron as
{ I }O = [ W] T { O }H
nx1 nxm mx1 (denotes matrix row, column size)
14
 fo
.in
rs
de
SC - NN – Back Propagation Network

ea
• Output Layer Computation

yr
.m
w
w
Shown below the qth neuron of the output layer. It has input from
,w
ty
or

the output of the hidden neurons layers.

ab
kr

If we consider transfer function as sigmoidal function then the output

ha
C
C

of the qth output neuron is given by

R

1
OOq = -λ (IOq – θOq)
(1+e )

where OOq is the output of the qth output neuron,

IOq is the input to the qth output neuron, and
θOq is the threshold of the qth neuron;

Note : A non zero threshold neuron, is computationally equivalent to

an input that is always held at -1 and the non-zero threshold becomes
the connecting weight value as shown in Fig. below.
Note : Here again the threshold may be tackled by considering
extra Oth neuron in the hidden layer with output of -1 and the threshold
value θOq becomes the connecting weight value as shown in Fig. below.

IH1 OH1 –
1
–
IH2 OH2 W1q 1
2 w2q { O }O = -λ (IOq – θOq)
(1+e )
IH3 OH3 W3q –
3 q –
Wmq OOq
IHm OHm
m
θOq Note : here again the threshold is not

O treated as shown in the Fig (left); the

IHO = -1 OHO = -1 Outputs of the output neurons given by
Fig. Example of Treating threshold the above equation.
in output layer
15
 fo
.in
rs
de
SC - NN – Back Propagation Network

ea
2.2 Calculation of Error

yr
.m
w
w
(refer the earlier slides - Fig. "Multi-layer feed-forward back-propagation network"
,w
ty
or

and a table indicating an 'nset' of input and out put data for the purpose of
ab
kr

training)
ha
C
C
R

th
Consider any r output neuron. For the target out value T, mentioned
in the table- 'nset' of input and output data" for the purpose of
training, calculate output O .
The error norm in output for the r th output neuron is
E1r = (1/2) e2r = (1/2) (T –O)2
where E1r is 1/2 of the second norm of the error er in the r th neuron
for the given training pattern.
e2r is the square of the error, considered to make it independent
of sign +ve or –ve , ie consider only the absolute value.

The Euclidean norm of error E1 for the first training pattern is given by
n
E1 = (1/2) Σ (Tor - Oor )
2
r=1
This error function is for one training pattern. If we use the same
technique for all the training pattern, we get
nset
E (V, W) = Σ E j (V, W, I)
r=1

where E is error function depends on m ( 1 + n) weights of [W] and [V].

All that is stated is an optimization problem solving, where the

objective or cost function is usually defined to be maximized or
minimized with respect to a set of parameters. In this case, the
network parameters that optimize the error function E over the 'nset'
nset nset
of pattern sets [I , t ] are synaptic weight values [ V ] and
[ W ] whose sizes are

[V] and [W]

ℓxm mxn

16
 fo
.in
rs
de
SC - NN - BPN – Algorithm

ea
3. Back-Propagation Algorithm

yr
.m
w
w
The benefits of hidden layer neurons have been explained. The hidden layer
,w
ty
or

allows ANN to develop its own internal representation of input-output

ab
kr

mapping. The complex internal representation capability allows the

ha
C
C

hierarchical network to learn any mapping and not just the linearly
R

separable ones.

The step-by-step algorithm for the training of Back-propagation network

is presented in next few slides. The network is the same , illustrated before,
has a three layer. The input layer is with ℓ nodes, the hidden layer with m
nodes and the output layer with n nodes. An example for training a
BPN with five training set have been shown for better understanding.

17
 fo
.in
rs
de
SC - NN - BPN – Algorithm

ea
3.1 Algorithm for Training Network

yr
.m
w
w
The basic algorithm loop structure, and the step by step procedure of
,w
ty
or

Back- propagation algorithm are illustrated in next few slides.

ab
kr
ha
C

• Basic algorithm loop structure

C
R

Initialize the weights

Repeat
For each training pattern
"Train on that pattern"
End
Until the error is acceptably low.

18
 fo
.in
rs
de
SC - NN - BPN – Algorithm

ea
• Back-Propagation Algorithm - Step-by-step procedure

yr
.m
w
w
,w

■ Step 1 :
ty
or
ab

Normalize the I/P and O/P with respect to their maximum values.
kr
ha
C

For each training pair, assume that in normalized form there are
C
R

ℓ inputs given by { I }I and

ℓx1
n outputs given by { O}O
nx1

■ Step 2 :

Assume that the number of neurons in the hidden layers lie

between 1 < m < 21
19
 fo
.in
rs
de
SC - NN - BPN – Algorithm

ea
■ Step 3 :

yr
.m
w
w
Let [ V ] represents the weights of synapses connecting input
,w
ty
or

neuron and hidden neuron

ab
kr
ha
C

Let [ W ] represents the weights of synapses connecting hidden

C
R

neuron and output neuron

Initialize the weights to small random values usually from -1 to +1;

0
[V] = [ random weights ]
0
[W] = [ random weights ]
0 0
[∆V] = [∆W] = [0]

For general problems λ can be assumed as 1 and threshold

value as 0.

20
 fo
.in
rs
de
SC - NN - BPN – Algorithm

ea
■ Step 4 :

yr
.m
w
w
For training data, we need to present one set of inputs and outputs.
,w
ty

Present the pattern as inputs to the input layer { I }I .

or
ab
kr

then by using linear activation function, the output of the input layer
ha
C
C

may be evaluated as
R

{ O }I = { I }I
ℓx1 ℓx1

■ Step 5 :

Compute the inputs to the hidden layers by multiplying corresponding

weights of synapses as

T
{ I }H = [ V] { O }I
mx1 mxℓ ℓx1

■ Step 6 :

Let the hidden layer units, evaluate the output using the
sigmoidal function as

–
–
1
{ O }H = - (IHi)
(1+e )

–
–
mx1

21
 fo
.in
rs
de
SC - NN - BPN – Algorithm

ea
■ Step 7 :

yr
.m
w
w
Compute the inputs to the output layers by multiplying corresponding
,w
ty

weights of synapses as
or
ab
kr
ha

T
C

{ I }O = [ W] { O }H
C
R

nx1 nxm mx1

■ Step 8 :

Let the output layer units, evaluate the output using sigmoidal
function as

–
–
1
{ O }O = - (IOj)
(1+e )

–
–

Note : This output is the network output

22
 fo
.in
rs
de
SC - NN - BPN – Algorithm

ea
■ Step 9 :

yr
.m
w
w
Calculate the error using the difference between the network output
,w
ty

th
and the desired output as for the j training set as
or
ab
kr
ha
C

√∑ (Tj - Ooj )2
C
R

EP = n

■ Step 10 :

Find a term { d } as

–
–

{d}= (Tk – OOk) OOk (1 – OOk )

–
–
nx1
23
 fo
.in
rs
de
SC - NN - BPN – Algorithm

ea
■ Step 11 :

yr
.m
w
w
Find [ Y ] matrix as
,w
ty
or

[ Y ] = { O }H 〈 d 〉
ab

mxn mx1 1xn

kr
ha
C

■ Step 12 :
C
R

t +1 t
Find [∆W] = α [∆W] + η[Y]
mxn mxn mxn

■ Step 13 :

Find {e} = [W] {d}

mx1 mxn nx1

–
–
(OHi) (1 – OHi )
{ d* } = ei
–
–
mx1 mx1
Find [ X ] matrix as
[X] = { O }I 〈 d* 〉 = { I }I 〈 d* 〉
1xm ℓx1 1xm ℓx1 1xm

24
 fo
.in
rs
de
SC - NN - BPN – Algorithm

ea
■ Step 14 :

yr
.m
t +1 t
w Find [∆V] = α [∆V] + η[X]
w
,w

1xm 1xm 1xm

ty
or
ab
kr

■ Step 15 :
ha
C

t +1 t t +1
C

Find [V] = [V ] + [∆V]

R

t +1 t t +1
[W] = [W ] + [∆W]

■ Step 16 :
Find error rate as
∑ Ep
error rate =
nset

■ Step 17 :
Repeat steps 4 to 16 until the convergence in the error rate is less
than the tolerance value

■ End of Algorithm

Note : The implementation of this algorithm, step-by-step 1 to 17,

assuming one example for training BackProp Network is illustrated in
the next section.

25
 fo
.in
rs
de
SC - NN - BPN – Algorithm

ea
3.2 Example : Training Back-Prop Network

yr
.m
w
w
• Problem :
,w
ty
or

Consider a typical problem where there are 5 training sets.

ab
kr
ha
C

Table : Training sets

C
R

S. No. Input Output

I1 I2 O
1 0.4 -0.7 0.1
2 0.3 -0.5 0.05
3 0.6 0.1 0.3
4 0.2 0.4 0.25
5 0.1 -0.2 0.12

In this problem,
- there are two inputs and one output.

- the values lie between -1 and +1 i.e., no need to normalize the values.

- assume two neurons in the hidden layers.

- the NN architecture is shown in the Fig. below.

0.4 0.1
0.2
0.4 -0.2 TO = 0.1

-0.7 -0.5
0.2

Input Hidden Output

layer layer layer

Fig. Multi layer feed forward neural network (MFNN) architecture

with data of the first training set

The solution to problem are stated step-by-step in the subsequent

slides.

26
 fo
.in
rs
de
SC - NN - BPN – Algorithm

ea
■ Step 1 : Input the first training set data (ref eq. of step 1)

yr
.m
w 0.4
w
,w

{ O }I = { I }I =
ty

-0.7
or

ℓx1 ℓx1
ab

2x1
kr

from training set s.no 1

ha
C
C

■ Step 2 : Initialize the weights as (ref eq. of step 3 & Fig)

R

0.1 0.4 0.2

0
[V] 0
= [W] =
-0.2 0.2 ; -0.5
2 x1
2x2

from fig initialization from fig initialization

■ Step 3 : Find { I }H = [ V] T { O }I as (ref eq. of step 5)

0.1 -0.2 0.4 0.18

{ I }H = =
-0.4 0.2 -0.7 0.02

Values from step 1 & 2

27
 fo
.in
rs
de
SC - NN - BPN – Algorithm

ea
■ Step 4 : (ref eq. of step 6)

yr
.m
w 1
w
,w

- (0.18)
ty

(1+e )
or
ab

0.5448
kr

{ O }H = =
ha

1
C

0.505
C
R

- (0.02)
(1+e )

Values from step 3 values

28
 fo
.in
rs
de
SC - NN - BPN – Algorithm

ea
■ Step 5 : (ref eq. of step 7)

yr
.m
w
w
0.5448
,w

= [ W] T
= ( 0.2 = - 0.14354
ty

{ I }O { O }H - 0.5 )
or

0.505
ab
kr
ha
C

Values from step 2 , from step 4

C
R

■ Step 6 : (ref eq. of step 8)

1
{ O }O = - (0.14354) = 0.4642
(1+e )

Values from step 5

■ Step 7 : (ref eq. of step 9)

Error = (TO – OO1 )2 = (0.1 – 0.4642)2 = 0.13264

table first training set o/p from step 6

29
 fo
.in
rs
de
SC - NN - BPN – Algorithm

ea
■ Step 8 : (ref eq. of step 10)

yr
.m
w
w
d = (TO – OO1 ) ( OO1 ) (1 – OO1 )
,w
ty

= (0.1 – 0.4642) (0.4642) ( 0.5358) = – 0.09058

or
ab
kr

Training o/p all from step 6

ha
C

(ref eq. of step 11)

C
R

0.5448 –0.0493
[ Y ] = { O }H (d ) = (– 0.09058) =
0.505 –0.0457

from values at step 4 from values at step 8 above

■ Step 9 : (ref eq. of step 12)

1 0
[∆W] = α [∆W] + η[Y] assume η =0.6

–0.02958
=
–0.02742
from values at step 2 & step 8 above

■ Step 10 : (ref eq. of step 13)

0.2 –0.018116
{e} = [W] {d}= (– 0.09058) =
-0.5 –0.04529
from values at step 8 above
from values at step 2
30
 fo
.in
rs
de
SC - NN - BPN – Algorithm

ea
■ Step 11 : (ref eq. of step 13)

yr
.m
w
w
,w

(–0.018116) (0.5448) (1- 0.5448) –0.00449

ty
or

{ d* } = =
ab

(0.04529) (0.505) ( 1 – 0.505) –0.01132

kr
ha
C
C
R

from values at step 10 at step 4 at step 8

■ Step 12 : (ref eq. of step 13)

0.4
[ X ] = { O }I ( d* ) = ( – 0.00449 0.01132)
-0.7

from values at step 1 from values at step 11 above

– 0.001796 0.004528

= 0.003143 –0.007924

■ Step 13 : (ref eq. of step 14)

– 0.001077 0.002716
1 0
[∆V] = α [∆V] + η[X] =
0.001885 –0.004754

from values at step 2 & step 8 above

31
 fo
.in
rs
de
SC - NN - BPN – Algorithm

ea
■ Step 14 : (ref eq. of step 15)

yr
.m
w
w
,w

0.1 0.4 – 0.001077 0.002716

1
ty

[V] = +
or

-0.2 0.2
ab

0.001885 –0.004754
kr
ha
C

from values at step 2 from values at step 13

C
R

– 0.0989 0.04027
=
0.1981 –0.19524

0.2 –0.02958 0.17042

1
[W] = + =
-0.5 –0.02742 –0.52742

from values at step 2, from values at step 9

■ Step 15 :

With the updated weights [ V ] and [ W ] , error is calculated again

and next training set is taken and the error will then get adjusted.

■ Step 16 :

Iterations are carried out till we get the error less than the tolerance.

■ Step 17 :

Once the weights are adjusted the network is ready for

inferencing new objects .

32
 fo
.in
rs
de
SC - NN - BPN – References

ea
4. References : Textbooks

yr
.m
w
w
1. "Neural Network, Fuzzy Logic, and Genetic Algorithms - Synthesis and
,w
ty

Applications", by S. Rajasekaran and G.A. Vijayalaksmi Pai, (2005), Prentice Hall,

or

Chapter 3, page 34-86.

ab
kr
ha
C

2. "Soft Computing and Intelligent Systems Design - Theory, Tools and Applications",
C
R

by Fakhreddine karray and Clarence de Silva (2004), Addison Wesley, chapter 5,

page 249-293.

3. "Elements of Artificial Neural Networks", by Kishan Mehrotra, Chilukuri K. Mohan

and Sanjay Ranka, (1996), MIT Press, Chapter 3, page 65-106.

4. "Fundamentals of Neural Networks: Architecture, Algorithms and Applications", by

Laurene V. Fausett, (1993), Prentice Hall, Chapter 6, page 289-332.

5. "Neural Network Design", by Martin T. Hagan, Howard B. Demuth and Mark

Hudson Beale, ( 1996) , PWS Publ. Company, Chapter 11-12, page 11-1 to 12-50.

6. Related documents from open source, mainly internet. An exhaustive list is

being prepared for inclusion at a later date.

33