Scribd
Upload
4 views

Lecture 10

Uploaded by

maniamritamysore
Copyright
© © All Rights Reserved
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
4 views

Lecture 10

Uploaded by

maniamritamysore
Copyright
© © All Rights Reserved
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
You are on page 1/ 19

An Introduction To The

Backpropagation Algorithm
Basic Neuron Model In A
Feedforward Network

 Inputs xi arrive through


pre-synaptic connections
 Synaptic efficacy is
modeled using real
weights wi
 The response of the
neuron is a nonlinear
function f of its
weighted inputs

2 Copyright G. A. Tagliarini, PhD 10/17/24


Inputs To Neurons

 Arise from other neurons or from outside the


network
 Nodes whose inputs arise outside the network
are called input nodes and simply copy values
 An input may excite or inhibit the response of
the neuron to which it is applied, depending
upon the weight of the connection

3 Copyright G. A. Tagliarini, PhD 10/17/24


Weights

 Represent synaptic efficacy and may be


excitatory or inhibitory
 Normally, positive weights are considered as
excitatory while negative weights are thought
of as inhibitory
 Learning is the process of modifying the
weights in order to produce a network that
performs some function

4 Copyright G. A. Tagliarini, PhD 10/17/24


Output

 Theresponse function is normally nonlinear


 Samples include
– Sigmoid
1
f ( x) 
1  e  x
– Piecewise linear
 x, if x 
f ( x) 
0, if x  
5 Copyright G. A. Tagliarini, PhD 10/17/24
Backpropagation Preparation
 Training Set
A collection of input-output patterns that are used to
train the network
 Testing Set
A collection of input-output patterns that are used to
assess network performance
 Learning Rate-η
A scalar parameter, analogous to step size in
numerical integration, used to set the rate of
adjustments

6 Copyright G. A. Tagliarini, PhD 10/17/24


Network Error

 Total-Sum-Squared-Error (TSSE)
1
TSSE    (desired  actual ) 2

2 patterns outputs
 Root-Mean-Squared-Error (RMSE)
2 * TSSE
RMSE 
# patterns*# outputs

7 Copyright G. A. Tagliarini, PhD 10/17/24


A Pseudo-Code Algorithm
 Randomly choose the initial weights
 While error is too large
– For each training pattern (presented in random order)
 Apply the inputs to the network
 Calculate the output for every neuron from the input layer, through
the hidden layer(s), to the output layer
 Calculate the error at the outputs
 Use the output error to compute error signals for pre-output layers
 Use the error signals to compute weight adjustments
 Apply the weight adjustments

– Periodically evaluate the network performance

8 Copyright G. A. Tagliarini, PhD 10/17/24


Possible Data Structures
 Two-dimensional arrays
– Weights (at least for input-to-hidden layer and hidden-to-
output layer connections)
– Weight changes (ij)
 One-dimensional arrays
– Neuron layers
 Cumulative current input
 Current output
 Error signal for each neuron

– Bias weights

9 Copyright G. A. Tagliarini, PhD 10/17/24


Apply Inputs From A Pattern
 Apply the value of each Feedforward
input parameter to each
input node
 Input nodes compute

Outputs
only the identity function

Inputs

10 Copyright G. A. Tagliarini, PhD 10/17/24


Calculate Outputs For Each Neuron
Based On The Pattern
 The output from neuron j for
pattern p is Opj where
Feedforward
1
O pj (net j )    net j
1 e

Outputs
Inputs
and
net j bias *Wbias   O pkWkj
k
k ranges over the input indices
and Wjk is the weight on the
connection from input k to
neuron j

11 Copyright G. A. Tagliarini, PhD 10/17/24


Calculate The Error Signal For Each
Output Neuron

 The output neuron error signal pj is given by


pj=(Tpj-Opj) Opj (1-Opj)
T is the target value of output neuron j for
pj
pattern p
 O is the actual output value of output neuron j
pj
for pattern p

12 Copyright G. A. Tagliarini, PhD 10/17/24


Calculate The Error Signal For Each
Hidden Neuron
 The hidden neuron error signal pj is given by

 pj O pj (1  O pj )  pkWkj
k

where pk is the error signal of a post-synaptic


neuron k and Wkj is the weight of the
connection from hidden neuron j to the post-
synaptic neuron k

13 Copyright G. A. Tagliarini, PhD 10/17/24


Calculate And Apply Weight
Adjustments
 Compute weight adjustments Wji at time t by

Wji(t)= η pj Opi

 Apply weight adjustments according to

Wji(t+1) = Wji(t) + Wji(t)

 Some add a momentum term Wji(t-1)


14 Copyright G. A. Tagliarini, PhD 10/17/24
An Example: Exclusive “OR”
 Training set
– ((0.1, 0.1), 0.1)
– ((0.1, 0.9), 0.9)
– ((0.9, 0.1), 0.9)
– ((0.9, 0.9), 0.1)
 Testing set
– Use at least 121 pairs equally spaced on the unit
square and plot the results
– Omit the training set (if desired)

15 Copyright G. A. Tagliarini, PhD 10/17/24


An Example (continued): Network
Architecture

output(s)
inputs

16 Copyright G. A. Tagliarini, PhD 10/17/24


An Example (continued): Network
Architecture

0.1 Target
output
Sample 1
input 0.9

0.9 1

17 Copyright G. A. Tagliarini, PhD 10/17/24


Feedforward Network Training by
Backpropagation: Process Summary

 Selectan architecture
 Randomly initialize weights
 While error is too large
– Select training pattern and feedforward to find actual
network output
– Calculate errors and backpropagate error signals
– Adjust weights
 Evaluate performance using the test set

18 Copyright G. A. Tagliarini, PhD 10/17/24


An Example (continued): Network
Architecture

?? Actual Target
0.1 ?? output output
Sample ?? ??
1
input ??? 0.9
??
0.9 ?? ?? 1
??
?? 1

19 Copyright G. A. Tagliarini, PhD 10/17/24

You might also like