Scribd
Upload
15 views19 pages

Lecture 10

Uploaded by

aa zezen zaenal abidin
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
15 views19 pages

Lecture 10

Uploaded by

aa zezen zaenal abidin
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
You are on page 1/ 19

An Introduction To The

Backpropagation Algorithm

Who gets the credit?


Basic Neuron Model In A
Feedforward Network

 Inputs xi arrive through


pre-synaptic connections
 Synaptic efficacy is
modeled using real
weights wi
 The response of the
neuron is a nonlinear
function f of its
weighted inputs

2 Copyright G. A. Tagliarini, PhD 08/30/23


Inputs To Neurons

 Arise from other neurons or from outside the


network
 Nodes whose inputs arise outside the network
are called input nodes and simply copy values
 An input may excite or inhibit the response of
the neuron to which it is applied, depending
upon the weight of the connection

3 Copyright G. A. Tagliarini, PhD 08/30/23


Weights

 Represent synaptic efficacy and may be


excitatory or inhibitory
 Normally, positive weights are considered as
excitatory while negative weights are thought
of as inhibitory
 Learning is the process of modifying the
weights in order to produce a network that
performs some function

4 Copyright G. A. Tagliarini, PhD 08/30/23


Output

 The response function is normally nonlinear


 Samples include
– Sigmoid
1
f ( x) 
1  e x
– Piecewise linear
 x, if x  
f ( x)  
0, if x  
5 Copyright G. A. Tagliarini, PhD 08/30/23
Backpropagation Preparation
 Training Set
A collection of input-output patterns that are used to
train the network
 Testing Set
A collection of input-output patterns that are used to
assess network performance
 Learning Rate-η
A scalar parameter, analogous to step size in
numerical integration, used to set the rate of
adjustments

6 Copyright G. A. Tagliarini, PhD 08/30/23


Network Error

 Total-Sum-Squared-Error (TSSE)
1
TSSE    (desired  actual ) 2

2 patterns outputs
 Root-Mean-Squared-Error (RMSE)
2 * TSSE
RMSE 
# patterns*# outputs

7 Copyright G. A. Tagliarini, PhD 08/30/23


A Pseudo-Code Algorithm
 Randomly choose the initial weights
 While error is too large
– For each training pattern (presented in random order)
 Apply the inputs to the network
 Calculate the output for every neuron from the input layer, through
the hidden layer(s), to the output layer
 Calculate the error at the outputs
 Use the output error to compute error signals for pre-output layers
 Use the error signals to compute weight adjustments
 Apply the weight adjustments
– Periodically evaluate the network performance

8 Copyright G. A. Tagliarini, PhD 08/30/23


Possible Data Structures
 Two-dimensional arrays
– Weights (at least for input-to-hidden layer and hidden-to-
output layer connections)
– Weight changes (ij)
 One-dimensional arrays
– Neuron layers
 Cumulative current input
 Current output
 Error signal for each neuron
– Bias weights

9 Copyright G. A. Tagliarini, PhD 08/30/23


Apply Inputs From A Pattern
 Apply the value of each Feedforward
input parameter to each
input node
 Input nodes compute

Outputs
only the identity function

Inputs

10 Copyright G. A. Tagliarini, PhD 08/30/23


Calculate Outputs For Each Neuron
Based On The Pattern
 The output from neuron j for
pattern p is Opj where Feedforward
1
O pj (net j )    net j
1 e

Outputs
and

Inputs
net j  bias *Wbias   O pkWkj
k
k ranges over the input indices
and Wjk is the weight on the
connection from input k to
neuron j

11 Copyright G. A. Tagliarini, PhD 08/30/23


Calculate The Error Signal For Each
Output Neuron

 The output neuron error signal pj is given by


pj=(Tpj-Opj) Opj (1-Opj)
 Tpj is the target value of output neuron j for
pattern p
 Opj is the actual output value of output neuron j
for pattern p

12 Copyright G. A. Tagliarini, PhD 08/30/23


Calculate The Error Signal For Each
Hidden Neuron

 The hidden neuron error signal pj is given by

 pj  O pj (1  O pj )  pkWkj
k

where pk is the error signal of a post-synaptic


neuron k and Wkj is the weight of the
connection from hidden neuron j to the post-
synaptic neuron k

13 Copyright G. A. Tagliarini, PhD 08/30/23


Calculate And Apply Weight
Adjustments

 Compute weight adjustments Wji at time t by

Wji(t)= η pj Opi

 Apply weight adjustments according to

Wji(t+1) = Wji(t) + Wji(t)

 Some add a momentum term Wji(t-1)

14 Copyright G. A. Tagliarini, PhD 08/30/23


An Example: Exclusive “OR”
 Training set
– ((0.1, 0.1), 0.1)
– ((0.1, 0.9), 0.9)
– ((0.9, 0.1), 0.9)
– ((0.9, 0.9), 0.1)
 Testing set
– Use at least 121 pairs equally spaced on the unit
square and plot the results
– Omit the training set (if desired)

15 Copyright G. A. Tagliarini, PhD 08/30/23


An Example (continued): Network
Architecture

output(s)
inputs

16 Copyright G. A. Tagliarini, PhD 08/30/23


An Example (continued): Network
Architecture

Target
0.1
output
Sample 1
input 0.9

0.9 1

17 Copyright G. A. Tagliarini, PhD 08/30/23


Feedforward Network Training by
Backpropagation: Process Summary

 Select an architecture
 Randomly initialize weights
 While error is too large
– Select training pattern and feedforward to find actual
network output
– Calculate errors and backpropagate error signals
– Adjust weights
 Evaluate performance using the test set

18 Copyright G. A. Tagliarini, PhD 08/30/23


An Example (continued): Network
Architecture

?? Actual Target
0.1 ?? output output
Sample ?? ?? 1
input ??? 0.9
??
0.9 ?? ?? 1
??
??
1

19 Copyright G. A. Tagliarini, PhD 08/30/23

You might also like