Scribd
Upload
59 views34 pages

Multilayered Network Architectures

The document describes the multilayered feedforward neural network architecture and the backpropagation algorithm for training such networks. The backpropagation algorithm uses supervised learning by calculating the error between the network's output and the desired output. It then propagates this error backwards to adjust the weights in order to minimize error through an iterative process of forward and backward passes over the training data.

Uploaded by

aniruthan venkatavaradhan
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
59 views34 pages

Multilayered Network Architectures

The document describes the multilayered feedforward neural network architecture and the backpropagation algorithm for training such networks. The backpropagation algorithm uses supervised learning by calculating the error between the network's output and the desired output. It then propagates this error backwards to adjust the weights in order to minimize error through an iterative process of forward and backward passes over the training data.

Uploaded by

aniruthan venkatavaradhan
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 34

Multilayered Network Architectures

Input layer Hidden layer Output layer

Linear neuron Sigmoidal neuron


Supervised Learning Procedure

Error information fed back for network adaptation

Xk Sk Error

Dx
Neural Network
Backpropagation Weight Update
Procedure
1. Select a pattern Xk from the training set T present it to
the network.
2. Forward Pass: Compute activations and signals of input,
hidden and output neurons in that sequence.
3. Error Computation: Compute the error over the output
neurons by comparing the generated outputs with the
desired outputs.
4. Compute Weight Changes: Use the error to compute the
change in the hidden to output layer weights, and the
change in input to hidden layer weights such that a global
error measure gets reduced.
Backpropagation Weight Update
Procedure
5. Update all weights of the network.

6. Repeat Steps 1 through 5 until the global error


falls below a predefined threshold.
Square Error Function
† The instantaneous summed squared error ε k is
the sum of the squares of each individual
output error je
k , scaled by one-half:
Error Surface
Gradient Descent Procedure
Recall: Gradient Descent Update
Equation

† It follows logically therefore, that the weight


component should be updated in proportion
with the negative of the gradient as follows:
Neuron Signal Functions
† Input layer neurons are
linear.

† Hidden and output layer


neurons are sigmoidal.

† A training data set is


assumed to be given
which will be used to
train the network.
Notation for Backpropagation
Algorithm Derivation
The General Idea Behind
Iterative Training…
† Employ the gradient of the pattern error in order
to reduce the global error over the entire training
set.
† Compute the error gradient for a pattern and use
it to change the weights in the network.
† Such weight changes are effected for a sequence
of training pairs (X1 ,D1 ), (X2 ,D2), . . . , (Xk,Dk), . . .
picked from the training set.
† Each weight change perturbs the existing neural
network slightly, in order to reduce the error on
the pattern in question.
Square Error Performance
Function
† The k th training pair (Xk ,Dk) then defines the
instantaneous error:
„ E k = Dk − S(Yk) where
„ Ek = (e 1,k . . . , e p)k
„ = (d −1kS(y 1k ), . . . , dpk − S(ypk))
† The instantaneous summed squared error E k is
the sum of the squares of each individual
output error ejk, scaled by one-half:
The Difference Between Batch and
Pattern Update
Derivation of BP Algorithm:
Forward Pass-Input Layer
Derivation of BP Algorithm:
Forward Pass-Hidden Layer
Derivation of BP Algorithm:
Forward Pass-Output Layer
Recall the Gradient Descent
Update Equation
† A weight gets
updated based on
the negative of the
error gradient with
respect to the
weight
Derivation of BP Algorithm:
Computation of Gradients
Derivation of BP Algorithm:
Computation of Gradients
Derivation of BP Algorithm:
Computation of Gradients
Derivation of BP Algorithm:
Computation of Gradients
Derivation of BP Algorithm:
Computation of Gradients
Generalized Delta
Generalized Delta Rule:
Rule: Momentum
† Increases the rate of learning while
maintaining stability
How Momentum Works
† Momentum should be less than 1
for convergent dynamics.
† If the gradient has the same
sign on consecutive iterations
the net weight change increases
over those iterations
accelerating the descent.
† If the gradient has different
signs on consecutive iterations
then the net weight change
decreases over those iterations
and the momentum decelerates
the weight space traversal. This
helps avoid oscillations.
Derivation of BP Algorithm:
Finally…!
Backpropagation Algorithm:
Operational Summary
Backpropagation Algorithm:
Operational Summary(contd.)
Summary(contd.)
Hand-worked
Hand- worked Example
Forward Pass 1/Backprop Pass 1
Weight Changes: Pass 1
Network
Network N
N2 after
after first
first Iteration
Iteration
Forward Pass 2/Backprop Pass 2
Weight Changes: Pass 2
Network
Network N
N3 after
after second
second Iteration
Iteration

You might also like