Multi-Layer Perceptron (MLP)

Back Propagation Algorithm (sequential)

1. Apply an input vector and calculate all activations, a and u

2. Evaluate k for all output units via:
 i (t )  ( d i (t )  yi (t )) g ' ( ai (t ))
(Note similarity to perceptron learning algorithm)
3. Backpropagate ks to get error terms  for hidden layers using:

 i (t )  g ' (ui (t ))  k (t ) wki

k

4. Evaluate changes using:

vij (t  1)  vij (t )   i (t ) x j (t )
wij (t  1)  wij (t )   i (t ) z j (t )
Once weight changes are computed for all units, weights are updated
at the same time (bias included as weights here). An example:

x1 v11= -1
w11= 1 y1
v21= 0 w21= -1
v12= 0
w12= 0
x2 v22= 1 y2
w22= 1
v10= 1
v20= 1

Use identity activation function (ie g(a) = a)

All biases set to 1. Will not draw them for clarity.
Learning rate  = 0.1

x1= 0 v11= -1
w11= 1 y1
v21= 0 w21= -1
v12= 0
w12= 0
x2= 1 v22= 1 y2
w22= 1

Have input [0 1] with target [1 0].

Forward pass. Calculate 1st layer activations:

v11= -1 u1 = 1
x1 w11= 1 y1
v21= 0 w21= -1
v12= 0
w12= 0
x2 v22= 1 y2
w22= 1
u2 = 2

u1 = -1x0 + 0x1 +1 = 1
u2 = 0x0 + 1x1 +1 = 2
Calculate first layer outputs by passing activations thru activation
functions

z1 = 1
x1 v11= -1
w11= 1 y1
v21= 0 w21= -1
v12= 0
w12= 0
x2 v22= 1 y2
w22= 1
z2 = 2

z1 = g(u1) = 1
z2 = g(u2) = 2
Calculate 2nd layer outputs (weighted sum thru activation functions):

x1 v11= -1
w11= 1 y1= 2
v21= 0 w21= -1
v12= 0
w12= 0
x2 v22= 1 y2= 2
w22= 1

y1 = a1 = 1x1 + 0x2 +1 = 2
y2 = a2 = -1x1 + 1x2 +1 = 2
Backward pass:
wij (t  1)  wij (t )   i (t ) z j (t )
  ( d i (t )  yi (t )) g ' ( ai (t )) z j (t )

x1 v11= -1
w11= 1 1= -1
v21= 0 w21= -1
v12= 0
w12= 0
x2 v22= 1 2= -2
w22= 1

Target =[1, 0] so d1 = 1 and d2 = 0

So:
1 = (d1 - y1 )= 1 – 2 = -1
2 = (d2 - y2 )= 0 – 2 = -2
Calculate weight changes for 1st layer (cf perceptron learning):

v11= -1 z1 = 1
x1 w11= 1 1 z1 =-1
v21= 0 w21= -1 1 z2 =-2
v12= 0
w12= 0
x2 v22= 1 2 z1 =-2
w22= 1 2 z2 =-4
z2 = 2

wij (t  1)  wij (t )   i (t ) z j (t )
Weight changes will be:

wij (t  1)  wij (t )   i (t ) z j (t )

x1 v11= -1
w11= 0.9
v21= 0 w21= -1.2
v12= 0
w12= -0.2
x2 v22= 1
w22= 0.6
But first must calculate ’s:

 i (t )  g ' (ui (t ))  k (t ) wki

k

v11= -1
x1 1 w11= -1 1= -1
v21= 0
2 w21= 2
v12= 0 1 w12= 0
x2 v22= 1 2= -2
2 w22= -2
’s propagate back:

 i (t )  g ' (ui (t ))  k (t ) wki

k

v11= -1 1= 1
x1 1= -1
v21= 0
v12= 0
x2 v22= 1 2= -2

2 = -2

1 = - 1 + 2 = 1
2 = 0 – 2 = -2
And are multiplied by inputs:

vij (t  1)  vij (t )   i (t ) x j (t )

x1= 0 v11= -1  1 x1 = 0
1= -1
v21= 0  1 x2 = 1
v12= 0
2 x1 = 0
x2= 1 v22= 1 2= -2

2 x2 = -2
Finally change weights:

vij (t  1)  vij (t )   i (t ) x j (t )

x1= 0 v11= -1
w11= 0.9
v21= 0 w21= -1.2
v12= 0.1
w12= -0.2
x2 = 1 v22= 0.8
w22= 0.6

Note that the weights multiplied by the zero input are

unchanged as they do not contribute to the error
We have also changed biases (not shown)
Now go forward again (would normally use a new input vector):

vij (t  1)  vij (t )   i (t ) x j (t )

v11= -1 z1 = 1.2
x1= 0 w11= 0.9
v21= 0 w21= -1.2
v12= 0.1
w12= -0.2
x2 = 1 v22= 0.8
w22= 0.6
z2 = 1.6
Now go forward again (would normally use a new input vector):

vij (t  1)  vij (t )   i (t ) x j (t )

x1= 0 v11= -1 y1 = 1.66

w11= 0.9
v21= 0 w21= -1.2
v12= 0.1
w12= -0.2
x2 = 1 v22= 0.8
w22= 0.6
y2 = 0.32

Outputs now closer to target value [1, 0]

 Activation Functions
How does the activation function affect the changes?
 i (t )  ( d i (t )  yi (t )) g ' ( ai (t ))
 i (t )  g ' (ui (t ))  k (t ) wki
k

dg ( a )
Where: g ' ( ai (t )) 
da

- we need to compute the derivative of activation function g

- to find derivative the activation function must be smooth
(differentiable)
 Sigmoidal (logistic) function-common in MLP
1 1
g ( ai (t ))  
1  exp( k ai (t )) 1  e  k ai ( t )

where k is a positive
constant. The sigmoidal
function gives a value in
range of 0 to 1.
Alternatively can use
tanh(ka) which is same
shape but in range –1 to 1.

Input-output function of a
neuron (rate coding
assumption)
Note: when net = 0, f = 0.5
 Derivative of sigmoidal function is
k exp( k ai (t ))
g ' (ai (t ))   k g (ai (t ))[1  g (ai (t ))]
[1  k exp(k ai (t ))]2

since : yi (t )  g (ai (t )) we have : g ' (ai (t ))  k yi (t )(1  yi (t ))

Derivative of sigmoidal function has max at a = 0., is symmetric about

this point falling to zero as sigmoid approaches extreme values
Since degree of weight change is proportional to derivative of
activation function,
 i (t )  ( d i (t )  yi (t )) g ' ( ai (t ))
 i (t )  g ' (ui (t ))  k (t ) wki
k

weight changes will be greatest when units

receives mid-range functional signal and 0 (or very small)
extremes. This means that by saturating a neuron (making the
activation large) the weight can be forced to be static. Can be a
very useful property
Summary of (sequential) BP learning algorithm

Set learning rate


Set initial weight values (incl. biases): w, v

Loop until stopping criteria satisfied:

present input pattern to input units
compute functional signal for hidden units
compute functional signal for output units

present Target response to output units

computer error signal for output units
compute error signal for hidden units
update all weights at same time
increment n to n+1 and select next input and target
end loop
Network training:

Training set shown repeatedly until stopping criteria are met

Each full presentation of all patterns = ‘epoch’
Usual to randomize order of training patterns presented for each
epoch in order to avoid correlation between consecutive training
pairs being learnt (order effects)

Two types of network training:

• Sequential mode (on-line, stochastic, or per-pattern)

Weights updated after each pattern is presented

• Batch mode (off-line or per -epoch). Calculate the

derivatives/wieght changes for each pattern in the training set.
Calculate total change by summing imdividual changes
Advantages and disadvantages of different modes

Sequential mode
• Less storage for each weighted connection
• Random order of presentation and updating per pattern means
search of weight space is stochastic--reducing risk of local
minima
• Able to take advantage of any redundancy in training set (i.e..
same pattern occurs more than once in training set, esp. for large
difficult training sets)
• Simpler to implement

Batch mode:
• Faster learning than sequential mode
• Easier from theoretical viewpoint
• Easier to parallelise
Dynamics of BP learning

Aim is to minimise an error function over all training

patterns by adapting weights in MLP
Recall, mean squared error is typically used

p
1
E(t)=

2 k 1
( d k (t )  Ok (t )) 2

idea is to reduce E

in single layer network with linear activation functions, the

error function is simple, described by a smooth parabolic surface
with a single minimum
 But MLP with nonlinear activation functions have complex error
surfaces (e.g. plateaus, long valleys etc. ) with no single minimum

valleys
Selecting initial weight values

• Choice of initial weight values is important as this decides starting

position in weight space. That is, how far away from global minimum

• Aim is to select weight values which produce midrange function

signals

• Select weight values randomly form uniform probability distribution

• Normalise weight values so number of weighted connections per unit

produces midrange function signal
Regularization – a way of reducing variance (taking less
notice of data)

Smooth mappings (or others such as correlations) obtained by

introducing penalty term into standard error function

E(F)=Es(F)+ ER(F)

where  is regularization coefficient

penalty term: require that the solution should be smooth,

etc. Eg

ER ( F )    y d x 2
without regularization

with regularization
Momentum

Method of reducing problems of instability while increasing the rate

of convergence

Adding term to weight update equation term effectively

exponentially holds weight history of previous weights changed

Modified weight update equation is

wij (n  1)  wij (n )   j (n )yi (n )

  [wij (n )  wij (n  1)]
 is momentum constant and controls how much notice is taken of
recent history

Effect of momentum term

• If weight changes tend to have same sign

momentum terms increases and gradient decrease
speed up convergence on shallow gradient
• If weight changes tend have opposing signs
momentum term decreases and gradient descent slows to
reduce oscillations (stablizes)
• Can help escape being trapped in local minima
 Stopping criteria
Can assess train performance using

p M
E    [ d j ( n)  y j ( n)]i2
i 1 j 1

where p=number of training patterns,

M=number of output units
Could stop training when rate of change of E is small, suggesting
convergence

However, aim is for new patterns to be

classified correctly
error Training error
Generalisation
error
Training time
Typically, though error on training set will decrease as training
continues generalisation error (error on unseen data) hitts a
minimum then increases (cf model complexity etc)
Therefore want more complex stopping criterion
Cross-validation
Method for evaluating generalisation performance of networks
in order to determine which is best using of available data

Hold-out method
Simplest method when data is not scare

Divide available data into sets

• Training data set
-used to obtain weight and bias values during network training
• Validation data
-used to periodically test ability of network to generalize
-> suggest ‘best’ network based on smallest error
• Test data set
Evaluation of generalisation error ie network performance
Early stopping of learning to minimize the training error and
validation error
Universal Function Approximation

How good is an MLP? How general is an MLP?

Universal Approximation Theorem

For any given constant and continuous function h (x1,...,xm),

there exists a three layer MLP with the property that

| h (x1,...,xm) - H(x1,...,xm) |< 

where H ( x1 , ... , xm )=  k
a
i=1 i f (  m
j=1 wijxj + bi )