Multilayer perceptron neural network

. Multi-layer perceptron (MLP) is a supplement of feed forward neural network supervisor

learning algorithm. It consists of three types of layers(the input layer, output layer and hidden
layer) . The input layer receives the input signal to be processed. An arbitrary number of hidden
layers that are placed in between the input and output layer are the true computational engine of
the MLP. The major use cases of MLP are pattern classification, recognition, prediction and
approximation.

Step 1: Initialize the following to start the training (Weights, Bias, Learning rate α), for easy calculation
and simplicity,
Step 2: Continue step 3-8 when the stopping condition is not true.
Step 3: Continue step 4-6 for every training vector x.
Step 4: Activate each input unit as follows

Step 5: Obtain the net input with the following relation:

Here „b‟ is bias and „n‟ is the total number of input neurons.

Step 6 − Apply the following activation function to obtain the final output for each output unit
j = 1 to m

1
Step 7 : update the weight and bias for x = 1 to n and j=1 to m as following :

− Case 1 : if yj # tj

Case 2 − if yjj = tjj then,

Here „y‟ is the actual output and „t‟ is the desired/target output.

Step 8: Test for the stopping condition, which will happen when there is no change in weight.

Adaptive Linear Neuron (Adaline)

Adaline which stands for Adaptive Linear Neuron is a network having a single linear unit. The weights are
updated by minimizing the cost function via gradient descent.. Some important points about Adaline are as
follows:
- It uses bipolar (-1, +1) activation function.
- It uses delta rule for training to least mean square (LMS) between the actual output and the
desired/target output.
- The weights from the input units are adjustable.

Architecture
The basic structure of Adaline is similar to perceptron having an extra feedback loop with the help of
which the actual output is compared with the desired/target output. After comparison on the basis of
training algorithm, the weights and bias will be updated

Training Algorithm

Step 1: Initialize the following to start the training (Weights, Bias and Learning rate)

For easy calculation and simplicity, weights and bias must be set equal to 0 and the
learning rate must be set equal to 1.

Step 2: Continue step 3-8 when the stopping condition is not true.
2
Step 3: Continue step 4-6 for every bipolar training pair s:t.

Step 4: Activate each input unit as follows −

Step 5: Obtain the net input with the following relation :

Here „b‟ is bias and „n‟ is the total number of input neurons.

Step 6: Apply the following activation function to obtain the final output :

Step 7 : Adjust the weight and bias as follows :

Case 1 − if y ≠ t then,

Case 2 : if y = t then,

Here „y‟ is the actual output and „t‟ is the desired/target output (t-yin) is the computed error.

Step 8 − Test for the stopping condition, which will happen when there is no change in weight or the
highest weight change occurred during training is smaller than the specified tolerance.
Multiple Adaptive Linear Neuron (Madaline)
Madaline which stands for Multiple Adaptive Linear Neuron, is a network which consists of many
Adalines in parallel. It will have a single output unit. Some important points about Madaline are as
follows:
- It is just like a multilayer perceptron, where Adaline will act as a hidden unit between the input and
the Madaline layer
- The weights and the bias between the input and Adaline layers, as in we see in the Adaline
architecture, are adjustable.
- only the weights for the hidden Adalines are adjusted; the weights for the output unit are fixed.
- Training can be done with the help of Delta rule.

Architecture
The architecture of Madaline consists of “n” neurons of the input layer, “m” neurons of the Adaline
layer, and 1 neuron of the Madaline layer. The Adaline layer can be considered as the hidden layer as it is
between the input layer and the output layer, i.e. the Madaline layer

3
Training Algorithm

By now that only the weights and bias between the input and the Adaline layer are to be adjusted, and the
weights and bias between the Adaline and the Madaline layer are fixed.
Step 1: Initialize the following to start the training (Weights, Bias and Learning rate)

For easy calculation and simplicity, weights and bias must be set equal to 0 and the learning rate mus t be
set equal to 1.
Step 2: Continue step 3-8 when the stopping condition is not true.
Step 3: Continue step 4-7 for every bipolar training pair s:

Step 4: Activate each input unit as follows:

Step 5: Obtain the net input at each hidden layer, i.e. the Adaline layer with the following relation :

Here „b‟ is bias and „n‟ is the total number of input neurons.

Step 6: Apply the following activation function to obtain the final output at the Adaline and the Madaline
layer :

{
Output at the hidden (Adaline) unit:

Final output of the network:

i.e ∑

Step 7 : Calculate the error and adjust the weights a follows –

Case 1 : if y ≠ t and t = 1 then,

4
 In this case, the weights would be updated on Qjj where the net input is close to 0 because t = 1

. Case 2 − if y ≠ t and t = -1 then :

In this case, the weights would be updated on Qk where the net input is positive because t = -1.

Here „y‟ is the actual output and „t‟ is the desired/target output.
Case 3 − if y = t then
There would be no change in weights.
Step 8: Test for the stopping condition, which will happen when there is no change in weight or the highest
weight change occurred during training is smaller than the specified tolerance.
Back Propagation Neural Networks
Back Propagation Neural (BPN) is a multilayer neural network consisting of the input layer, at least one
hidden layer and output layer. As its name suggests, back propagating will take place in this network. The
error which is calculated at the output layer, by comparing the target output and the actual output, will be
propagated back towards the input layer.
Architecture
As shown in the diagram, the architecture of BPN has three interconnected layers having weights on them.
The hidden layer as well as the output layer also has bias, whose weight is always 1, on them. As is clear
from the diagram, the working of BPN is in two phases. One phase sends the signal from the input layer to
the output layer, and the other phase back propagates the error from the output layer to the input layer.

Training Algorithm

5
For training, BPN will use binary sigmoid activation function. The training of BPN will have the
following three phases.
- Phase 1 − Feed Forward Phase.
- Phase 2 − Back Propagation of error.
- Phase 3 − Updating of weights.
All these steps will be concluded in the algorithm as follows:

Step 1 : Initialize the following to start the training (Weights, Bias and Learning rate α)

For easy calculation and simplicity, take some small random values.

Step 2: Continue step 3-11 when the stopping condition is not true.
Step 3 : Continue step 4-10 for every training pair.
Phase 1
Step 4: Each input unit receives input signal xii and sends it to the hidden unit for all i = 1 to n

Step 5: Calculate the net input at the hidden unit using the following relation :

Here b0j is the bias on hidden unit, vij is the weight o j unit of the hidden layer coming from i unit of the input
layer.
Now calculate the net output by applying the following activation function

Send these output signals of the hidden layer units to the output layer units.

Step 6 − Calculate the net input at the output layer unit using the following relation −

Here b0k is the bias on output unit; wjk is the weight on k unit of the output layer coming from j unit of the hidden
layer.

Calculate the net output by applying the following activation function:

Phase 2
Step 7 − Compute the error correcting term, in correspondence with the target pattern received at each output unit,
as follows
)
On this basis, update the weight and bias as follows:

Then send δk back to the hidden layer

Step 8: Now each hidden unit will be the sum of its delta inputs from the output units:

17
 ∑

Error term can be calculated as follows:

On this basis, update the weight and bias as follows:

Phase 3
Step 9 : Each output unit (ykk = 1 to m) updates the weight and bias as follows:

Step 10: Each output unit (zjjj = 1 to p) updates the weight and bias as follows :

Step 11: Check for the stopping condition, which may be either the number of epochs reached or the target output
matches the actual output.
Generalized Delta Learning Rule
Delta rule works only for the output layer. On the other hand, generalized delta rule, also called as back-propagation
rule, is a way of creating the desired values of the hidden layer.
Mathematical Formulation
For the activation function the derivation of net input on Hidden layer as well as on output layer can be
given by:

And

Now the error which has to be minimized is :

∑

By using the chain rule, we have:

∑

Now let us say

The weights on connections to the hidden unit zz can be given by :

18
Putting the value of y ink we will get the following

Weight updating can be done as follows , For the output unit :

For the hidden unit

\
For the hidden unit

19