Soft Computing Unit-2

Introduction to Soft Computing

4th Semester CSE (AR-21)

1 Department of CSE, GIET University, Gunupur Friday, May 3, 2024

 Soft Computing Lecture -8

Outline of the Lecture

 Concept of Learning

 Single Layer feed forward NN training

 Multi Layer feed forward NN training

 Backpropagation Algorithm

2 Department of CSE, GIET University, Gunupur Friday, May 3, 2024

Soft Computing The Concept of Learning

 The learning is an important feature of human computational

ability.
 Learning may be viewed as the change in behavior acquired due to
practice or experience, and it lasts for relatively long time.
 As it occurs, the effective coupling between the neuron is
modified.
 In case of artificial neural networks, it is a process of modifying
neural network by updating its weights, biases and other
parameters, if any.
 During the learning, the parameters of the networks are optimized
and as a result process of curve fitting.
 It is then said that the network has passed through a learning
phase.

3 Department of CSE, GIET University, Gunupur Friday, May 3, 2024

Soft Computing Single Layer feed forward NN training

 We know that, several neurons are arranged in one layer with

inputs and weights connect to every neuron.
 Learning in such a network occurs by adjusting the weights
associated with the inputs so that the network can classify the
input patterns.
 A single neuron in such a neural network is called perceptron.
 The algorithm to train a perceptron is stated below.
 Let there is a perceptron with (n + 1) inputs x 0, x1, x2,….., xnwhere
x0 = 1 is the bias input.
 Let f denotes the transfer function of the neuron. Suppose, and
denotes the input-output vectors as a training data set. denotes the
weight matrix.
With this input-output relationship pattern and configuration of a
perceptron, the algorithm Training Perceptron to train the
perceptron is stated in the following slide.

4 Department of CSE, GIET University, Gunupur Friday, May 3, 2024

Soft Computing Single Layer feed forward NN training

1. Initialize W = w0, w1, … ,wn to some random weights.

2. For each input pattern x do Here, x = {x0, x1, … , xn}
Compute I =
Compute observed output y = f (I) =
= + y Add y to , which is initially empty
3. If the desired output matches the observed output then
output and exit.
4. Otherwise, update the weight matrix as follows :
For each output y do
If the observed out y is 1 instead of 0, then w i = wi - αxi,
(i = 0, 1, 2, ... , n)
Else, if the observed out y is 0 instead of 1, then w i = wi + αxi,
(i = 0, 1, 2, ... , n)
5. Go to step 2.

5 Department of CSE, GIET University, Gunupur Friday, May 3, 2024

Soft Computing Single Layer feed forward NN training

In the above algorithm, α is the learning parameter and is a constant

decided by some empirical studies.
Note :
 The algorithm Training Perceptron is based on the supervised
learning technique
 ADALINE : Adaptive Linear Network Element is also an
alternative term to perceptron
 If there are 10 number of neurons in the single layer feed
forward neural network to be trained, then we have to iterate
the algorithm for each perceptron in the network.

6 Department of CSE, GIET University, Gunupur Friday, May 3, 2024

Soft Computing Specification of Multi Layer feed forward NN

• Like single layer feed forward neural network, supervisory

training methodology is followed to train a multilayer feed
forward neural network.
• Before going to understand the training of such a neural network,
we redefine some terms involved in it.
• A block diagram and its configuration for a three layer multilayer
FF NN of type l - m - n is as follows.

7 Department of CSE, GIET University, Gunupur Friday, May 3, 2024

Soft Computing Specification of Multi Layer feed forward NN

8 Department of CSE, GIET University, Gunupur Friday, May 3, 2024

Soft Computing Specification of Multi Layer feed forward NN

• For simplicity, we assume that all neurons in a particular layer

follow same transfer function and different layers follows their
respective transfer functions as shown in the configuration.
• Let us consider a specific neuron in each layer say i-th, j-th and k-
th neurons in the input, hidden and output layer, respectively.
• Also, let us denote the weight between i-th neuron (i = 1, 2, ... , l )
in input layer to j-th neuron (j = 1, 2, ... , m) in the hidden layer is
denoted by vij.
• The weight matrix between the input to hidden layer say V is
denoted as follows.

V=

9 Department of CSE, GIET University, Gunupur Friday, May 3, 2024

 Soft Computing Specification of Multi Layer feed forward NN

• Similarly, wjk represents the connecting weights between j-th

neuron(j = 1, 2, ... ,m) in the hidden layer and k-th neuron (k = 1,
2, ... , n) in the output layer as follows:

W =

10 Department of CSE, GIET University, Gunupur Friday, May 3, 2024

 Soft Computing Learning of Multi Layer feed forward NN

Whole learning method consists of the following three computations:

 Input layer computation

 Hidden layer computation

 Output layer computation

In our computation, we assume that < T 0, TI > be the training set of

size |T|.

11 Department of CSE, GIET University, Gunupur Friday, May 3, 2024

 Soft Computing Input Layer Computation

• Let us consider an input training data at any instant be

• {I}I = [I11, I12, … , I1i, I1l ] where {I}I {T}I
• Consider the outputs of the neurons lying on input layer are the
same with the corresponding inputs to neurons in hidden layer.
• That is, {O}I = {I}I
[l1] [l1] [Output of the input layer]
• The input of the j-th neuron in the hidden layer can be calculated
as follows.
• IHj = V1j .OI1 + V2j .OI2 +, … , + Vij .OIj + … + Vlj .OIl
where j = 1, 2, …, m.
• [Calculation of input of each node in the hidden layer]
• In the matrix representation form, we can write
{I}H = [V]T {O}I
[m1] [ml] [l1]

12 Department of CSE, GIET University, Gunupur Friday, May 3, 2024

 Soft Computing Hidden Layer Computation

• Let us consider any j-th neuron in the hidden layer.

• Since the output of the input layer’s neurons are the input to the j-
th neuron and the j-th neuron follows the sigmoidal activation
function, we have OHj =

• where j = 1, 2, … , m and αH is the constant co-efficient of the

transfer function.
• Note that all output of the nodes in the hidden layer can be
expressed as a one-dimensional column matrix.
….
….
1
{O}H =
(1+𝑒¿¿−𝛼 ( 𝐼 𝐻𝑗−𝜃 𝐻𝑗 ))¿
….
…. [m 1]

13 Department of CSE, GIET University, Gunupur Friday, May 3, 2024

 Soft Computing Output Layer Computation
• Let us calculate the input to any k-th node in the output layer.
Since, output of all nodes in the hidden layer go to the k-th layer
with weights w1k , w2k , … ,wmk , we have
• IOk = W1k . OH1 + W2k . OH2 + … + Wmk . Ohm where k = 1, 2, … ,
n.
• In the matrix representation, we have
{I}O = [W]T {O}H
[n1] [n] [m1]
Now, we estimate the output of the k-th neuron in the output layer. We consider the
O =
sigmoidal activation function. Ok
…. for k = 1, 2, …n.,
….
1
{O}O =
(1+𝑒¿¿−𝛼 ( 𝐼𝑂𝑘−𝜃𝑂𝑘 ) )¿
….
….
[n 1]
14 Department of CSE, GIET University, Gunupur Friday, May 3, 2024
 Soft Computing Backpropagation Algorithm

 The above discussion comprises how to calculate values of

different parameters in l - m - n multiple layer feed forward
neural network.
 Next, we will discuss how to train such a neural network.
 We consider the most popular algorithm called Back-Propagation
Algorithm, which is a supervised learning.
 The principle of the Back-Propagation algorithm is based on
the error-correction with Steepest-descent method.
 We first discuss the method of steepest descent followed by its
use in the training algorithm.

15 Department of CSE, GIET University, Gunupur Friday, May 3, 2024

 Soft Computing Method of Steepest Descent

 Supervised learning is, in fact, error-based learning.

 In other words, with reference to an external (teacher) signal (i.e.
target output) it calculates error by comparing the target output
and computed output.
 Based on the error signal, the neural network should modify its
configuration, which includes synaptic connections, that is , the
weight matrices.
 It should try to reach to a state, which yields minimum error.
 In other words, its searches for a suitable values of parameters
minimizing error, given a training set.
 Note that, this problem turns out to be an optimization problem.

16 Department of CSE, GIET University, Gunupur Friday, May 3, 2024

 Soft Computing Method of Steepest Descent

(a) Searching for a (b) Error surface with two

minimum error parameters V and W

17 Department of CSE, GIET University, Gunupur Friday, May 3, 2024

 Soft Computing Method of Steepest Descent

 For simplicity, let us consider the connecting weights are the only
design parameter.
 Suppose, V and W are the weights parameters to hidden and
output layers, respectively.
 Thus, given a training set of size N, the error surface, E can be
represented as E=

 where Ii is the i-th input pattern in the training set and e i (…)
denotes the error computation of the i-th input.
 Now, we will discuss the steepest descent method of computing
error, given a changes in V and W matrices.

18 Department of CSE, GIET University, Gunupur Friday, May 3, 2024

 Soft Computing Calculation of Error in Neural Network

• Let us consider any k-th neuron at the output layer. For an input
pattern Ii TI (input in training) the target output T Ok of the k-th
neuron be TOk .
• Then, the error ek of the k-th neuron is defined corresponding to
the input Ii as ek = (TOk - OOk )2
• where OOk denotes the observed output of the k-th neuron.
• For a training session with Ii TI , the error in prediction considering
all output neurons can be given as
• e= =
• where n denotes the number of neurons at the output layer.
• The total error in prediction for all output neurons can be
determined considering all training session < T I , TO > as
• e=

19 Department of CSE, GIET University, Gunupur Friday, May 3, 2024