Recurrent Network of a Single Layer

EL
PT
N
Recurrent Neural Network (RNNs)
• A recurrent neural network has
both feed-forward and feed-back
connections
• Information can be processed

EL
from input layer to output layer
and vice-versa

PT
• It forms a cycle or loop
• It may be preferred to MLFFNN

N
for modeling a highly dynamic
process Elman Network

Elman Network
 EL
PT
N
Jordan Network
 EL
PT
N
Combined Elman and Jordan Network
Self-Organizing Map (SOM)/ Kohonen Network

• Proposed by T. Kohonen before

EL
technique or dimensionality
reduction technique (topology

PT
preserving tool)
• It can be used as a clustering

N
algorithm also
• It woks based on unsupervised and Self-Organizing Map
competitive learning

Self-Organizing Map
• Non-linear generalization of principal component analysis
• Consists of two layers: Input layer and competition layer
• On competition layer, there are three basic operations, namely

EL
competition, cooperation and updating

PT
N
Self-Organizing Map (SOM)/ Kohonen Network (Cont.)

EL
PT
N
Self-Organizing Map
Competition
Let us assume that there are N points (neurons) in the input
layer and each point has m dimension
Xi = [xi1, xi2,.. . . . , xim]T,
where i = 1, 2, . . . . ., N

EL
Let the synaptic weight vector between input neuron i and
neuron j lying in the competition layer is denoted by

PT
T
W ji =  w ij 1 , w ij 2 , ....., w ijm
 ,
 

N
where j=1, 2, . . . . , N
Let n: neuron lying in the competition layer that has the best
match with the input vector Xi Winning neuron

EL
Euclidean distance between n and Xi can be expressed as
follows:
n(Xi) = Minimum of ( X i − W j )
2

PT
i

where j = 1, 2, . . . ., N

N
Cooperation
Surrounding a winning neuron, a neighborhood of excited neurons is defined
for cooperation in order to update their synaptic weights
 d 2j , n( x ) 
(
h j , n( x )= t ) exp  − i
,
i
 2σ t2 
 

EL
where t = 0 , 1, 2, . . .

PT
d j , n( x ) : Lateral distance between the winning neuron n and excited neuron j
i

σt : Standard deviation at t-th iteration

N
σt = σo exp(-t/τ),
σo: Initial value of standard deviation
τ: Predefined number of maximum iterations
Updating

Synaptic weights of the winning neuron and excited neurons

EL
W ji ( t +=
1) W ji ( t ) + η ( t ) h j , n( x ) ( t )  X i − W ji ( t )  ,

PT
i

N
η(t): learning rate (0.0, 1.0)
Final Mapping

EL
PT
N
Simulation Results

Schaffer’s first test function

EL
sin 2 xi2 − 0.5
i =1
=
y 0.5 +

PT
2
 4

1.0 + 0.001 

∑
i =1
2
xi 


N
Counter-Propagation Neural Network (CPNN)
 Introduced by Robert Hecht-Nielsen (1987)
 It consist of 3 layers:
(i) Input layer

EL
(ii) Unsupervised Kohonen layer
(iii) Teachable output layer performing Grossberg

PT
learning (supervised)
 It contains in-star model and out-star model

N
In-star model: Formed by input and Kohonen layers
Out-star model: Formed by Kohonen and output layers
 CPNN is faster than BPNN and there is no chance of its weight to get
trapped into local minima

EL
 CPNN could be inferior to BPNN in mapping applications.

PT
 CPNN could be either a full CPNN or a forward-only CPNN.

N
 Full CPNN
• It consist of two input
layers and two output
layers with a common
hidden layer to them.

EL
PT
N
Training of In-star Model

EL
PT
N
 Training of In-star Model(continued)

 Generate the connecting weight [u] and [v] at random in the range of
(0,1)
 α: learning rate between x inputs layer and hidden layer

EL
β: learning rate between y inputs layer and hidden
layer

PT
 Kohonen’s SOM is used during the training of in-star model.

N
Training of In-star Model (continued)

m n
d=
=i 1=
j
k 1
∑ i ij ∑ k kj
(x − u )2
+ (y − v ) 2

EL
PT
N
Training of Out-star Model

EL
PT
N
 Training of Out-Star Model (continued)

 Say zj be the winner hidden neuron

 Generate the connecting weights [w] and [s] at random in

EL
the range of (0,1)

PT
 γ : learning rate between zj and x*output layer

N
δ : learning rate between zj and y*output layer
 Training of Out-Star Model (continued)
 Grossberg’s learning is used during the training of out-star
model
wji(updated) = wji (previous)+γ (xi-wji(previous))
where i = 1,2,……,m

EL
sjk(updated) = sjk (previous)+δ (yk-sjk(previous))

PT
where k = 1,2,……,n
xi*= wji (updated)

N
yk*= sjk (updated)
 It completes one iteration of out-star training.
 Forward-Only CPNN
• It consists of one input layer and one output layer with a common hidden
layer, which performs the clustering

EL
PT
N
 A Numerical Example
Let us assume that a full CPNN is to be used to model a dataset having three inputs
and two outputs. One such data is represented as (x1 , x2 , x3 ) = (0.3,0.5,0.6) and
(y1 ,y2) = (0.3,0.4). Let us consider two hidden neurons only.

EL
PT
N
The connecting weights are initially assumed to be as follows:

EL
PT
N
N
PT
EL
In-star model
In order to decide the winner out of two hidden neurons,
we calculate Euclidean distances as follows:

∑( x ) + ∑( y − v k1 )

EL
3 2
2 2
d=
1 i i1
−u k
i 1= k 1

PT
( 0.3 − 0.2 ) + ( 0.5 − 0.1) + ( 0.6 − 0.8 ) + ( 0.3 − 0.4 ) + ( 0.4 − 0.2 )
2 2 2 2 2
= = 0.51

N
 ∑( x ) + ∑( y − vk2 )
3 2
2 2
d 2= i i2
−u k
i 1= k 1

( 0.3 − 0.3 ) + ( 0.5 − 0.6 ) + ( 0.6 − 0.5 ) + ( 0.3 − 0.7 ) + ( 0.4 − 0.3 )
2 2 2 2 2
=

EL
= 0.44

As d2 < d1 , z2 is the winner.

PT
Its connecting weights are updated as

N
u12 (updated) = u12 (previous) + α (x1 - u12 (previous))
= 0.3 + 0.2 (0.3 - 0.3)
= 0.3
u22 (updated) = u22 (previous) + α (x2 - u22 (previous))
= 0.6 + 0.2 (0.5 - 0.6) = 0.58

u32 (updated) = u32 (previous) + α (x3 - u32 (previous))

EL
= 0.5 + 0.2 (0.6 - 0.5) = 0.52

PT
v12 (updated) = v12 (previous) + β (y1 - v12 (previous))
= 0.7 + 0.3 (0.3 - 0.7) = 0.58

N
v22 (updated) = v22 (previous) + β (y2 - v22 (previous))
= 0.3 + 0.3 (0.4 - 0.3) = 0.33
Out-star model

It involves only the winner neuron .

EL
PT
N
Out-star model

Its connecting weights are updated as

w21 (updated) = w21 (previous) + γ (x1 - w21 (previous))

EL
= 0.21
w22 (updated) = w22 (previous) + γ (x2 - w22 (previous)) = 0.3 + 0.1(0.5-0.3) = 0.32

PT
w23 (updated) = w23 (previous) + γ (x3 - w23 (previous)) = 0.4 + 0.1(0.6-0.4) = 0.42

N
s21 (updated) = s21 (previous) + δ (y1 - s21 (previous))
= 0.5 + 0.4(0.3 - 0.5) = 0.42
N
PT
EL
References:
D.K. Pratihar Soft computing: fundamentals and
applications. New Delhi, India: Narosa Publishing
House, 2013.

EL
PT
N
Conclusion:

• Working principles of MLFFNN; RBFNN; RNN; SOM;

CPNN have been discussed with suitable examples

EL
PT
N
N
PT
EL