0% found this document useful (0 votes)
11 views

Soft Computing Module 2

The document covers memory models in soft computing, focusing on pattern association through associative memory networks, including auto and hetero associative models. It details training algorithms like the Hebb rule and outer products rule, as well as the concept of Hamming distance for measuring similarity between vectors. Additionally, it discusses the architecture and testing algorithms for autoassociative networks, illustrating their application in various fields such as speech and image processing.

Uploaded by

gx4z4ytf8z
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
11 views

Soft Computing Module 2

The document covers memory models in soft computing, focusing on pattern association through associative memory networks, including auto and hetero associative models. It details training algorithms like the Hebb rule and outer products rule, as well as the concept of Hamming distance for measuring similarity between vectors. Additionally, it discusses the architecture and testing algorithms for autoassociative networks, illustrating their application in various fields such as speech and image processing.

Uploaded by

gx4z4ytf8z
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 78

SOFT COMPUTING-SWE1011

MODULE-2: Memory Models


SYLLABUS

• Pattern association
• Auto associative memory models
• Hetero associative memory models
• BAM
• Hopfield network
PATTERN ASSOCIATION
PATTERN ASSOCIATION

• Pattern association is carried out efficiently by associative


memory networks
• An associative network is a single-layer net in which the
weights are determined in such a way that the net can store a
set of pattern associations.
• Each association is an input-output vector ( pair s:t)
• The two main algorithms used for training a pattern
association network are
• The Hebb rule
• The outer products rule
ASSOCIATIVE MEMORY NETWORK

• It stores a set of patterns as memories


• When it is presented with a key pattern, it responds by
producing one of the stored patterns
• The selected pattern resembles or relates to the key pattern
• The recall is through association of the key pattern
• These types of memories are called content- addressable
memories (CAM)
• In digital computers we have address- addressable memories
• As in RAM/ROM it is also a matrix memory
ASSOCIATIVE MEMORY NETWORK CONTD…

• The input data is correlated with that of the stored data in


CAM
• The stored patterns must be unique (That is different
patterns are stored in different locations)
• Each data stored has an address
• If multiple copies are stored the data will be correct. But the
address will be ambiguous
• The concept behind this search is to output any one or all
stored items which matches the given search argument
• The stored data is retrieved completely or partially
ASSOCIATIVE MEMORY NETWORK CONTD…

• Two types of such networks are there


• AUTO ASSOCIATIVE
• HETERO ASSOCIATIVE
• If the output vector is same as the input vectors it is Auto
Associative
• If the output vectors are different from the input vectors it is
Hetero Associative
• To find the similarity we use the Hamming distance
HAMMING DISTANCE

• Suppose we have two vectors,


x = ( x1 , x2 ,...xn )T and y = (y1 , y2 ,...yn )T
• Then the Hamming Distance is defined as
n
 xi − yi , if xi , yi  {0,1};
 i =1
HD( x, y ) =  n
1
 2 
xi − yi , if xi , yi  {−1,1}.
i =1

• The hamming distance basically denotes the number of


places where the two vectors differ
TRAINING ALGORITHMS FOR PATTERN
ASSOCIATION
• There are two algorithms for training of pattern association
nets
1. Hebb Rule
2. Outer Products Rule
• Hebb Rule
• This is widely used for finding the weights of an associative
memory neural net
• Here, the weights are updated until there is no weight
change
ALGORITHMIC STEPS FOR HEBB RULE

• STEP 0: Set all the initial weights to zero


( wij = 0, i = 1, 2...n; j = 1, 2,...m )
• STEP 1: For each training target input output vector pairs (s:t)
perform steps 2 to 4
• STEP 2: Activate the input layer units to current training input
( xi = si , i = 1, 2,...n)
• STEP 3: Activate the output layer units to the target output
( y j = t j , j = 1, 2,...m)
• STEP 4: Start the weight adjustments
wij (new) = wij (old) + x i  y j , i = 1, 2,...n; j = 1, 2,...m
OUTER PRODUCT RULE

• Here, we have
• Input: s = ( s1 , s2 ,..., si ,...sn )
• Output: t = (t1 , t2 ,...t j ,...tm )
• The outer product is defined as: The product of the two
matrices: S = sT and T = t
• So, we get the weight matrix W as:
 s1   s1t1 . s1t j s1tm 
.  
W=S.T =   =  
 si  . t1 . t j . tm   si t1 si t j si tm 
   
.  
 sn 
 snt1 sn t j sntm 

OUTER PRODUCT RULE CONTD…

• In case of a set of patterns, s(p): t(p), p=1,…P; we have


s( p) = ( s1 ( p),..., si ( p),..., sn ( p))
t( p) = (t1 ( p),..., t i ( p),..., t m ( p))

• For the weight matrix W = ( wij )


P
wij =  siT ( p).t j ( p), i = 1, 2,...n; j = 1, 2,...m.
p =1
AUTOASSOCIATIVE MEMORY NETWORK

• The training input and target output vectors are the same
• The determination of weights of the association net is called
storing of vectors
• The vectors that have been stored can be retrieved from
distorted (noisy) input if the input is sufficiently similar to it
• The net’s performance is based on its ability to reproduce a
stored pattern from a noisy input

• NOTE: The weights in the diagonal can be set to ‘0’. These


nets are called auto-associative nets with no self-connection
ARCHITECTURE OF AUTOASSOCIATIVE
MEMORY NETWORK
• A
x1 w11 y1
X1 Y1
wi1
wn1

w1i
xi yi
Xi wii Yi
wni

w1n
win

xn yn
Xn wnn Yn
TRAINING ALGORITHM

• This is same as that for the Hebb rule. Except that there are
same number of output units as the number of input units
• STEP 0: Initialize all the weights to ‘0’
• ( wij = 0, i = 1, 2,...n; j = 1, 2,...n )
• STEP 1: For each of the vector that has to be stored, perform
• steps 2 to 4
• STEP 2: Activate each of the input unit ( xi = si , i = 1, 2,...n )
• STEP 3: Activate each of the output units (y j = s j , j = 1,2,...n )
• STEP 4: Adjust the weights, i, j = 1,2,…n;
wij (new) = wij (old ) + xi . y j = wij (old ) + si .s j
TESTING ALGORITHM

• The associative memory neural network can be used to


determine whether the given input vector is a “known” one
or an unknown one
• The net is said to recognize a vector if the net produces a
pattern of activation as output, which is the same as the one
given as input
• STEP 0: Set the weights obtained from any of the two
methods described above
• STEP 1: For each of the testing input vector perform steps 2-4
• STEP 2: Set the activations of the input unit as equal to that of
the input vector
TESTING ALGORITHM CONTD…

• STEP 3: Calculate the net input to each of the output unit:


n
( yin ) j =  xi .wij , j = 1, 2,...n
i =1

• STEP 4: Calculate the output by applying the activation over


the net input:
1, if ( yin ) j  0;
y j = f (( yin ) j ) = 
−1, if ( yin ) j  0.
• THIS TYPE OF NETWORK IS USED IN SPEECH PROCESSING,
IMAGE PROCESSING, PATTERN CLASSIFICATION ETC.
EXAMPLE---AUTOASSOCIATIVE NETWORK

• Train the autoassociative network for input vector [-1 1 1 1]


and also test the network for the same input vector.
• Test the autoassociative network with one missing, one
mistake, two missing and two mistake entries in test vector.

• The weight matrix W is computed from the formula


P
W =  sT ( p).s ( p )
p =1

• Here, P = 1.
COMPUTATIONS
• S

 −1  1 −1 −1 −1
1  −1 1 1 1 
W =   . −1 1 1 1 =  
1  −1 1 1 1 
   
 
1  −1 1 1 1 
• Case-1: testing the network with the same input vector
• Test input: [-1 1 1 1]
• The weight obtained above is used as the initial weights
 1 −1 −1 −1
 −1 1 1 1 
( yin ) j = x. W = [−1 111].   = [−4 4 4 4]
 −1 1 1 1
 
 −1 1 1 1 
COMPUTATIONS

• Applying the activation function

1, if ( yin ) j  0;

y j = f (( yin ) j ) = 
−1, if ( yin ) j  0.

• Over net input, we get
y = [-1 1 1 1]
• Hence, the correct response is obtained
• TESTING AGAINST ONE MISSING ENTRY
• Case 1: [0 1 1 1] (First Component is missing)
• Compute the net input
COMPUTATIONS
• S

 1 −1 −1 −1
 −1 1 1 1 
( yin ) j = x. W = [0 111].   = [ −3 3 3 3]
 −1 1 1 1
 
 −1 1 1 1 

• Applying the activation function taken above, we get


• y = [-1 1 1 1]. So, the response is correct
• TESTING AGAINST ONE MISSING ENTRY
• Case 2: [-1 1 0 1] (Third Component is Missing)
• Compute the net input
COMPUTATIONS
• S

 1 −1 −1 −1
 −1 1 1 1 
( yin ) j = x. W = [−1 1 0 1].   = [ −3 3 3 3]
 −1 1 1 1
 
 −1 1 1 1 
• Applying the activation function taken above, we get
• y = [ -1 1 1 1]
• The response is correct.
• WE CAN TEST FOR OTHER MISSING ENTRIES SIMILARLY
• Testing the network against one mistake entry
• Case 1: Let the input be [-1 -1 1 1] (Second Entry is a Mistake)
COMPUTATIONS

• Compute the net input


 1 −1 −1 −1
 −1 1 1 1 
( yin ) j = x. W = [−1 − 111].   = [−2 2 2 2]
 −1 1 1 1
 
 −1 1 1 1 

• Applying the same activation function taken above,


• y = [-1 1 1 1]
• So, the response is correct
• Case 2: Let the input be [1 1 1 1] (First entry is mistaken)
COMPUTATIONS

• Compute the net input


 1 −1 −1 −1
 −1 1 1 1 
( yin ) j = x. W = [1 111].   = [−2 2 2 2]
 −1 1 1 1
 
 −1 1 1 1 
• Applying the activation function taken above
• y = [-1 1 1 1]
• So, the response is correct
• Testing the net against two missing entries
• Case 1: Let the input be [0 0 1 1] (First two entries are
missing)
COMPUTATIONS

• Compute the net input


 1 −1 −1 −1
 −1 1 1 1 
( yin ) j = x. W = [0 0 11].   = [−2 2 2 2]
 −1 1 1 1
 
 −1 1 1 1 

• Applying the activation function taken above


• y = [-1 1 1 1], which is the correct response
• Case 2: Let the input be [-1 0 0 1] (Second & Third inputs
are missing)
• Compute the net input
COMPUTATIONS
• S

 1 −1 −1 −1
 −1 1 1 1 
( yin ) j = x. W = [−1 0 0 1].   = [−2 2 2 2]
 −1 1 1 1
 
 −1 1 1 1

• Applying the input function taken above


• y = [-1 1 1 1], which is the correct response
• Testing the net against two mistaken entries
• Let the input be [-1 -1 -1 1] (Second & Third inputs are
missing)
COMPUTATIONS

• Compute the net input


 1 −1 −1 −1
 −1 1 1 1 
( yin ) j = x. W = [−1 − 1 − 11].   = [0 0 0 0]
 −1 1 1 1
 
 −1 1 1 1
• Applying the activation function taken above, we get
• y = [0 0 0 0]. Which is not correct.
• SO, THE NETWORK FAILS TO RECOGNISE INPUTS WITH TWO
MISTAKES

• NOTE: WE HAVE TO CHECK FOR ALL POSSIBLE INPUTS UNDER


EACH CASE TO HAVE A POSITIVE CONCLUSION
HETEROASSOCIATIVE MEMORY NETWORK

• The training input and target output are different


• The weights are determined in such a way that the net can
store a set of ‘P’ pattern associations
• Each input vector s(p) has ‘n’ components and each output
vector t(p) has ‘m’ components (p = 1, 2, ….P)
• Determination of weights: Hebb Rule or Delta Rule is used

• THE NET FINDS AN APPROPRIATE OUTPUT VECTOR


CORRESPONDING TO AN INPUT VECTOR, WHICH MAY BE
ONE OF THE STORED PATTERNS OR A NEW PATTERN
ARCHITECTURE OF HETEROASSOCIATIVE
MEMORY NETWORK
• A

x1 w11 y1
X1 Y1
wi1
wn1

w1i
xi yi
Xi wii Yi
wni

w1m
wim

xn ym
Xn wnm Ym

Differs from Auto- associative network in the number of output units


TESTING ALGORITHM FOR HETEROASSOCIATIVE
MEMORY NETWORK
• STEP 0: Initialize the weights from the training algorithm
• STEP 1: Perform steps 2 – 4 for each input vector presented
• STEP 2: Set the activation for the input layer units equal to
that of the current input vector given, xi
• STEP 3: Calculate the net input to the output units using
n
( yin ) j =  xi .wij , j = 1, 2,...m
i =1

• STEP 4: Determine the activations for the output units as:


1, if ( yin ) j  0;

y j = 0, if ( yin ) j = 0; Differs from auto-associative here

 −1, if ( yin ) j  0.
HETEROASSOCIATIVE MEMORY NETWORK
CONTD…
• There exist weighted interconnections between the input and
output layers
• The input and output layers are not correlated with each
other
• We can also use the following binary activation function

1, if ( yin ) j  0;

yj = 
−1, if ( yin ) j  0.

EXAMPLE-HETEROASSOCIATIVE MEMORY
NETWORKS
• Train a heteroassociative memory network using Hebb rule
to store input row vector s = ( s1 , s2 , s3 , s4 ) to the output
vector t = (t1 , t2 ) as given in the table below:

Input targets s1 s2 s3 s4 t1 t2
1st 1 0 1 0 1 0
2nd 1 0 0 1 1 0
3rd 1 1 0 0 0 1
4th 0 0 1 1 0 1
THE NEURAL NET
• s

x1
X1 w11
y1
w12 Y1
x2 w21
X2 w22

w31
x3 w32 y2
X3 Y2
w41
x4 w42
X4
COMPUTATIONS

• We use Hebb rule to determine the weights

• The initial weights are all zeros

• For the first pair


x1 = 1, x2 = 0, x3 = 1, x4 = 0, y1 = 1, y2 = 0
• Set the input and output pairs

• Weight Updation: Formula


• wij (new) = wij (old) + x i . y j
UPDATED WEIGHTS

w11 (new) = w11 (old) + x1 . y1 = 0 + 1 1 = 1


w12 (new) = w12 (old) + x1 . y2 = 0 + 0  1 = 0
w21 (new) = w21 (old) + x 2 . y1 = 0 + 1 1 = 1
w22 (new) = w22 (old) + x 2 . y2 = 0 + 1 1 = 0
w31 (new) = w31 (old) + x 3 . y1 = 0 + 1 0 = 0
w32 (new) = w32 (old) + x 3 . y2 = 0 + 0  0 = 0
w41 (new) = w41 (old) + x 4 . y1 = 0 + 1 0 = 0
w42 (new) = w42 (old) + x 4 . y2 = 0 + 0  0 = 0
UPDATED WEIGHTS
• 2

𝑤11 (𝑛𝑒𝑤) = 𝑤11 (𝑜𝑙𝑑) + 𝑥1 . 𝑦1 = 0 + 1 × 1 = 2


𝑤12 (𝑛𝑒𝑤) = 𝑤12 (𝑜𝑙𝑑) + 𝑥1 . 𝑦2 = 0 + 0 × 1 = 0
𝑤21 (𝑛𝑒𝑤) = 𝑤21 (𝑜𝑙𝑑) + 𝑥2 . 𝑦1 = 0 + 1 × 1 = 0
𝑤22 (𝑛𝑒𝑤) = 𝑤22 (𝑜𝑙𝑑) + 𝑥2 . 𝑦2 = 0 + 1 × 1 = 0
𝑤31 (𝑛𝑒𝑤) = 𝑤31 (𝑜𝑙𝑑) + 𝑥3 . 𝑦1 = 0 + 1 × 0 = 1
𝑤32 (𝑛𝑒𝑤) = 𝑤32 (𝑜𝑙𝑑) + 𝑥3 . 𝑦2 = 0 + 0 × 0 = 0
𝑤41 (𝑛𝑒𝑤) = 𝑤41 (𝑜𝑙𝑑) + 𝑥4 . 𝑦1 = 0 + 1 × 0 = 1
𝑤42 (𝑛𝑒𝑤) = 𝑤42 (𝑜𝑙𝑑) + 𝑥4 . 𝑦2 = 0 + 0 × 0 = 0
COMPUTATIONS

• For the third pair


• The initial weights are the outputs from the above updation
• The input-output vector pair is
x1 = 1, x2 = 1, x3 = 0, x4 = 0, y1 = 0, y2 = 1
• Set the input and output pairs
• The weight updation formula is same as above
• The weights change in only two out of 8 cases:
w12 (new) = w12 (old) + x1 . y2 = 0 + 1 1 = 1
w22 (new) = w22 (old) + x 2 . y2 = 0 + 1 1 = 1
UPDATED WEIGHTS
• 3

𝑤11 (𝑛𝑒𝑤) = 𝑤11 (𝑜𝑙𝑑) + 𝑥1 . 𝑦1 = 0 + 1 × 1 = 2


𝑤12 (𝑛𝑒𝑤) = 𝑤12 (𝑜𝑙𝑑) + 𝑥1 . 𝑦2 = 0 + 0 × 1 = 1
𝑤21 (𝑛𝑒𝑤) = 𝑤21 (𝑜𝑙𝑑) + 𝑥2 . 𝑦1 = 0 + 1 × 1 = 0
𝑤22 (𝑛𝑒𝑤) = 𝑤22 (𝑜𝑙𝑑) + 𝑥2 . 𝑦2 = 0 + 1 × 1 = 1
𝑤31 (𝑛𝑒𝑤) = 𝑤31 (𝑜𝑙𝑑) + 𝑥3 . 𝑦1 = 0 + 1 × 0 = 1
𝑤32 (𝑛𝑒𝑤) = 𝑤32 (𝑜𝑙𝑑) + 𝑥3 . 𝑦2 = 0 + 0 × 0 = 0
𝑤41 (𝑛𝑒𝑤) = 𝑤41 (𝑜𝑙𝑑) + 𝑥4 . 𝑦1 = 0 + 1 × 0 = 1
𝑤42 (𝑛𝑒𝑤) = 𝑤42 (𝑜𝑙𝑑) + 𝑥4 . 𝑦2 = 0 + 0 × 0 = 0
COMPUTATIONS

• For the fourth pair


• The initial weights are the outputs from the above updation
• The input-output vector pair is
x1 = 0, x2 = 0, x3 = 1, x4 = 1, y1 = 0, y2 = 1
• Set the input and output pairs
• The weight updation formula is same as above
• The weights change in only two of the 8 cases:
w32 (new) = w32 (old) + x1 . y2 = 0 + 1 1 = 1
w42 (new) = w42 (old) + x 4 . y2 = 0 + 1 1 = 1
COMPUTATIONS

• The final weights after all the input/output vectors are used
are
 w11 w12   2 1
w w22   0 1
W =  21 =
 w31 w32  1 1
   
 w41 w42  1 1
BIDIRECTIONAL ASSOCIATIVE MEMORY

• It was developed by Kosko in 1988


• It performs both forward and backward searches
• It uses Hebb rule
• It associates patterns from set A to set B and vice versa
• It responds for input from both layers
• It is a hetero-associative pattern matching network that
encodes binary or bipolar patterns using Hebbian learning
rule
TYPES OF BAM

• There are two types of BAMs


➢ Discrete
➢ Continuous
• The characterization depends upon the activation functions
used
• Otherwise, the architecture is same
• Discrete BAM:
• Binary or Bipolar activation functions are used
• Continuous BAM:
• Binary sigmoid or Bipolar sigmoid functions are used
THE BAM ARCHITECTURE
• A

Layer X W Layer Y
x1 w11 y1
X1 Y1
wi1
wn1

w1i
xi yi
Xi wii Yi
wni

w1m
wim

xn ym
Xn wnm Ym
WT
BAM ARCHITECTURE

• Consists of two layers of neurons


• These layers are connected by directed weighted path
interconnections

• The network dynamics involves two layers of interaction


• The signals are sent back and forth between the two layers
until all neurons reach equilibrium

• The weighs associated are bidirectional


• It can respond to inputs in either layer
BAM ARCHITECTURE CONTD…

• The weight matrix from one direction (say A to B) is W


• The weight matrix from the other direction (B to A) is W T

• A definite non-zero threshold is assigned


• The activation function is a step function

• When compared to binary vectors the bipolar vectors


improve the performance of the net to a large extent
THE WEIGHT MATRICES
 w11 . w1 j . w1m 
 . . . . . 

W =  wi1 . wij . wim 
 
 . . . . . 
 wn1 . wnj . wnm 

 w11 . wi1 . wn1   w11 . w1i . w1n 


 . . . . .   . . . . . 

W =  w1 j
T
. wij . wnj  =  w j1 . w ji . w jn 
   
 . . . . .   . . . . . 
 w1m . wim . wnm   wm1 . wmi . wmn 
DISCRETE BIDIRECTIONAL ASSOCIATIVE
MEMORY
• The diagram is as shown earlier
• When the memory neurons are being activated by putting an
initial vector at the input of a layer, the network evolves a two-
pattern stable state
• Each pattern at the output of one layer
• The network involves two layers of interaction between each
other
• The two bivalent forms (Binary and Bipolar) of BAM are related
to each other
• The weights in both the cases are found as the sum of the
outer products of the bipolar form of the given training vector
pairs
DETERMINATION OF WEIGHTS

• Let the ‘P’ number of input/target pairs of vectors be denoted


by (s(p), t(p)), p = 1,2,…P
• Then the weight matrix to store a set of input/target vectors
P=1, 2,…P
s ( p ) = ( s1 ( p ),..., si ( p ),..., sn ( p ))
t(p) = (t1 ( p ),..., t j ( p ),..., t m ( p ))

• can be determined by Hebb rule for determining weights for


associative networks P

• In case of binary input vectors wij =  [2si ( p) − 1].[2t j ( p) − 1]


p =1
P
• In case of bipolar input vectors wij =  siT ( p).t j ( p)
p =1
ACTIVATION FUNCTIONS FOR BAM

• The step activation function with nonzero threshold is used


as the activation function for discrete BAM network
• The activation function is based on whether the input vector
pairs (at the input and target vector) are binary or bipolar
• ACTIVATION FUNCTION FOR ‘Y’ LAYER
• 1. If both the layers have binary input vectors
1, if ( yin ) j  0;

y j =  y j , if ( yin ) j = 0;

0, if ( yin ) j  0.
ACTIVATION FUNCTIONS FOR BAM

2. When the input vector is bipolar


1, if ( yin ) j   j ;

y j =  y j , if ( yin ) j =  j ;

−1, if ( yin ) j   j .
• INPUT FUNCTION FOR ‘X’ LAYER
1. When the input vector is binary
1, if (x in )i  0;

xi =  xi , if (x in )i = 0;
0, if (x )  0.
 in i
ACTIVATION FUNCTIONS FOR BAM

2. When the input vector is bipolar

1, if (x in )i  i ;

xi =  xi , if (x in )i = i ;
−1, if (x )   .
 in i i
TESTING ALGORITHM FOR DISCRETE BAM

• This algorithm is used to test the noisy patterns entering into


the network
• Basing upon the training algorithm, weights are determined
• Using the weights the net input is calculated for the given test
pattern
• Activations are applied over it to recognize the test patterns
• ALGORITHM:
• STEP 0: Initialize the weights to store ‘p’ vectors
• STEP 1: Perform Steps 2 to 6 for each testing input
• STEP 2: (i) Set the activations of the X layer to the current
input pattern, that is presenting the input pattern x to X.
TESTING ALGORITHM FOR DISCRETE BAM
CONTD…
• (ii) Set the activations of the ‘Y’ layer similarly by presenting
the input pattern ‘y’.
• (iii) Though it is bidirectional memory, at one time step,
signals can be sent from only one layer. So, one of the input
patterns may be zero vector
• STEP 3: Perform steps 4 to 6 when the activations are not
converging
• STEP 4: Update the activations of units in Y layer. Calculate the
net input by n
( yin ) j =  xi .wij
i =1
TESTING ALGORITHM FOR DISCRETE BAM
CONTD…
• Applying activations, we obtain y j = f (( yin ) j )
• Send this signal to the X layer
• STEP 5: Update the activations of units in X layer
• Calculate the net input m
( xin )i =  y j .wij
j =1
• Apply the activations over the net input xi = f (( xin )i )
• Send this signal to the Y layer
• STEP 6: Test for convergence of the net. The convergence
occurs if the activation vectors x and y reach equilibrium.
• If this occurs stop, otherwise continue.
CONTINUOUS BAM

• It transforms the input smoothly and continuously in the


range 0-1 using logistic sigmoid functions as the activation
functions for all units
• This activation function may be binary sigmoidal function or
it may be bipolar sigmoidal function
• When a bipolar sigmoidal function with a high gain is chosen,
the continuous BAM might converge to a state of vectors
which will approach to the vertices of a cube
• In such a circumstance it acts like a discrete BAM
CONTINUOUS BAM CONTD…

• In case of binary inputs, (s(p), t(p)), p = 1,2,…P; the weight is


determined by the formula
P
wij =  [2si ( p) − 1].[2t j ( p) − 1]
p =1

• The activation function is the logistic sigmoidal function


• If it is binary logistic function then the activation function is
given by
1
f (( yin ) j ) = −( y )
1 + e in j
CONTINUOUS BAM CONTD…

• If the activation function used is a bipolar function then it is


given by
−( y )
2 1 − e in j
f (( yin ) j ) = − ( yin ) j
−1 = −( y )
1+ e 1 + e in j

• These activations are applied over the net input to calculate


the output. The net input can be calculated with a bias as
n
( yin ) j = b j +  xi .wij
i =1
• All these formulae are applicable to the X layer also
EXAMPLE-5

• Construct and test a BAM network to associate letters E and F


with single bipolar input-output vectors. The target output
for E is [-1, 1] and F is [1, 1]. The display matrix size is 5 x 3.
The input patterns are:
     
 (-1)   
   (1) 
 
   
• Target out puts are [-1, 1] [1, 1]
COMPUTATIONS

• The input patterns are

E 1 1 1 1 -1 -1 1 1 1 1 -1 -1 1 1 1
F 1 1 1 1 1 1 1 -1 -1 1 -1 -1 1 -1 -1

• The output and weights are

E -1 1 W1
F 1 1 W2
COMPUTATIONS

• Since we are considering bipolar input and outputs, the


weight matrix is computed by using the formula:

W =  sT ( p ). t ( p )

• There are two weight components W1 and W2

W1 = [1 111 − 1 − 11111 − 1 − 1111]T [ −11]


W2 = [1 111111 − 1 − 11 − 1 − 11 − 1 − 1]T [11]
COMPUTATIONS
 −1 1 1 1
• So,  −1 1 1 
 1  
 −1 1 1 1
   
 −1 1
1 1
1 1 1
−1  
 
1 −1 1 1
 −1 1 1
1  
 
• W1 =  −1 1 and W2 =  −1 −1
   
 −1 1  −1 −1
 −1 1 1 1
   
1 −1  −1 −1
1  −1 −1
−1  
 
 −1 1 1 1
 −1  −1 −1
1  
 
 −1 1   −1 −1
COMPUTATIONS
0 2
0 2
• The total weight matrix is  
0 2
 
 0 2 
2 0
 
2 0
0 2
 
W = W1 + W2 =  −2 0 
 
 − 2 0 
 0 −2 
 
 0 −2 
 0 −2 
 
0 2
 −2 0 
 
 −2 0 
TESTING THE NETWORK

• We test the network with test vectors E and F


• Test pattern E 0 2
0 2 

0 2
 
0 2
2 0
 
 −2 0
0 2
 
yin = [1111 − 1 − 11111 − 1 − 1111]  −2 0  = [ −12 18]
 
 −2 0
0 2
 
0 −2 
• Applying the activations, we get 0 −2 
 
• Y = [-1 1], which is the correct response 0 2
 −2 0
 
 −2 0 
TESTING THE NETWORK
0 2
• Test pattern F 0 2
 
0 2
 
 0 2 
2 0
 
 −2 0 
0 2
 
yin = [1111111 − 1 − 11 − 1 − 11 − 1 − 1]  −2 0  = [12 18]
 
 −2 0 
0 2
 
 0 −2 
• Applying activations over the net input  0 −2 
 
we get y = [1 1], which is correct 0 2
 −2 0 
 
 −2 0 
BACKWARD MOVEMENT

• Y vector is taken as the input

• The weight matrix here is the transpose of the original


• So,

 0 0 0 0 2 2 0 −2 −2 0 0 0 0 −2 −2 
T
=
0 
W
2 2 2 2 0 0 2 0 0 2 −2 −2 2 0
TESTING THE NETWORK

• For the test pattern E, the net input is [-1 1]

yin = x. W T
0 0 0 0 2 2 0 −2 −2 0 0 0 0 −2 −2 
= [−11]  
 2 2 2 2 0 0 2 0 0 2 −2 −2 2 0 0 
yin =  2 2 2 2 −2 −2 2 2 2 2 −2 −2 2 2 2
• Applying the activation functions, we get
y = 1 1 1 1 −1 −1 1 1 1 1 −1 −1 1 1 1

• Which is the correct response


TESTING THE NETWORK

• For the pattern F:


• Here, the input is [1 1]
yin = x. W T
 0 0 0 0 2 2 0 −2 −2 0 0 0 0 −2 −2 
= [11] 
2 2 2 2 0 0 2 0 0 2 −2 −2 2 0 0 

yin =  2 2 2 2 2 2 2 −2 −2 2 −2 −2 2 −2 −2 

• Applying the activation function we get


y = 1 1 1 1 1 1 1 −1 −1 1 −1 −1 1 −1 −1
• This is the correct response
HOPFIELD NETWORK
HOPFIELD NETWORK

• Introduced by J. J. Hopfield in the year 1982


• It has got many useful applications in associative memory
and various optimisation problems
• There are two types of Hopfield networks:
• Discrete Hopfield networks
• Continuous Hopfield networks
• Biological neurons are asynchronous (not existing or
occurring at the same time ) by nature
• Hopfield networks is developed to confirm this nature
• Only one unit updates its activation at a time
HOPFIELD NETWORK CONTD…

• Each unit is found to continuously receive an external signal


along with the signals it receives from the other units in the
net
• When a single layer recurrent network is performing a
sequential updating process, an input pattern is first applied
to the network
• The network output is found to be initialised accordingly
• After that the input pattern is removed and the output that is
initialised becomes the new updated input through the
feedback connections
• The first updated input forces the first updated output, which
in turn acts as the second updated input through the
feedback and results in second updated output
HOPFIELD NETWORK CONTD…

• This transition process continues until no new updated


responses are produced and the network reaches its
equilibrium
DISCRETE HOPFIELD NETWORK

• It is network which is:


• Auto-associative
• Fully interconnected
• Single layer
• Feed back
• It is called as recurrent network
• In it each processing element has two outputs
• One output is inverting and the other one non-inverting
• The output from each processing element is fed back to the
input of other processing elements but not to itself
DISCRETE HOPFIELD NETWORK

• It takes two-valued inputs: {0, 1} or {-1, +1}


• The use of bipolar inputs makes the analysis easier
• The net uses symmetrical weights with no self-connections
• That is 𝑤𝑖𝑗 = 𝑤𝑗𝑖 and 𝑤𝑖𝑖 = 0, ∀𝑖
• Key Points:
• Only one unit updates its activation at a time
• Each unit is found to continuously receive an external signal
along with the signals it receives from the other units in the
net
• When a single-layer recurrent network is performing a
sequential updating process, an input pattern is first applied
to the network and the output is initialized accordingly
ARCHITECTURE OF A DISCRETE HOPFIELD NET
• g
x1 x2 xi xn

wi1 win wn1


w21 w2n
w2i wi2 wn2
w1n
w1i wni
w12

Y1 Y2 Yi Yn

y1 y2 yi yn
TRAINING ALGORITHM FOR DISCRETE
HOPFIELD NETWORK
• Hopfield’s first description used binary input vectors and only
later on bipolar input vectors were used
• For storing a set of binary patterns s(p), p = 1, 2,…P, where
s(p) = 𝑠1 (𝑝), 𝑠2 (𝑝), . . . 𝑠𝑛 (𝑝) , the weight matrix W is given
by
𝑃
• 𝑤𝑖𝑗 = ෍ 2𝑠𝑖 (𝑝) − 1][2𝑠𝑗 (𝑝) − 1 , for 𝑖 ≠ 𝑗
𝑝=1
• For storing a set of bipolar inputs the weight matrix W is given
by
𝑃
• 𝑤𝑖𝑗 = ෍ 𝑠𝑖 (𝑝). 𝑠𝑗 (𝑝൯ , for 𝑖 ≠ 𝑗
𝑝=1
TRAINING ALGORITHM FOR DISCRETE
HOPFIELD NETWORK
• STEP 0: Initialize the weights to store patterns, i. e. weights
obtained from training algorithm using Hebb rule
• STEP 1: When the activations of the net are not converged,
perform steps 2 - 8
• STEP 2: Perform steps 3 -7 for each input vector X
• STEP 3: Make the initial activation of the net equal to the
external input vector X (i. e. 𝑦𝑖 = 𝑥𝑖 (𝑖 = 1,2. . . 𝑛)
• STEP 4: Perform steps 5 – 7 for each unit 𝑌𝑖
• STEP 5: Calculate the net input of the network:

• 𝑦𝑖𝑛 𝑖 = 𝑥𝑖 + ෍ 𝑦𝑗 𝑤𝑗𝑖
𝑗
TRAINING ALGORITHM FOR DISCRETE
HOPFIELD NETWORK
• STEP 6: Apply the activations over the net input to calculate
the output:
1, if ( yin )i  i ;

yi =  yi , if ( yin )i = i ;
0, if ( y )   .
 in i i

• where 𝜃𝑖 is the threshold and is normally taken as zero


• STEP 7: Now feed back the obtained output 𝑦𝑖 to all other
units. Thus the activation vectors are updated
• STEP 8: Finally test the network for convergence
CONTINUOUS HOPFIELD NETWORK

• It is a discrete Hopfield network in which the time is assumed


to be a continuous variable
• The nodes of this network have a continuous graded output
rather than a two-state binary output
• The asynchronous updation of the units allows a function
called as energy function or Liapunov function, for the net
• The existence of this function enables us to prove that the net
will converge to a stable set of activations
• The energy of the network decreases continuously with time

You might also like