UNIT II ASSOCIATIVE AND

UNSUPERVISEDLEARNING NETWORKS

TRAINING ALGORITHMS FOR PATTERN ASSOCIATION-AUTO-ASSOCIATIVE

MEMORY NETWORK-HETERO-ASSOCIATIVE MEMORY NETWORK-
BIDIRECTIONAL ASSOCIATIVE MEMORY (BAM)-HOPFIELD NETWORKS-
ITERATIVE AUTO-ASSOCIATIVE MEMORY NETWORKS-TEMPORAL
ASSOCIATIVE MEMORY NETWORK-FIXED WEIGHT COMPETITIVE NETS-
KOHONEN SELF-ORGANIZING FEATURE MAPS-LEARNING VECTOR
QUANTIZATION-COUNTER PROPAGATION NETWORKS-ADAPTIVE
RESONANCE THEORY NETWORK

Topic 2 : Bidirectional Associative

Memory (BAM)
 Bidirectional Associative
Memory (BAM):Introduction 1/3
It is a recurrent neural network (RNN) of a
special type,
initially proposed by Bart Kosko, in the early
1980s,
to overcome several known drawbacks of
the auto-associative Hopfield network, and
ANNs,
that learn associations of data from
continuous training.
2
 Bidirectional Associative
Memory (BAM):Introduction 2/3
It is a hetero-associative memory
It provides an ability to store the associated data,
regardless of its type and structure, without continuous
learning.
The associations of data are simultaneously stored, only
once, prior to recalling them from the BAM’s memory.
The BAM outruns the existing ANNs, providing significantly
better performance while recalling the associations.
BAM perfectly solves the known bitwise XOR problem,
compared to the existing ANNs,
which makes it possible to store the data, encoded to
3
 Bidirectional Associative Memory (BAM):Introduction 3/3
 It recall the associations for data, assigned to its either
inputs or outputs, bidirectionally.
 It retrieves the correct associations for even incomplete or
distorted input data.
 It is efficient when deployed as a part of an AI-based
decision-making process,
 Solution to a specific data analysis problem, based on
various associations of many interrelated data.
 The BAM model can be used for applications, such as:
 Classification and clustering data
 Incomplete data augmentation
 Recovering damaged or corrupted data 4
 Bidirectional Associative Memory (BAM)
For example, the prediction of words missed out from incomplete texts
with ANN,
In this case,
 using the BAM provides an ability to store and recall all
possible associations of the data entities,
 such as words-to-sentences, sentences-to-
words, words-to-words, and sentences-to-
sentences, and vice versa.
 This, in turn, notably increases the quality of prediction.
 Associating various entities of data multi-directionally is also
5
 Bidirectional Associative Memory (BAM)
 The memory capacity of each ANN’s layer is bound to the size of the
largest floating-point type: float64.
 For instance, the capacity of a single ANN layer of shape (100×100)
is only 1024 bytes.
 the number of the ANN’s layers, the number of neurons, in each of
its layers, must be increased to have the ability to store the
associations and recall them multi-directionally.
 So, It negatively impacts the ANN-based memory latency, since its
learning and prediction workloads are growing, proportionally to
the sizes of the ANN.
 Despite this, the high performance of learning and prediction of
the BAM provides an ability to allocate as large amounts of
memory space as required, beyond the conventional ANN’s memory
capacity limits.
 As well, it’s possible to aggregate multiple BAMs into the 6
 Associations Stored In Memory
 The BAM learns the associations of various data,
converted to bipolar patterns.
 A bipolar pattern is a special case of binary vector,
the values of 1’s and -1’s, respectively.
 Multiple patterns (columns) are arranged into the
2D-pattern maps, along the y-axis of a matrix.
 The pattern maps, each column corresponds to a
specific input and output data item, stored in
the BAM’s memory:
7
Associations Stored In Memory

Encoding Data Into Bipolar 2D-Patterns Map of Shape (s x n)

8
 Associations Stored In Memory
The BAM model simultaneously learns the associated data from
pattern maps, assigned to its corresponding inputs and outputs,
respectively:

Input (X) And Output (Y) Pattern Maps Assigned To The BAM Model 9
 Associations Stored In Memory
 In the above figure, the BAM model learns the associations
of data, encoded into the pattern maps 𝙓 and 𝒀 of the
shapes (𝙨 x 𝙣) and (𝙨 x 𝙢), respectively.
 While learning, the values from the input and output patterns

𝙮ⱼ∈ 𝒀 of the BAM’s model.

are assigned to the corresponding inputs 𝙭ᵢ ∈𝙓 and outputs

 In turn, both input and output pattern maps (columns)

must have an equal number of patterns (𝙨 ).
 Since the BAM is a hetero-associative memory model, it has a
different number of inputs (𝙣 ) and outputs (𝙢 ), and,

maps, 𝙓, and 𝒀, correspondingly.

thus, the different number of rows in those pattern
1
 BAM’s Topology And Structure
 Generally, the BAM is a
primitive neural
network (NN),
consisting of only
input and output
layers, interconnected
by the synaptic
weights:

The BAM Model’s NN-Topology (4 x 3)

1
 BAM’s Topology And Structure
 In the figure, above, the neurons (𝒏) of the input layer perform no
output computation, feedforwarding the inputs 𝑿 to the memory
cells (𝒎) in the BAM’s output layer.
 Its sizes basically depend on the quantities of inputs and outputs of the
BAM (e.g., the dimensions of the input and output pattern maps).
 The NN, the shape (4 x 3), consists of 4 neurons and 3 memory
cells in its input and output layers, respectively.
 Unlike the conventional ANNs, the output layer of BAM consists of
memory cells that perform the BAM’s outputs computation.
 Each memory cell computes its output 𝒀, based on the multiple
inputs 𝑿, forwarded to the cell by the synaptic weights 𝑾, each one
is of specific strength.
1
BAM’s Topology And Structure

The Outputs (Y) Of The BAM’s Memory Cells

1
 BAM’s Topology And Structure
 The BAM’s outputs are computed as the weighted sum
of all BAM’s inputs 𝑾ᵀ𝙓, applied as an argument to the
bipolar threshold function.
 The positive (+) or negative (-) value (e.g., 1 or -1)
of each output 𝒀 is always proportional to the
magnitude of the sum of the cell’s weighted inputs.
 All synaptic weights, interconnecting the input neurons
and memory cells store the fundamental memories,
 learned by the BAM, based on the Hebbian supervised
learning algorithm, discussed, below.
1
 BAM’s Learning Algorithm
 Algebraically, the ability of BAMs to store and recall
associations solely relies on the bidirectional property
of matrices, studied by T. Kohonen and J. Anderson, in the
mids of 1950s:
 The inner dot product of two matrices 𝑿 and 𝒀, which
gives a correlation matrix 𝑾 is said to be
bidirectionally stable, as the result of recollection and
feedback of 𝑿 and 𝒀 into 𝑾 .
 Generally, it means that the corresponding rows and
columns of both matrices 𝑿 and 𝒀, as well as their
scalar vector products, equally contribute to the values
1
 In this case, 𝑾 is a matrix, which infers the correlation of the 𝑿ₖ
BAM’s Learning Algorithm
or 𝒀ₖ vectors.
 An entire learning process is much similar to
 keeping (𝙬ᵢⱼ≠ 0) or
 discarding (𝙬ᵢⱼ=0) the specific weights,
 that interconnect input neurons and memory cells of the
BAM.
 The values of synaptic weights 𝙬ᵢⱼ∈ ** 𝑾 can be either
greater than, less, or equal to 0.
 The BAM keeps only those synaptic weights, which values
are either positive or negative.
 1
 BAM’s Learning Algorithm
 The bidirectional matrix property was formulated by Donald
Hebb as the famous supervised learning rule,
 which is fundamental for the Hopfield network, and other
associative memory models.
 According to the Hebbian algorithm, all associated patterns

initializing the matrix 𝑾ₙₓₘ of the synaptic weights (e.g.,

are stored in the BAM’s memory space, only once, while

memory storage space).

 The matrix 𝑾ₙₓₘ is obtained by taking an inner dot product of
the input 𝑿ₚₓₙ and output 𝒀ₚₓₘ pattern maps (i.e.,
matrices), respectively: 1
 Alternatively, the weights matrix 𝑾ₙₓₘ can be
BAM’s Learning Algorithm

the corresponding patterns 𝙭ₖ and 𝙮ₖ, from the 𝑿ₚₓₙ

obtained as the sum of outer Kronecker products of

, 𝒀ₚₓₘ matrices.

 Unlike the inner dot product of 𝑿 and 𝒀, the outer

products of (𝙭ₖ,𝙮ₖ) ∈ 𝑿,𝒀 vectors yield matrices 𝙬ₖ
∈ 𝑾 (i.e., the fundamental memories).
 Finally, all of the matrices 𝙬ₖ are combined into a
single matrix 𝑾, by obtaining their sum,
1
 BAM’s Learning Algorithm
An entire process of the BAM’s synaptic weights
initialization is illustrated in the figure, below:

Learning BAM Model Based On The Hebbian Algorithm

1
 However, the correlation matrix 𝑾 provides the most stable and
BAM’s Learning Algorithm
consistent memory storage, for the patterns, which are the orthogonal
vectors.
 The product of orthogonal vectors in 𝑿,𝒀 gives the symmetric
correlation matrix 𝑾, in which the fundamental memories of
associations, learned from the pattern maps 𝑿 and 𝒀, respectively.
 Thus, it’s highly recommended to perform the orthogonalization of 𝑿 and
𝒀 pattern maps, prior to training the model.
 Since the 𝑿 and 𝒀 are bipolar, the orthogonalization of 𝒀 can be easily
done as:
𝒀 = -𝑿.
 By taking the negative -𝑿, the 𝒀 patterns will have the same values
as 𝑿, the sign (+/-) of which is changed to an opposite. 2
 Recalling Patterns From The Memory
 The input and output pattern stored in
the BAM’s memory.
 The associations for specific patterns are
recalled by computing the BAM’s full
output

pattern 𝑿 was retrieved.

 until the correct association for an input

each of the iterations 𝞭 -iterations, 𝒌=𝟏..𝞭,

 To compute the full memory output, in

 the BAM’s inputs 𝙭ᵢ of the pattern 𝙓 are

multiplied by the corresponding synaptic 2
 Recalling Patterns From The Memory
The bipolar threshold function 𝙁
 positive (+) or 0 inputs is 1,
 and negative (-), unless otherwise.

 Bipolar Threshold Function Y=F(X)

 The bipolar threshold function’s plot diagram is
shown, below:

Bipolar Threshold Activation Function

2
 Recalling Patterns From The Memory
 The memory cells in the output layer also compute the

𝙓 ₖ₊₁ for the next iteration 𝙠+𝟏.

new inputs

 Then, it performs a check on whether the current 𝙓 and new

inputs 𝙓 ₖ₊₁ are not equal (𝙓 ≠𝙓 ₖ₊₁).

compute the output 𝒀 in the next iteration.

 If not, it assigns the new inputs to the BAM, proceeding to

 Otherwise, if 𝙓=𝙓ₖ₊₁, then it means that the output 𝒀 is the

correct association for the input 𝙓 , and the algorithm has
converged, returning the pattern 𝒀 from the procedure.

either inputs 𝙓 and outputs 𝒀, by computing the memory

 Also, the BAM provides an ability to recall the associations for

outputs, bidirectionally, based on the same algorithm.

2
 The patterns recalling process is illustrated in
the figure

Memory Cell Output (Y) Computation

course title
2
 An algorithm for recalling the association
from the BAM’s memory is listed below:
For each of the 𝞭-iterations,𝟏 ≤ 𝞭 ≤ 𝒕, do the following:
1. Initialize the BAM’s inputs with the input vector 𝑿₁←𝑿.
2. Compute the BAM output vector 𝒀ₖ , as an inner dot product of the weights matrix
ᵀ transpose, and the input vector 𝑿ₖ, for the 𝒌-th iteration:

3. Obtain the new input vector 𝑿ₖ₊₁ for the next (𝒌 ＋𝟏 )-th iteration, such as:

4. Check if new and existing vectors 𝑿ₖ₊₁≠𝑿ₖ, are NOT the same:
5. If not, return to step 1, to compute the output 𝒀ₖ₊₁, for the (𝒌 ＋𝟏 )-th iteration, or
proceed to the next step 5, unless otherwise.
6. Return the output vector 𝒀←𝒀ₖ from the procedure, as the correct association for
the input 𝑿 vector.
7. Proceed with steps 2–4, until convergence. 2
Memory Evaluation (Testing)

It builds the BAM model of a specific shape

(patterns × neurons ×memory cells), and
generates 1D-array, consisting of the 1 and -
1 values.
Then, it reshapes the array into the input 2D-
patterns map 𝑿.
To obtain the patterns map 𝒀, it performs the
orthogonalization of 𝑿, such as: 𝒀=−𝑿. 2
 Memory Evaluation (Testing)

associations of patterns in 𝑿 and 𝒀 into the BAM’s

Next, the correlation matrix is computed to store the

memory.
Perform the model evaluation for each input pattern
𝙭ᵢ ∈ 𝙓 it recalls an association from the BAM’s
memory, bidirectionally.
When an association y⃗ₚ for the input pattern x⃗ was
recalled from the memory, it does the consistency
check whether the output y⃗ₚ target y⃗ pattern are
identical,
displaying the results for each of the input patterns 𝙭ᵢ
2
THANK YOU