CHAPTER 5

UNSUPERVISED
LEARNING NETWORKS

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
UNSUPERVISED LEARNING
 No help from the outside.

 No training data, no information available on the desired output.

 Learning by doing.

 Used to pick out structure in the input:

• Clustering,
• Reduction of dimensionality  compression.

 Example: Kohonen’s Learning Law.

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
FEW UNSUPERVISED LEARNING NETWORKS
There exists several networks under this category, such as

 Max Net,

 Mexican Hat,

 Kohonen Self-organizing Feature Maps,

 Learning Vector Quantization,

 Counterpropagation Networks,

 Hamming Network,

 Adaptive Resonance Theory.

“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
COMPETITIVE LEARNING
 Output units compete, so that eventually only one neuron (the one
with the most input) is active in response to each output pattern.

 The total weight from the input layer to each output neuron is
limited. If some connections are strengthened, others must be
weakened.

 A consequence is that the winner is the output neuron whose

weights best match the activation pattern.

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
MAX NET
 Max Net is a fixed weight
competitive net.

 Max Net serves as a subnet for

picking the node whose input is
larger. All the nodes present in this
subnet are fully interconnected and
there exist symmetrical weights in
all these weighted interconnections.

 The weights between the neurons

are inhibitory and fixed.

 The architecture of this net is as

shown:
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
MEXICAN HAT NETWORK
 Kohonen developed the Mexican hat network which is a more
generalized contrast enhancement network compared to the earlier
Max Net.

 Here, in addition to the connections within a particular layer of

neural net, the neurons also receive some other external signals.
This interconnection pattern is repeated for several other neurons
in the layer.

 The architecture for the network is as shown below:

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
MEXICAN HAT FUNCTION OF LATERAL
CONNECTION

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
MEXICAN HAT NETWORK
 The lateral connections are used to create a competition between
neurons. The neuron with the largest activation level among all
neurons in the output layer becomes the winner. This neuron is the
only neuron that produces an output signal. The activity of all
other neurons is suppressed in the competition.

 The lateral feedback connections produce excitatory or inhibitory

effects, depending on the distance from the winning neuron. This
is achieved by the use of a Mexican Hat function which
describes synaptic weights between neurons in the output layer.

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
HAMMING NETWORK
 The Hamming network selects stored classes, which are at a
maximum Hamming distance (H) from the noisy vector
presented at the input.

 The Hamming distance between the two vectors is the number

of components in which the vectors differ.

 The Hamming network consists of two layers.

• The first layer computes the difference between the total
number of components and Hamming distance between the
input vector x and the stored pattern of vectors in the feed
forward path.
• The second layer of the Hamming network is composed of
Max Net (used as a subnet) or a winner-take-all network
which is a recurrent network.
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
ARCHITECTURE OF HAMMING NET

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
SELF-ORGANIZATION
 Network Organization is fundamental to the brain

• Functional structure.
• Layered structure.
• Both parallel processing and serial processing require
organization of the brain.

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
SELF-ORGANIZING FEATURE MAP
Our brain is dominated by the cerebral cortex, a very complex
structure of billions of neurons and hundreds of billions of synapses.
The cortex includes areas that are responsible for different human
activities (motor, visual, auditory, etc.) and associated with different
sensory inputs. One can say that each sensory input is mapped into a
corresponding area of the cerebral cortex. The cortex is a self-
organizing computational map in the human brain.

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
SELF-ORGANIZING NETWORKS
 Discover significant patterns or features in the input data.

 Discovery is done without a teacher.

 Synaptic weights are changed according to local rules.

 The changes affect a neuron’s immediate environment until a final

configuration develops.

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
KOHONEN SELF-ORGANIZING FEATURE MAP
(KSOFM)

 The Kohonen model provides a topological mapping.

 It places a fixed number of input patterns from the input layer

into a higher dimensional output or Kohonen layer.

 Training in the Kohonen network begins with the winner’s

neighborhood of a fairly large size. Then, as training proceeds,
the neighborhood size gradually decreases.

 Kohonen SOMs result from the synergy of three basic processes

• Competition,
• Cooperation,
• Adaptation.
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
ARCHITECTURE OF KSOFM

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
COMPETITION OF KSOFM
 Each neuron in an SOM is
assigned a weight vector with the
same dimensionality N as the
input space.

 Any given input pattern is

compared to the weight vector of
each neuron and the closest
neuron is declared the winner.

 The Euclidean norm is commonly

used to measure distance.

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
CO-OPERATION OF KSOFM
 The activation of the winning neuron is spread to neurons in its
immediate neighborhood.
• This allows topologically close neurons to become sensitive to
similar patterns.

 The winner’s neighborhood is determined on the lattice topology.

• Distance in the lattice is a function of the number of lateral
connections to the winner.

 The size of the neighborhood is initially large, but shrinks over

time.
• An initially large neighborhood promotes a topology-preserving
mapping.
• Smaller neighborhoods allow neurons to specialize in the
latter stages of training.
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
ADAPTATION OF KSOFM
During training, the winner neuron and
its topological neighbors are adapted to
make their weight vectors more similar
to the input pattern that caused the
activation.

Neurons that are closer to the winner

will adapt more heavily than neurons
that are further away.

The magnitude of the adaptation is

controlled with a learning rate, which
decays over time to ensure convergence
of the SOM.

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
KSOFM ALGORITHM

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
EXAMPLE OF KSOFM

Find the winning neuron using the Euclidean distance:

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
Neuron 3 is the winner and its weight vector W3 is updated
according to the competitive learning rule:

The updated weight vector W3 at iteration (p+1) is

determined as:

The weight vector W3 of the winning neuron 3 becomes closer

to the input vector X with each iteration.

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
LEARNING VECTOR QUANTIZATION (LVQ)
 This is a supervised version of vector quantization. Classes are
predefined and we have a set of labeled data.

 The goal is to determine a set of prototypes that best represent

each class.

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
BASIC SCHEME OF LVQ

Step 1: Initialize prototype vectors for different classes.

Step 2: Present a single input.
Step 3: Identify the closest prototype, i.e., the so-called winner.
Step 4: Move the winner
• closer toward the data (same class),
• away from the data (different class).

VARIANTS OF LVQ

• LVQ 1
• LVQ 2
• LVQ 2.1
• LVQ 3

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
COUNTERPROPAGATION NETWORK
 Another variant of the BPN is the counterpropagation network
(CPN).

 Although this network uses linear neurons, it can learn

nonlinear functions by means of a hidden layer of competitive
units.

 Moreover, the network is able to learn a function and its inverse

at the same time.

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
TYPES OF COUNTERPROPAGATION NETWORK
 FULL COUNTERPROPAGATION NET
• Full counterpropagation net (full CPN) efficiently represents
a large number of vector pairs x:y by adaptively constructing a
look-up-table.

 FORWARD-ONLY COUNTERPROPAGATION NET

• A simplified version of full CPN is the forward-only CPN. The
approximation of the function y = f(x) but not of x = f(y) can
be performed using forward-only CPN.
• In forward-only CPN only the x-vectors are used to form the
clusters on the Kohonen units.

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
BASIC STRUCTURE OF FULL CPN

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
FIRST PHASE OF CPN

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
SECOND PHASE OF CPN

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
CPN LEARNING PROCESS

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
COUNTERPROPAGATION NETWORK
 After the first phase of the training, each hidden-layer neuron is
associated with a subset of input vectors.

 The training process minimized the average angle difference

between the weight vectors and their associated input vectors.

 In the second phase of the training, we adjust the weights in the

network’s output layer in such a way that, for any winning hidden-
layer unit, the network’s output is as close as possible to the
desired output for the winning unit’s associated input vectors.

 The idea is that when we later use the network to compute

functions, the output of the winning hidden-layer unit is 1 and the
output of all other hidden-layer units is 0.

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
 In the first training phase, if a hidden-layer unit does not win for
a long period of time, its weights should be set to random values
to give that unit a chance to win subsequently.

 There is no need for normalizing the training output vectors.

 After the training has finished, the network maps the training
vectors onto output vectors that are close to the desired ones.
The more hidden units, the better the mapping.

 CPN can be widely used in data compression and image

compression applications.

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
ADAPTIVE RESONANCE THEORY (ART)
NETWORK

 Adaptive Resonance Theory (ART) is a family of algorithms for

unsupervised learning developed by Carpenter and Grossberg.

 ART is similar to many iterative clustering algorithms where each

pattern is processed by

• finding the "nearest" cluster (a.k.a. prototype or template) to

that exemplar (desired).
• updating that cluster to be "closer" to the exemplar.

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
ARCHITECTURES OF ART NETWORK
 ART1, designed for binary features.

 ART2, designed for continuous (analog) features.

 ARTMAP, a supervised version of ART.

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
FUNDAMENTAL ALGORITHM OF ART NETWORK

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
BASIC ARCHITECTURE OF ART1

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
ART1 UNITS
ART1 Network is made up of two units

 Computational units
• Input unit (F1 unit – input and interface).
• Cluster unit (F2 unit – output).
• Reset control unit (controls degree of similarity).

 Supplemental units
• One reset control unit.
• Two gain control units.

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
ART2 NETWORK

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
BASIC ARCHITECTURE OF ART2

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
ART2 ALGORITHM

“Principles of Soft Computing, 2nd Edition”

by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
SUMMARY
This chapter discussed on the various unsupervised learning networks
like:

 Max Net

 Mexican Hat

 Kohonen Self-organizing Feature Maps

 Learning Vector Quantization

 Counterpropagation Networks

 Hamming Network

“Principles of Soft Computing, 2nd Edition”

 Adaptive Resonance Theory by S.N. Sivanandam & SN Deepa
Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.