5: Associative memories - the Hopeld net

Kevin Gurney
Dept. Human Sciences, Brunel University
Uxbridge, Middx. UK

1 The nature of associative memory

`Remembering' something in common parlance usually consists of associating something with
a sensory cue. For example, someone may say something, like the name of a celebrity, and
we immediately recall a chain of events or some experience related to the celebrity - we may
have seen them on TV recently for example. Or, we may see a picture of a place visited in
our childhood and the image recalls memories of the time. The sense of smell (olfaction) is
known to be especially evocative in this way.
On a more mundane level, but still in the same category, we may be presented with a
partially obliterated letter, or one seen through a window when it is raining (letter + noise)
and go on to recognise the letter.

slide of `T's
1
Hopeld 1 2

The common paradigm here may be described as follows. There is some underlying
collection of data which is ordered and interrelated in some way and which is stored in memory.
The data may be thought of, therefore, as forming a stored pattern. In the recollection
examples above, it is the cluster of memories associated with the celebrity or the phase in
childhood. In the case of character recognition, it is the parts of the letter (pixels) whose
arrangement is determined by an archetypal version of the letter. When part of the pattern
of data is presented in the form of a sensory cue, the rest of the pattern (memory) is recalled
or associated with it. Notice that it often doesn't matter which part of the pattern is used as
the cue, the whole pattern is always restored.
Conventional computers (von Neumann machines) can perform this function in a very
limited way. The typical software for this is usually referred to as a database. Here, the
`sensory cue' is called the key which is to be searched on. For example, the library catalogue
is a database which stores the authors, titles, classmarks and data of publication of books
and journals. We may search on any one of these discrete items for a catalogue entry by
typing the complete item after selecting the correct option from a menu. Suppose now we
have only the fragment `ion, Mar' from the full title `Vision, Marr D.'. There is no way that
the database can use this fragment of information to even start searching. We don't know if
it pertains to the author or the title, and even if we did, we might get titles or authors that
start with `ion'. The kind of input to the database has to be very specic and complete.

2 A physical analogy with memory

The networks that are used to perform associative recall are specic examples of a wider
class of physical systems which may be thought of as doing the same thing. This allows
the net operation to be viewed as a the dynamics of a physical system and its behaviour to
be described in terms of the network's `energy'. Consider a bowl in which a ball bearing is
allowed to roll freely

bowl and ball bearing in 3D

This is more easily drawn using a 2D cross section

2d X-section of bowl
Suppose we let the ball go from a point somewhere up the side of the bowl with, possibly,
a push to one side as well. The ball will roll back and forth and around the bowl until it
comes to rest at the bottom.
Hopeld 1 3

The physical description of what has happened may be couched in terms of the energy
of the system. The ball initially has some potential energy. That is work was done to push
it up the side of the bowl to get it there and it now has the potential to gain speed and
acquire energy. When the ball is released, the potential energy is released and the ball rolls
around the bowl (it gains kinetic energy). Eventually the ball comes to rest where its energy
(potential and kinetic) is zero. (The kinetic energy gets converted to heat via friction with
the bowl and the air). The main point is that the ball comes to rest in the same place every
time and this place is determined by the energy minimum of the system (ball + bowl). The
resting state is said to be stable because the system remains there after it has been reached
There is another way of thinking of this process which ties in with our ideas about
memory. We suppose that the ball comes to rest in the same place each time because it
`remembers' where the bottom of the bowl is. We may push the analogy further by giving
the ball a coordinate description. Thus, its position or state at any time is given by the three
coordinates ( x; y; z) or the position vector x. The location of the bottom of the bowl, x0
represents the pattern which is stored. By writing the ball's vector as the sum of x0 and a
displacement
x thus, x = x0 +
x, we may think of the ball's initial position as representing
the partial knowledge or cue for recall, since it approximates to the memory x0 .
If we now use a corrugated surface instead of a single depression (the bowl) we may
store many `memories'.

x1 x2 x3

{ x1 , x2 , x3 ... xn } are the stored memories

X-section through corrugated surface
If now the ball is started somewhere on this surface, it will eventually come to rest at
the local depression which is closest to its initial starting point. That is it evokes the stored
pattern which is closest to its initial partial pattern or cue. Once again, this is an energy
minimum for the system.
There are therefore two complementary ways of looking at what is happening. One is to
say that the system falls into an energy minimum; the other is that it stores a set of patterns
and recalls that which is closest to its initial state. If we are to build a network which behaves
like this we must include the following key elements
1. It is completely described by a state vector v = ( v1 ; v2 ; : : : ; vn )
2. There are a set of stable states v1 v2 v1
; ; v These will correspond to the stored
;:::; n

patterns and, in, the corrugated surface example, were the bottoms of the depressions
in the surface.
3. The system evolves in time from any arbitrary starting state v to one of the stable states,
and this may be described as the system decreasing its energy E. This corresponds to
the process of memory recall.
Hopeld 1 4

3 The Hopeld net

Consider the network consisting of three TLU nodes shown below

-2

1 1
-2

1
1

3 node Hopeld net

Every node is connected to every other node (but not to itself) and the connection
strengths or weights are symmetric in that the weight from node i to node j is the same as
that from node j to node i. That is, = , and = 0 for all . Notice that the ow
wij wj i wii i; j

of information in this net is not in a single direction as it has been in the nets dealt with so
far. It is possible for information to ow from a node back to itself via other nodes. That
is, there is feedback in the network and so they are known as feedback or recurrent nets as
opposed to feedforward nets which were the subject of the Backpropagation algorithm.
The state of the net at any time is given by the vector of the node outputs ( 1 2 3 ).
x ;x ;x

Suppose we now start this net in some initial state and choose a node at random and let it
update its output or `re'. That is, it evaluates its activation in the normal way and outputs
a `1' if this is greater than or equal to zero and a `0' otherwise. The net now nds itself either
in the same state as it started in, or in a new state which is at Hamming distance one from
the old. We now choose a new node at random to re and continue in this way over many
steps. What will the behaviour of the net be? For each state, we may evaluate the next state
given each of the three nodes res. This gives the following table.
State New state
Number vector (after node has red)
x1 x2 x3 Node 1 Node 2 Node 3
0 0 0 0 4 2 1
1 0 0 1 1 3 1
2 0 1 0 6 2 3
3 0 1 1 3 3 3
4 1 0 0 4 6 4
5 1 0 1 1 7 3
6 1 1 0 6 6 6
7 1 1 1 3 7 6
 The weight from node i to node j is sometimes also denoted by wij
Hopeld 1 5

This information may be represented in graphical form as a state transition diagram.

state transition diagram for 3 node net

States are represented by the circles with their associated state number. Directed
arcs represent possible transitions between states and the number alongside each arc is the
probability that each transition will take place. The states have been arranged in such a way
that transitions tend to take place down the diagram; this will be shown to re ect the way
the system decreases its energy. The important thing to notice at this stage is that, no matter
where we start in the diagram, the net will eventually nd itself in one of the states `3' or
`6'. These reenter themselves with probability 1. That is they are stable states - once the
net nds itself in one of these it stays there. The state vectors for `3' and `6' are (0,1,1) and
(1,1,0) respectively and so these are the `memories' stored by the net.

3.1 Dening an energy for the net

The dynamics of the net are described perfectly by the state transition table or diagram.
However, greater insight may be derived if we can express this in terms of an energy function
and, using this formulation, it is possible to show that stable states will always be reached in
such a net.
Consider two nodes in the net which are connected by a positive weight and where
i; j

j is currently outputting a `0' while i is outputting a `1'.

i j
wij

two nodes in con ict

Hopeld 1 6

If j were given the chance to update or re, the contribution to its activation from i is
positive and this may well serve to bring j's activation above threshold and make it output
a `1'. A similar situation would prevail if the initial output states of the two nodes had
been reversed since the connection is symmetric. If, on the other hand, both units are `on'
they are reinforcing each other's current output. The weight may therefore be thought of as
xing a constraint between i and j that tends to make them both take on the value `1'. A
negative weight would tend to enforce opposite outputs. One way of viewing these networks
is therefore as constraint satisfaction nets.
This idea may be captured quantitatively in the form of a suitable energy function.
Dene
= eij wij xi xj (1)
The values that eij take are given in the table below
xi xj eij

0 0 0
0 1 0
1 0 0
1 1 wij

If the weight is positive then the last entry is negative and is the lowest value in the
table. If is regarded as the `energy' of the pair ij then the lowest energy occurs when
eij

both units are on which is consistent with the arguments above. If the weight is negative, the
`11' state is the highest energy state and is not favoured. The energy of the net is found by
summing over all pairs of nodes
E =
X eij =
X wij xi xj (2)
pairs pairs

This may be written

= 1
X (3)
2
E wij xi xj
i;j

Since the sum includes each pair twice (as and ) and = = 0.
wij xi xj wj i xj xi wij wj i ; wii

It is now instructive to see what the change in energy is when a node res. Suppose
node k is chosen to be updated. Write the energy E by singling out the terms involving this
node.
= 12
X 1 1 X (4)
X
E
= i
2 k
2
wij xi xj wki xk xi wik xi xk

=
6

i i
j 6 k

Now, because wik = wki , the last two sums may be combined
= 1
X X (5)
2 =
E wij xi xj wki xk xi
i k
=
6

i
j 6 k

For ease of notation, denote the rst sum by S and take the xk outside the sum since it is
constant throughout, then
Hopeld 1 7

E = S xk
X wki xi (6)
i

but the sum here is just the activation of the kth node so that
= E S xk a
k
(7)
Let the energy after k has updated be and the new output be 0 . Then
0
E xk

=
E
0 0
S xk a
k
(8)
Denote the change in energy E
0
E by
and the change in output
E
0
xk xk by
, then
xk

subtracting (7) from (8)

=

E xk a
k
(9)
There are now two cases to consider
1.  0. Then the output goes from `0' to `1' or stays at `1'. In either case
 0.
a
k
xk

Therefore
k
xk a  0 and so,
 0 E

2.a
k
0. Then the output goes from `1' to `0' or stays at `0'. In either case
 0.
< xk

Therefore, once again
xk a  0 and
 0

k
E

Thus, for any node being updated we always have
 0 and so the energy of the net
E

decreases or stays the same. But the energy is bounded below by a value obtained by putting
all the = 1 in (3). Thus must reach some xed value and then stay the same. Once
xi ; xj E

this has occurred, it is possible for further changes in the network's state to take place since

= 0 is still allowed. However, for this to be the case (
6= 0 and
= 0) we must
E xk E

have = 0. This implies the change must be of the form 0 ! 1. There can be at most
a
k
N

(of course there may be none) of these changes, where is the number of nodes in the net.
N

After this there can be no more change to the net's state and a stable state has been reached.

Minimum E
Hopeld 1 8

state transitions to stable state

In the example given above, all state have zero energy except for state 5 which has
energy 2, and the stable states 3 and 6 which have energy -1.

3.2 Asynchronous vs. synchronous update

So far we have allowed only a single node to update or re at any time step. All nodes
are possible candidates for update and so they operate asynchronously; that is, there is no
coordination between them in time. The other extreme case occurs if we make all nodes re
at the same time, in which case we say there is synchronous update. To do this, we must
ensure that each nodes previous output is available to the rest of the net until all nodes
have evaluated their activation and been updated. It is therefore necessary to store both the
current state vector and the next state vector. The behaviour is now deterministic; given
any state, a state transition occurs to a well dened next state, there being no probabilistic
behaviour. The analysis, in terms of energy changes at each update, given above no longer
applies but there is now a simplied type of state diagram in which only a single arc emerges
from any state. This allows us to predict, once again, the general type of behaviour. In
particular, state-cycles occur again but now there is the possibility for multiple-state cycles.

OR

Cant have...

state diagrams for synchronous update

These may be useful in storing sequences of events or patterns. A little thought will
show that the (single) state cycles remain the same under synchronous dynamics so the single
stored patterns remain the same under both dynamics.

3.3 Ways of inputting information

So far it has been assumed that the net is started in some initial state (the memory cue) and
the whole net allowed to run freely until a state cycle is encountered (recall slide of noisy
`T'). There is another possibility in which some part of the net has its outputs xed while
the remainder is allowed to update. The part that is xed is said to be clamped and if the
Hopeld 1 9

clamp forms part of a state cycle, the remainder (unclamped) part of the net will complete
the pattern stored at that cycle (recall slide of partially correct `T'). Which mode is used will
depend on any prior knowledge about parts of the cue being uncorrupted or noise free.
The problem of how to x the weights in Hopeld nets will be dealt with next time.

References
Aleksander, I. and Morton, H. (1990). Neural Computing. Chapman hall.
Quite good on Hopeld nets and contains a similar example to the one given in these
notes.
Hopeld, J. (1982). Neural networks and physical systems with emergent collective computa-
tional properties. Proceedings of the National Academy of Sciences of the USA, 79:2554 {
2588. Hopeld has another, related, model which uses continuous outputs. Beware when
reading the literature which model is being discussed.
McEliece, R., Posner, E., Rodemich, E., and Venkatesh, S. (1987). The capacity of the hopeld
associative memory. IEEE Transactions on Information Theory, IT-33:461 { 482.
There are many papers on this area, but this has some non- technical material near the
beginning before getting into the welter of maths needed for this kind of analysis.
 6: Hopeld nets (contd.)
Kevin Gurney
Dept. Human Sciences, Brunel University
Uxbridge, Middx. UK

1 Finding the weights

So far we have described the dynamics of Hopeld nets but nothing has been said about
the way the weights are established for a particular problem. In his original paper, Hopeld
(1982) did not give a method for training the nets, rather he gave a prescription for making a
weight set, given a set of patterns to be stored. Here, we shall relate the storage prescription,
later on, to a biologically inspired learning rule - the Hebb rule - and show that the nodes
may also be trained individually using the delta rule.

1.1 The storage prescription

The rationale behind the prescription is based on the desire to capture, in the value of the
weights, local correlations between node outputs when the net is in one of the required stable
states. Recall that these correlations also gave rise to the energy description and the same
kind of arguments will be used again.
Consider two nodes which, on average over the required pattern set, tend to take on the
same value. That is, they tend to form either the pair (0, 0) or (1, 1). The latter pairing will be
reinforced by there being a positive weight between the nodes, since each one is then making a
positive contribution to the others activation which will tend to foster the production of a `1'
at the output. Now suppose that the two nodes, on average, tend to take on opposite values.
That is they tend to form either the pair (0, 1) or (1, 0). Both pairings are reinforced by a
negative weight between the two nodes, since there is a negative contribution to the activation
of the node which is `o' from the node which is `on', supporting the former's output state
of `0'. Note that, although the pairing (0, 0) is not actively supported by a positive weight
per se, a negative weight would support the mixed output pair-type just discussed.
These observations may be encapsulated mathematically in the following way. First
we introduce an alternative way of representing binary quantities. Normally these have been
denoted by 0 or 1. In the polarised or spin representation they are denoted by -1 and 1
respectively, so there is the correspondence 0 $ 1; 1 $ 1. Now let v ; v be components
p
i
p
j

This name is derived from the fact that Hopeld nets have many similarities with so-called spin glasses
in physics, the prototypical example of which is a collection of magnetic domains whose polarisation of 1 is
determined by the average spin of the electrons in each domain

1
Neural Nets 6 2

of the pth pattern to be stored where these are in the spin representation. Consider what
happens if the weight between the nodes i and j is given by
wj=
i
Xv vp
i
p
j (1)
p

Where the sum is over all patterns p to be stored. If, on average, the two components
take on the same value then the weight will be positive since we get terms like 1  1 and
1  1 predominating. If, on the other hand, the two components, on average, take on
opposite values we get terms like 1  1 and 1  1 predominating which gives a negative
weight. This is just what was required according to the arguments given above. Equation
(1) is therefore the storage prescription used with Hopeld nets. Note that, the same weights
would accrue if we had tried to learn the inverse of the patterns formed by taking each
component of every pattern and changing it to the opposite value. The net therefore, always
learns the patterns and their inverses.

1.2 The Hebb rule

The use of (1) which is an algorithm or recipe for xing the weights without adapting to
a training set may appear to run counter to the ideas being promoted in the connectionist
cause. It is possible, however, to view the prescription in (1) as a short-cut to a process of
adaptation which would take place if we were to obey the following training algorithm
1. present the components of one of the patterns to be stored at the outputs of the corre-
sponding nodes of the net.
2. If two nodes have the same value then make a small positive increment to the inter-
node weight. If they have opposite values then make a small negative decrement to the
weight.
Steps 1) and 2) then get repeated many times with a dierent pattern selection in 1).
Symbolically step 2) (which is the learning rule) may be written

w = v v
ij
p
i
p
j (2)
where, as usual  is a rate constant and 0 <  < 1. It is clear that the storage
prescription is just the result of adding all the weight changes that would accumulate under
this scheme if enough pattern presentations were made. The rule in (2) is one of a family
of rules known as Hebb rules after D. O. Hebb. The characteristic of such a rule is that it
involves the product of a pair of node activations or outputs.
As a variation, suppose we had used the usual Boolean representation for the compo-
nents x so that x is 0 or 1 The Hebb rule would now be
w = x x . Interpreting this,
p
i
p
i ij
p
i
p
j

we have that the change in weight is only ever positive and only occurs if both nodes are
ring (output `1'). This is, in fact closer to the original rule proposed by Hebb (1949) in a
neurophysiological context. In his book The Organization of behaviour Hebb postulated that
When an axon of cell A is near enough to excite a cell B and repeatedly or persistently
takes part in ring it, some growth process or metabolic change takes place in one or
both cells such that A's eciency, as one of the cells ring B, is increased.
Neural Nets 6 3

That is, the correlation of activity between two cells is reinforced by increasing the
synaptic strength (weight) between them. Simpson (course book) contains a list of Hebb rule
variants.

1.3 Using the delta rule

We may draw the connections in, say, a 3 node Hopeld net as follows.
Nodes

1 2 3

is equivalent
to... 1
3

1 2 3

Distribution
terminals
3 node Hopeld net as feedforward with recurrence
Each node may now be thought of as taking part in some input- output function between
the distribution terminals and the node outputs. They may therefore each be trained with
the delta rule. If the corresponding training set for each one is linearly separable then the
set of stable states may be learnt. However, there is no guarantee now that w = w . Theij ji

change in energy at each node update is now no longer necessarily less than or equal to zero.
The consequence is that, given the stable states have been trained, the system moves through
state space with decreasing error towards a stable state but has, superimposed on this, some
noise.
Energy Energy

may go up or down
always decreasing or the same

time time

Symmetric Asymmetric
energy v time for symmetric and asymmetric nets
Neural Nets 6 4

2 Storage capacity
How good is the storage prescription (1) at storing the patterns so that they are stable states?
Clearly, as the number of patterns m increases, the chances of accurate storage must decrease.
In some empirical work in his 1982 paper, Hopeld showed that about half the memories were
stored accurately in a net of N nodes if m = 0:15N . The other patterns did not get stored as
stable states. In a more rigorous piece of analysis McCliece et al. (1987) showed theoretically
that, if we require almost all the required memories to be stored accurately, then the maximum
number of patterns m is N=2 log N . For N = 100 this gives m = 11.
Suppose a pattern which was required to be stored did not, in fact produce a stable
state and we start the net in this state. The net must evolve to some stable state and this is
usually not related to any of the original patterns used in the prescription. The stable state
represents a spurious energy minimum of the system - one that is not there by design.

3 The analogue Hopeld model

In a second important paper (Hopeld, 1984) Hopeld introduced a variant of the discrete
time model discussed so far which uses nodes described by their rate of change of activation.
This kind of node was discussed in the last part of lecture 1 but we review it here. Denote
the sum of excitation from other nodes for the j node by s so that j

s =
j
Xw x ji i (3)
i

then the rate of change of activation da =dt is given by

j

da = ks
j
j
ca j
(4)
dt
here, k and c are constant. The rst term (if positive) will tend to make the activation
increase while the second term is a decay term (see lecture 1). The output y is then just the
j

sigmoid of the activation as usual. Hopeld also introduced the possibility of external input
at this stage and a variable threshold.
In the previous TLU model, the possible states of an N node net are just the corners
of the N-dimensional hypercube. In the new model, because the outputs can take any values
between 0 and 1, the possible states include now, the interior of the hypercube. Hopeld
dened an energy function for the new network and showed that if the inputs and thresholds
were set to zero, as in the TLU discrete time model, and if the sigmoid was quite `steep', then
the energy minima were conned to regions close to the corners of the hypercube and these
corresponded to the energy minima of the old model.
There, however, are two advantages of the new model. The rst, is that the use of
the sigmoid and time integration make more contact possible with real biological neurons.
The second is that it is possible to build the new neurons out of simple, readily available
hardware. In fact, Hopeld writes the equation for the dynamics - eqn (4) - as if it were
built from an operational amplier and resistor network. This kind of circuit was the basis of
several implementations - see for example Graf et al. (1987).
Neural Nets 6 5

4 Combinatorial optimisation
Another innovation made by Hopeld was to show how to solve large combinatorial optimi-
sation problems on neural nets (Hopeld and Tank, 1985). The classical example of this is
the so- called Travelling salesman Problem (TSP). Here, a travelling salesman has to visit
each of a set of cities in turn in such a way as to minimise the total distance travelled. Each
city must be visited once and only once. This kind of problem is computationally dicult
in a technical sense (NP-complete) in that the time to solution scales with the number cities
faster than the time t raised to any xed power and therefore might scale like e . t

The solution of the TSP consists of a sequence of cities. The problem for N cities may
be coded into an N by N network as follows. Each row of the net corresponds to a city. The
position of the city in the solution sequence is given by putting a `1' at the corresponding place
in the row and 0's everywhere else in that row. Thus, if the city corresponding to row 5 was
7th in the sequence there would be a 1 in the 7th place of the 5th row and zeros everywhere
else. Note that most states of the net do not correspond to valid tours - there must be only
one `1' per row. The problem then, is to construct an energy function (and hence a set of
weights) which lead to stable states (states of lowest energy) of the network that, not only
express valid city tours, but which also are tours of short length. The validity criterian results
in negative weights (inhibition) between nodes in the same row, and between nodes in the
same column. The path-length criterion leads to inhibition between adjacent columns (cities
in a path) proportional to the path length between the cities (row positions). The net is now
allowed to run until an energy minimum is reached which should now correspond to a solution
of the problem.
Neural Nets 6 6

References
Graf, H., Hubbard, W., Jackel, L., and deVegvar P.G.N (1987). A cmos associative memory
chip. In 1st Int. Conference Neural Nets, San Diego.
(I have this).
Hebb, D. (1949). The Organization of behaviour. John Wiley.
Hopeld, J. (1982). Neural networks and physical systems with emergent collective computa-
tional properties. Proceedings of the National Academy of Sciences of the USA, 79:2554 {
2588. Hopeld has another, related, model which uses continuous outputs. Beware when
reading the literature which model is being discussed.
Hopeld, J. (1984). Neurons with graded response have collective computational properties
like those of two-state neurons. Proceedings of the National Academy of Sciences of the
USA, 81:3088 { 3092.
Hopeld, J. and Tank, D. (1985). Neural computation of decisions in optimization problems.
Biological Cybernetics, 52:141 { 152.
McEliece, R., Posner, E., Rodemich, E., and Venkatesh, S. (1987). The capacity of the hopeld
associative memory. IEEE Transactions on Information Theory, IT-33:461 { 482.
There are many papers on this area, but this has some non- technical material near the
beginning before getting into the welter of maths needed for this kind of analysis.
 7: Competition and self-organisation:
Kohonen nets
Kevin Gurney
Dept. Human Sciences, Brunel University
Uxbridge, Middx. UK

1 Competitive dynamics
Consider a layer or group of units as shown in the diagram below.

Only one nodes

connections shown
for clarity
Inputs
Competitive layer
Each cell receives the same set of inputs from an input layer and there are intralayer
or lateral connections such that each node is connected to itself via an excitatory (positive)
weight and inhibits all other nodes in the layer with negative weights.
Now suppose a vector x is presented at the input. Each unit now computes a weighted
sum s of the inputs provided by this vector. That is
X
s= wx i i (1)
i

In vector notation this is, of course, just the dot product w
x. This is the way of
looking at things which will turn out to be most useful. Then some node k, say, will have a
value of s larger than any other in the layer. It is now claimed that, if the node activation
is allowed to evolve by making use of the lateral connections, then node k will develop a
maximal value for a while the others get reduced. The time evolution of the node is usually
governed by an equation which determines the rate of change of the activation (Lecture 1,
section `Introducing time'). This must include the input from the lateral connections as well

1
Neural Nets 7 2

as the `external' input given by s. Thus if l is the weighted sum of inputs from the lateral
connections
da
= s+l a (2)
dt s l

Recall that da=dt is the rate of change of a. There will usually be a sigmoid output
relation y = (a)
What happens is that the node with greatest excitation s from the input has its acti-
vation increased directly by this and indirectly via the self-excitatory connection. This then
inhibits the neighbouring nodes, whose inhibition of k is then further reduced. This process
is continued until a stability is reached. There is therefore a `competition' for activation
across the layer and the network is said to evolve via competitive dynamics. Under suitable
conditions, the nodes whose input s was less than that on the `winning node' k will have their
activity reduced to zero. The net is then sometimes referred to as `winner-takes-all' net, since
the node with largest input `wins' all the available activity.
If the net's activity is represented in prole along the string of nodes then an initial
situation in part a) of the diagram below will evolve into the situation shown in part b).
activation

after

before

time evolution under competitive dynamics

Competitive dynamics are obviously useful in enhancing the activation `contrast' over
a network layer and singling out the node which is responding most strongly to its input. We
now examine how this process may be useful in a learning situation.

2 Competitive learning
Consider a training vector set whose vectors all have the same length, and suppose, without
loss of generality, that this is one. Recall that the length kxk of a vector x is given by
X
kxk = x2 i
(3)
i

A vector set for which kxk = 1 for all x is said to be normalised. If the components are
all positive or zero  then this is approximately equivalent to the condition
 this is consistent with the interpretation of the input as derived from the output of a previous layer
Neural Nets 7 3

X
x =1
i (4)
i

Since the vectors all have unit length, they may be represented by arrows from the
origin to the surface of the unit (hyper)sphere.

vectors on unit hypersphere

Suppose now that a competitive layer has had its weight vectors normalised according
to (4). Then these vectors may also be represented on the same sphere.
What is required for the net to encode the training set is that the weight vectors become
aligned with any clusters present in this set and that each cluster is represented by at least
one node. Then, when a vector is presented to the net there will be a node, or group of nodes,
which respond maximally to the input and which respond in this way only when this vector
is shown at the input.

vectors
weights
weights and vectors aligned
If the net can learn a weight vector conguration like this, without being told explicitly
of the existence of clusters at the input, then it is said to undergo a process of self-organised
or unsupervised learning. This is to be contrasted with nets which were trained with the delta
rule or BP where a target vector or output had to be supplied.
In order to achieve this goal, the weight vectors must be rotated around the sphere so
that they line up with the training set. The rst thing to notice is that this may be achieved
in an gradual and ecient way by moving the weight vector which is closest (in an angular
sense) to the current input vector towards that vector slightly. The node k with the closest
vector is that which gives the greatest input excitation s since this is just the dot product of
Neural Nets 7 4

the weight and input vectors. As shown below, the weight vector of node k may be aligned
more closely with the input if a change
w is made according to

w = (x w) (5)
x-w

w
x

w = w + w
w = (x - w)
vector triangle - weights and inputs
Now it would be possible to use a supervisory computer to decide which node had the
greatest excitation s but it is more satisfactory if the net can do this itself. This is where
the competitive dynamics comes in to play. Suppose the net is winner- take-all so that the
winning node has value 1 and all the others have value close to zero. After letting the net
reach equilibrium under the lateral connection dynamics we now enforce the rule

w = (x w)y (6)
across the whole net. Then there will only be a single node (the one whose dot-product s was
greatest) which has y = 1 and for which the weight change is the same as in (6). All other
nodes will have y = 0 and so there weight change will also be zero. The stages in learning
(for a single vector presentation) are then
1. apply vector at input to net and evaluate s for each node.
2. update the net (in practice, in discrete steps) according to (2)for a nite time or until
it reaches equilibrium.
3. train all nodes according to (6)
There are a few points about the learning rule worth noting. First, if the weights are
initially normalised according to (4) and the input vectors are normalised in the same way,
then the normalisation of the weights is preserved under the learning rule. The change in the
length of w is given by the sum of the changes in its components
X X X !

w = y
i x w
i i (7)
i i i

and each of the sums in the bracket is 1 so that the right hand side is zero. The object
of normalisation is a result of a subtlety that has been ignored so far in order to clarify the
essentials of the situation. It was assumed the dot product s, gives an indication of the angular
separation of the weight and input vectors. This is true up to a point but recall that the dot
Neural Nets 7 5

product also involves the product of vector lengths. If either the input or weight vectors are
large, then s may also be large, not as a result of angular proximity (the vectors being aligned)
but simply by virtue of their magnitude. We want a measure of vector alignment which does
not require a separate computation of vector lengths, and the normalisation process is one
way of achieving this.
Secondly, the learning rule may be expanded to the sum of two terms

w = xy wy (8)
The rst of these looks like a Hebb term while the second is a weight decay. Thus we
may see competitive self-organisation as Hebb learning but with a decay term that guarantees
normalisation. This latter property may be thought of in biological terms as a conservation
of metabolic resources; thus, the sum of synaptic strengths may not exceed a certain value
which is governed by physical characteristics of the cell to support synaptic and post-synaptic
activity. There are several architectures that have used the basic principles outlined above.
Rumelhart & Zipser (1986) (henceforth R & Z) give a good discussion of competitive learning
and several examples. There is only space to discuss one of these here.

2.1 Letter and `word' recognition

R & Z train using pairs of characters, each one being based on a 7 by 5 pixel grid. In the rst
set of experiments they used the four letter pairs AA AB BA BB. With just two units in a the
competitive net, each unit learned to detect either A or B in a particular serial position. Thus,
in some experiments, unit 1 would respond if there was an A in the rst position while unit 2
would respond if there was a B in the rst position. Alternatively the two units could respond
to the letter in the second position. Note that these are, indeed, the two possible `natural'
pairwise groupings of these letter strings. R & Z call this net a `letter detector'. With four
units each node can learn to respond to each of the four pairs - it is a `word detector'.
In another set of experiments, R & Z used the letter pairs AA, AB, AC, AD, BA, BB,
BC, BD. When a net with only two units was used, one unit learned to recognise the pairs
which started with A, while the other learned to respond to those that began with B. When 4
units were used each unit learned to recognise the pairs that ended in one of the four dierent
letters A, B, C, D. This represents two dierent ways of clustering the training set. If the
patterns are to be put into two groups then, clearly, it is the rst letter which characterises the
group. On the other hand, if there are to be four clusters, the four value feature determined
by the second letter is the relevant distinction.

3 Kohonen's self-organising feature maps

3.1 Topographic maps in the visual cortex
It often occurs that sensory inputs may be mapped in such a way that it makes sense to talk
of one stimulus being `close to' another according to some metric property of the stimulus.
The simplest example of this occurs when the metric is just the spatial separation of localised
sources. A slightly more abstract example is provided by the cells in visual area 1 of the
Neural Nets 7 6

mammalian brain, which are `tuned' to orientation of the stimulus. That is, if a grid or
grating of alternating light and dark lines is presented to the animal, the cell will respond
most strongly when the lines are oriented in a particular direction and the response will fall
o as the grating is rotated either way from this preferred direction. This was established
in the classic work of Hubel & Weisel (1962) using microelectrode studies with cats. Two
grating stimuli are now `close together' if their orientations are similar. This denes a metric
or measure for the stimuli.
If we were to train a competitive network on a set of gratings then each cell (unit) would
learn to recognise a particular orientation. However there is an important property of the way
cells are organised in biological nets which will not be captured in our scheme as described
so far. That is, cells which are tuned to similar orientations tend to be physically located
in proximity with one another. In visual cortex, cells with the same orientation tuning are
placed vertically below each other in columns perpendicular to the surface of the cortex. If
recordings are made from an electrode which is now inserted parallel to the cortex surface
and gradually moved through the tissue, the optimal response from cells will be obtained at
a series of orientations that vary, in general, smoothly across the cortex. There are, however
occasional discontinuities as shown in the slideorienttrack

slide of data on orientation tuning

The orientation tuning over the surface forms a kind of map with similar tunings being
found close to each other. These maps are called topographic feature maps. It is possible to
train a network using methods based on activity competition in a such a way as to create
such maps automatically. This was shown by C. von der Malsburg in 1973 specically for
orientation tuning, but Kohonen (1982) popularised and generalised the method and it is in
connection with his name that these nets are usually known. The best exposition is given in
his book (Kohonen, 1984). These nets consist of a layer of nodes each of which is connected
to all the inputs and which is connected to some neighbourhood of surrounding nodes.
Neural Nets 7 7

Kohonen neighbourhoods
3.2 The algorithm
At each vector presentation the following sequence of steps occurs
 Find the node k whose weight vector is closest to the current input vector.
 Train node k and all nodes in some neighbourhood of k.
 Decrease the learning rate slightly
 After every M cycles, decrease the size of the neighbourhood
In connection with 1), it is important to realise that Kohonen postulates that this is
done with a supervisory computational engine and that his results are not based on the use
of competitive dynamics to nd the `winning' node. The justication for this is that it could
have been done by the net itself with lateral connections. The use of competitive dynamics
would slow things down considerably, is very much dependent on the parameters, and does
not always work, `cleanly'. (recall the video demo). In fact the rule Kohonen uses in his
examples is to look for the node which simply has the smallest value for the length of the
dierence vector x w. This also appears to obviate the need for input vector normalisation
(and hence for the restriction to positive components) which was a prerequisite with the inner
product activation measure of proximity. However, this method cannot be the basis of any
biologically plausible algorithm.
The key point in this algorithm is 2). It is through this that the topographic mapping
arises. It is the use of training over a neighbourhood that ensures that nodes which are
close to each other learn similar weight vectors. Decreasing the neighbourhood ensures that
progressively ner features or dierences are encoded and the gradual lowering of the learn
rate ensures stability (otherwise the net may exhibit oscillation of weight vectors between two
clusters).

3.3 A graphic example

It is possible to illustrate the self-organisation of a Kohonen net graphically using a net where
the input space has just two components. Consider a net with just 6 nodes on a rectangular
grid.
Neural Nets 7 8

6 node net on grid

Another representation of this net is in weight space. Since there are only 2 weights we
may draw this on the page. Initially the weights will be random, say

random weights
The lines are drawn to connect nodes which are physically adjacent (rst diagram).
Suppose now that there are 6 input vectors which may be represented in pattern space as
shown below

input vectors for 6 node K. net

In a well trained (ordered) net that has developed a topographic map the diagram in
weight space should have the same topology as that in physical space and will re ect the
properties of the training set.

Trained weight space

Neural Nets 7 9

The case of 2-component vectors which are drawn randomly from the unit square and in
which the initial weights are close to the centre of the unit square, is dealt with by Kohonen
(1984).

slide of kohonens results for 2D square

The weight diagram starts as a `crumpled ball' at the centre and expands like a shing
net being unravelled. Kohonen deals with several other similar examples.
When the input space is more than 2-dimensional, the higher dimensions have to get
`squashed' onto the grid. This will be done in such a way as to preserve the most important
variations in the input data. Thus, it is often the case that the underlying dimensionality
of the input space is smaller than the number inputs. This is illustrated below for 2-D data
which has an underlying dimensionality of 1
x2

x1
points on arc of circle
Neural Nets 7 10

The points lie on an arc of a circle and each point may be specied by stating how far
round the arc it is. A more convincing example with 3D data with an underlying dimensionalty
of 2 is shown in g 5.9 of Kohonen's book

slide of 3d projection
The topographic map will also refelect the underlying distribution of the input vectors.
(Kohonen g 5.18)

slide of `cactus' distribution

Returning to the original example of orientation maps in the visual system some of
my own recent work has focussed on training nets whose cells form a map of image velocity
(Gurney and Wright, 1992)
Neural Nets 7 11

4 Other competitive nets

Fukushima (1975) has developed a multilayered net called the `neocognitron' which recognises
characters and which is loosely based on early visual processing. The structure is quite
complex and, although some of the features seem rather ad hoc, it is a very impressive example
of modelling a large system which has many similarities with its biological counterpart.
No account of competitive learning would be complete without reference to the work
of Stephen Grossberg. His Adaptive Resonance Theory (ART) has been the subject of a
enormous number of papers. ART concerns the development of networks in which the number
of nodes required for classication is not assigned ab initio but is determined by the net's
sensitivity to detail within the data set (given by the so-called vigilance parameter). The
network is embedded in a control loop which is an integral part of the entire system. It would
require a complete lecture to do justice to Grossberg's networks however and it will have to
suce here to simply give a reference. This in itself is not easy - Grossberg's work is often
quite hard to read and any single reference will undoubtedly be inadequate. One possible
route is his 1987 paper in Cognitive Science (Grossberg, 1987).
Neural Nets 7 12

References
Fukushima, K. (1975). Cognitron: a self-organizing multilayered neural network. Biological
Cybernetics, 20:121 { 136.
Grossberg, S. (1987). Competitive learning: from interactive activation to adaptive resonance.
Cognitive Science, 11:23 { 63.
Gurney, K. and Wright, M. (1992). A self-organising neural network model of image velocity
encoding. Biological Cybernetics, 68:173 { 181.
Hubel, D. and Wiesel, T. (1962). Receptive elds, binocular interaction and functional archi-
tecture in the cat's visual cortex. Journal of Physiology, 160:106 { 154.
Kohonen, T. (1982). Self-organized formation of topologically correct feature maps. Biological
Cybernetics, 43:59 { 69.
Kohonen, T. (1984). Self-organization and associative memory. Springer Verlag.
Rumelhart, D., McCllelland, J., and The PDP Research Group (1986). Feature disc overy by
competitive learning. In Parallel Distributed Processing.
von der Malsburg, C. (1973). Self-organization of orientation sensitive cells in the striate
cortex. Kybernetik, 14:85 { 100.