Basic Learning Laws

31

of an
will be activated to the maximum extent. Thus the operation
the case
instar can be viewed as content addressing the memory. In
connections
of an outstar, during learning, the weight vector for the
when
from the jth unit in F2 approaches the activity pattern in Fy
an input vector a is presented at F1. During recall,
whenever the unit
j is activated, the signal pattern (aw 82 8," wl be
transmitted to F1, where s, is the output of the jth unít. This signal
to
pattern then produces the original activity pattern corresponding
the input vector a, although the input is absent. Thus the operation
of an outstar can be viewed as memory addressing the contents.
When all the connections from the units in F to F2 are made as
in Figure 1.7c, we obtain a heteroassociation network. This network
can be viewed as a group of instars, if the flow is from F to F,. On
the other hand, if the flow is from F to F, then the network can be
viewed as a grôup of outstars (Figure 1.7d).
When the flow is bidirectional, we get a bidirectional associative
memory (Figure 1.7e), where either of the layers can be used as
input/output.
If the two layers F and F2 coincide and the weights are
symmetric, i.e., w = w i # j, then we obtain an autoassociative
memory in which eaeh unit is connected to every other unit and to
itself (Figure 1.7f).

KBasic Learning Laws

The operation of a neural network is governed by neuronal dynamics.
Neuronal dynamics consists of two parts: one corresponding to the
dynamics of the activation state and the other corresponding to the
dynamics of the synaptic weights. The Short Term Memory (STM) in
neural networks is modelled by the activation state of the network.
The Long Term Memory (LTM)_corresponds to the encoded pattern
informationin the synaptic weights due to learning. We will discuss
models of neuronal dynamics in Chapter 2. In this section we discuss
me basic learning laws [Zurada, 1992, Sec. 2.5; Hassoun, 1995,
Ch. 3). Learning laws are merely implementation of
models synaptie
dynamics. Typically, a model of synaptic dynamics is described in
terms of expressions for the first derivative of the weights. They are
called learning equations.
Learning laws describe the weight vector for the ith processing
unit at time instant (t + 1) in terms of the weight vector at time
instant (t) as follows:
w t +1) = w,t) + Aw,() |
(1.1)
where Aw,(t) is the change in the
weight vector.
There are different methods for
feature of a neural network, leading to implementing
the learning
several learning laws. Some
 Networks
Neural
Artificial
Basics of
laws use
32 learning
below. All these the
connection

discussed of
laws are
the weight
ESC earning
local
information for adjusting
ony
between two units.

1.6.1 Hebb's Law

vector is given by
Here the change in the weight (1.2)
Aw, nfw;a) a
=

the jth component of Aw,

is given by
Therefore,

Aw= nAw;a) a,
for j 1, 2,.., M (1.3)
the
the ith unit. The law states that
where s is the output signal of data and
is proportional to the product of the input
weight increment This law requires weight
the resulting output signal of the unit.
initialization to small random values around wj = 0 prior to tearning.

This law represents an unsupervieed learning.

1.6.2 Perceptron Learning Law pert-

Here the change in the weight vector is given|

Aw, =
n lb,-sgn(w;a)) a (1.4)
where sgn(x) is sign of x. Therefore, we have

A wy =
n b; sgn(w a)] a,
-

n (6,-8,) a, for j =1, 2, .., M (1.5)

This law is applicable only for bipolar output functions
f). This
s also called discrete perceptron
learning law. The expression
for
Aw shows that the weights are adjusted only if the actual
sis incorrect, since the term in the square brackets is output
zero for the
correet 6utput. This is a supervised learning law, as the law
eired output for ach inpyt. In requires
he initializedto any implementation,
random initial values, as they are not weights can
the
eihts converge to the final eritical
values eventually by repeated use ofThe
input-output pattern pas, proaeu te pattern pairs are the
by the system. These 155ues will be discussed in Chapter 4representable
resentable

1.6.3 Deta Learning Law

Here the change in the weight vector is given by

aw, =n lb, -Aw{a)l fiw/a) a

(1.6)
 Basic Learning Laws 33

where f ) is the derivative with respect to x. Hence,

Aw = n lb, -fw a)lfw/a) a

= n lb,-sl fa) a, for j = 1, 2, .., M (1.7)

This law is valid only for a differentiable output function, as it
depends on the derivative of the output function i). It 1s a supervised
learning law since the change in the weight is based on the
error

between the desired and the actual output values for a given input.
Delta learning law can also be viewed as a continuous perceptron
learning law.)
random
In implementation, theweights can be initialized to any
values as the values are not very critical. The weights
converge to
use of the input-output
the final values eventually by repeated
be more or less guaranteed by
pattern pairs. The convergence can
more layers of processing units in between the
input and output
using to the case of multiple
layers. The delta learning law can be generalized delta
We will discuss the generalized
layers of a feedforward network. law in Chapter 4.
rule or the error backpropagation learning
we

1.6.4 Widrow and Hoff LMS Learning Law

vector is givenby
Here the change in the weight
S (1.8)
Aw =
n lb;-wal a Na

Hence 1, 2, M (1.9)
Aw =
n b;-wal , forj = .,

law and is a special case of the delta

This is a supervised learning function is assumed linear, i.e.,
law, where the output
learning in the weight is made proportional
this case the change and
flx) . In
=

error, between the desired output

gradient of the
tothe negative activation which is also the continuous output
the continuous value, called
function. Hence, this is also
signal due to linearity of the output is same as
(LMS) error learning law.. This
the Least Mean Squared of neuron. In
t h e . A d a l i n e m o d e l of neuran.
law úsedin
the learning the weights may be initialized to any values The
implementation, to times achieve
input-output pattern
pairs data is applied several data. The
for a given set o f training
of the weights data set.
convergence
arbitrary training
convergence
is not guaranteed for any

1.6.5 Correlation Learning Law

vector is given by
Here the change in the weight
(1.10)
AW; =
n 6; a
 Networks
Neural
Artificial
Basics of
34
(1.11)
Therefore 1, 2,,M
for j =

auny
=
nbaj, output
signal
with the
Hebbian
learning Hebbian learning
special case of the (b,). But
the
ning is a
8a the desired signal
learni
(s) being replaced by
correlation

whereas the to adjust

is an unsupervised learning,
the desired
output value
since it u s e s the weights
Spervised learning, of the learning
law, * .
implementation
The weights. In the zero, i.e., Wj
close to
are initialised to small
random values

Law
1.6.6 Instar (Winner-take-all) Learning as

of neurons, organized in a layer

a collection the units
This is relevant for All the inputs are connected to each of
shown in Figure 1.8.

a
of units for instar learning, where the adjusted
Figure 1.8 Arrangement
weights are highlighted.

in the output layer in a feedforward manner. For a given input vector

is computed using the weighted
sum
a, the output from each unit i That is
wa. The unit k that gives maximum Output 1s identified.
wa = max (wa) (1.12)
i
Then the weight vector leading to the kth unit is adjusted as follows:
Aw, = n (a-w) (1.13)
Therefore,
for j = 1, 2, .., M (1.14)
The final weight vector tends to represent a group of input vectors
within a small neighbourh0od. "This is a case of unsupervised
learning. In impBementation, the values. of the weight vectors are
initialized to raFdom values priof to Tearning, and the vector lengths
are normalized during learning
Law
1.6.7 Outstar Learning
The outstar learning law is als0 related to a group of units arranged
in a layer as shown in Figure 1.9. In this law the weights are adiusted
sO as to capture the desred output pattern characteristics. The
adjustment of the weights is given by
Aw n (6-wja), for j = 1, 2, . , M
(1.15)
 Basic Learning Laws

(M)
b, b b
Figure 1.9 Arrangement of units for 'outstar learning, where the adjusted
weights are highlighted.

where the kth unit is the only active unit in the input layer. The
vector b = (b1, 62, .., b u i s the desired response from the layer of
M units. The outstar learning is a supervised learning law, and it is
used with a network of înstars to capture the characteristics ot the
input and output patterns for data compression. In implementation,
the weight vectors are initialized to zero prior to learning.

1.6.8 Discussion on Basic Learning Laws

Table 1.2 gives a summary of the basic learning laws described so

Table 1.2 Summary of Basic Learning Laws (Adapted from [Zurada, 1992)

Learning Weight adjustment Initial Learning

law weights
Near zero Unsupervised
Hebbian Aw nfwa) a,
for i = 1,2,..,M
Random Supervised
Perceptron uw, =
n lb,-sgn(w a)]la,
1, 2, M
for j= -.,

Random Supervised
3 Delta Aw, =
n [b, -fwa)] fw a) a,
=
n lb, -s] fiz) a,
for j = 1, 2, .., M

w = n lb-wal a, Random Supervised

Widrow-
Hoff forj = 1,2,.,M
Near zero Supervised
Correlation Aw, nba,
for j = 1, 2, .., M
Random but Unsupervised
Winner
normalised
take-all k is the winning unit,
for j 1,2, M
=
..,

Supervised
Outstar Aw=n 6,-w), Zero
forj= 1, 2,.., M
 Networks
Neural
Basics of Artificial
36
and the
(supervised/unsupervised)

Tar. It shows the type of learning for continuous)

function (sgn for discrete/ f.) the

ature of the output applicable. The most important issue i ns o m e
each law is the weights t0
Tor
which of
laws is the convergence
appication of these convergence and the limit values
The
Tinal limit values as desired.
the initial setting of
the weights prior to
he weights tepend on
learning, and on the learning rate parameter.lead to the, s u m of the
Hebb's law and the corpelation law
The law) components
correlations and output (for Hebb's
between input
law)
correlation
and between input and desired output (for
achieve this, the starting
components, respectively. But in order to zero. The
values should be small random values
near
nta weghtrate parameter n should be close to one. Pypically, the
set
earning
of patterns are applied only once in the training process. In some
variations of these (as in the principal component learning to be
discussed in Chapter 6), the learning rate parameter is set to a small
value (< 1) and the training_patterns are applied several times to
achieve convergence.
The
perceptron, delta and LMS learning laws lead to final steady
state values (provided they
converge), only when the weight
adjustments are small. Since the correction depends on the error
between the desired output and the actual output, only a small
portion of the error is used for adjustment of the weights each time.
Thus the learning rate
parameter n
be set to random values. The
<< 1. The initial weights could
set
applied several times to achieve orraining patterns need to be
convergence will naturally be
convergence, if it exists. The
to the final faster if the
starting weights are close
steady valués.
Theweights in the instar and outstar learning laws
the mean values of a get of input and converge to
desired output
respectively. these cases the learning rate
In patterns,
set to a value less than one (n< 1). The parameter is typically
can initialized to any randem
be weights in the case
to small random values values, and in the case ofofoutstar
instar
applied several times to near zero. The set of training
achieve patterns are
Besides these basic convergence.
laws evolved primarily learning
for laws there are many other
Hassoun, 1995, Ch. 3). Some of application in different learning
discussed at situations
places in the later chapters. them will be
appropriate
1.7 Summary
In this chapter we have seen the
n t interest in the motivation and
artificial neural
study problems basedbackground
of
We have on
for the
reviewed the models
hiological features ofusing
neural network and discussed
the
feasibility the
of
realizing
 Review GQuestions
37
some of these features
(PDP) models (Appendixthrough parallel and distributed
A). In particular, the associativeprocessing
fault tolerance and memory,
concept learning features could be demonstrated
through these PDP models. Some key
neural networks developments in artificial
to the
were
presented to show how the field has evolved
present state of
An artificial neural understanding.
network is built using a few basic
blocks. The building blocks were building
of artificial neurons and the
introduced starting with the models
topology of a few basic structures. While
developing artificial neural networks for specific applications, the
weights are
adjusted in
systematic
a
using learning laws.
manner
We have discussed some basic
learning
But the full potential of a neural
laws and their characteristics.
network can be exploited if we can
incorporate in its operation the neuronal activation and synaptic
dynamics of a biological neural network. Some features of these
dynamics are discussed in the next chapter.
Review Questions
1. Describe some attractive features of the biological neural
network that make it superior to the most sophisticated Artificial
Intelligence computer system for pattern recognition tasks.
2. Explain briefly the terms cell body, axon, synapse, dendrite and
neuron with reference to a biological neural network.
3. Explain briefly the operation of a biological neural network.
4. Compare the performance of a computer and that of a biological
neural network in terms of speed of processing, size and
complexity, storage, fault tolerance and control mechanism.
5. Give two examples of pattern recognition tasks to illustrate the
superiority of the biological neural network over a conventional
computer system.
6. What are the main differences among the three models of artificial
neuron, namely, MeCulloch-Pitts, perceptron and adaline?
7. What is meant by topology of artificial neural networks? Give a
few basic topological structures of artificial neural networks.
8. What is the distinction between learning equation and learning
law?
9. What are the basic learning laws?
10. Explain the significance of the initial values of weights and the
learning rate parameter in the seven basic learning laws.
11. Identify supervised and unsupervised basic learning laws.
12. Compare LMS, perceptron and delta learning laws.