ITF309

Artificial Neural Networks

3/9/2024 EELU ITF309 Neural Network Lecture 2 1

 Outline
• Supervised Hebbian Learning

3/9/2024 EELU ITF309 Neural Network Lecture 2 2

Review: Perceptron Architecture
w 1, 1 w 1, 2  w 1, R

W = 2, 1 2, 2  2, R
w w w

w S, 1 w S, 2  w S, R

T
w i, 1 1w

w i, 2 T
iw = W = 2w

w i, R T
Sw

T
ai = har dlimn i  = hardlim iw p + bi

3/9/2024 EELU ITF309 Neural Network Lecture 2 3

Review: Perceptron Learning Rule
• Perceptron learning rule

W new
W old
 ep T

b new
b old
e
• How can we derive the perceptron learning
rule or justify it?

3/9/2024 EELU ITF309 Neural Network Lecture 2 4

 Objectives
• The Hebb rule, proposed by Donald Hebb in
1949, was one of the first neural network learning
laws.
• A possible mechanism for synaptic modification
in the brain.
• Use the linear algebra concepts to explain why
Hebbian learning works.
• The Hebb rule can be used to train neural
networks for pattern recognition.

3/9/2024 EELU ITF309 Neural Network Lecture 2 5

 Hebb’s Postulate
• Hebbian learning
(The Organization of Behavior)
When an axon of cell A is near enough to excite a cell B
and repeatedly or persistently takes part in firing it; some
growth process or metabolic change takes place in one
or both cells such that A’s efficiency, as one of the cells
firing B, is increased.

3/9/2024 EELU ITF309 Neural Network Lecture 2 6

 Hebb’s Postulate

3/9/2024 EELU ITF309 Neural Network Lecture 2 7

 Hebb’s Postulate
• The Hebb’s postulate
– When an axon of cell A is near enough to excite a cell B and
repeatedly or persistently takes part in firing it, some growth
process or metabolic change takes place in one or both cells such
that A’s efficiency, as one of the cells firing B, is increased

3/9/2024 EELU ITF309 Neural Network Lecture 2 8

 Hebb (1949)
“When an axon of cell A is near enough to
excite a cell B and repeatedly or persistently
takes part in firing it, some growth process or
metabolic change takes place in one or both
cells such that A’s efficiency, as one of the
cells firing B, is increased”.

3/9/2024 EELU ITF309 Neural Network Lecture 2 9

 Hebb’s rule
• Each time that a particular synaptic connection is
active, see if the receiving cell also becomes active.
If so, the connection contributed to the success
(firing) of the receiving cell and should be
strengthened. If the receiving cell was not active in
this time period, our synapse did not contribute to
the success the trend and should be weakened.

3/9/2024 EELU ITF309 Neural Network Lecture 2 10

 Hebb Rule
• If two neurons on either side of a synapse are
activated simultaneously, the strength of the
synapse will increase

w new
ij = wol d +  f  a g  p 
ij i iq j jq

Presynaptic Signal

Postsynaptic Signal

3/9/2024 EELU ITF309 Neural Network Lecture 2 11

 Hebb Rule – cont.

Simplified Form:

wijnew = w old
ij + aiq p jq
Supervised Form:

winew
j = w ol d + t p
ij iq jq
Matrix Form:
ne w old T
W =W + t q pq

3/9/2024 EELU ITF309 Neural Network Lecture 2 12

 Linear Associator
• Associative memory
– The task is to learn Q pairs of prototype
input/output vectors
{p1, t1}, {p2, t2}, ....., {pQ, tQ}
– In other words, if the network receives an input p
= pq, then it should produce an output a = tq
– In addition, for an input similar to a prototype the
network should produce an output similar to the
corresponding output

3/9/2024 EELU ITF309 Neural Network Lecture 2 13

 Linear Associator – cont.

3/9/2024 EELU ITF309 Neural Network Lecture 2 14

 Linear Associator – cont.
• Hebb rule for the linear associator
W new
W old
 t q pq
T

3/9/2024 EELU ITF309 Neural Network Lecture 2 15

 Batch Operation
Q
T T
W= t 1 p1 + t 2 p2 +  + t QpTQ = 
T
t q pq (Zero Initial
q= 1 Weights)

Matrix Form:
T
p1 P = p1 p2  pQ
T
W = t1 t2  tQ p2 = T PT


T
pQ T = t1 t2  tQ

3/9/2024 EELU ITF309 Neural Network Lecture 2 16

 Performance Analysis
 Q  Q
a = Wpk =   t q pTq  pk =  t q pTq pk 
q = 1  q= 1

Case I, input patterns are orthogonal.

T
 pq p k  = 1 q = k

= 0 qk
Therefore the network output equals the target: a = Wpk = t k

Case II, input patterns are normalized, but not orthogonal.

T
a = Wp k = t k +  q q pk 
t  p
q k

Error

3/9/2024 EELU ITF309 Neural Network Lecture 2 17

 Linear Associator – cont.

3/9/2024 EELU ITF309 Neural Network Lecture 2 18

 Example: Orthonormal Case

  0.5    0.5  
  0.5   0.5  
     
1    
1
p1  , t1    , p 2  , t 2   
  0.5   1    0.5 1 
       
  0 .5    0.5
 1 1 0.5  0.5 0.5  0.5
 W  TP   T
 0.5 0.5  0.5  0.5
  1 1 
1 0 0  1
 
 0 1  1 0 
1 1
 Wp1   , Wp 2   . Success!
 1 1
3/9/2024 !
EELU ITF309 Neural Network Lecture 2 19
 Example: Not Orthogonal Case
Banana Apple Normalized Prototype Patterns

–1 1  – 0.5774   0.5774 
   
p1 = 1 p2 = 1  p 1 = 0.5774 , t1 = – 1   p 2 = 0.5774 , t 2 = 1 
   
–1 –1  – 0.5774   – 0.5774 
Weight Matrix (Hebb Rule):
W = TP T = – 1 1 – 0.5774 0.5774 – 0.5774 = 1.1548 0 0
0.5774 0.5774 – 0.5774
Tests:
–0.5774
Banana Wp1 = 1.1548 0 0 0.5774 = – 0.6668
–0.5774

0.5774
Apple Wp2 = 0 1.1548 0 0.5774 = 0.6668
– 0.5774
3/9/2024 EELU ITF309 Neural Network Lecture 2 20
 Not Orthogonal Case

  0.5774    0.5774  
      
p1   0.5774, t1   1, p 2   0.5774 , t 2  1
        
  0.5774   0. 5774 
0.5774  0.5774  0.5774
 W  TP   1 1
T

 0.5774 0. 5774  0.5774 
 0 1.547 0
 Wp1   0.8932, Wp 2  0.8932.

The outputs are close, but do not quite match

the target outputs.
3/9/2024 EELU ITF309 Neural Network Lecture 2 21
 Pseudoinverse Rule
Performance Index: Wpq = tq q = 1, 2,  , Q

Q
2
F  W =  | |t q – Wpq | |
q= 1

Matrix Form: WP = T

T = t1 t2  tQ P = p1 p2  pQ

2 2
FW = | |T – WP| | = | |E| |

  ei j
2 2
|| E || =
i j

3/9/2024 EELU ITF309 Neural Network Lecture 2 22

 Pseudoinverse Rule - (2)
WP = T
Minimize: 2 2
FW = | |T – WP| | = | |E| |

If an inverse exists for P, F(W) can be made zero:

W = TP–1

When an inverse does not exist F(W) can be minimized

using the pseudoinverse:
W = TP +
–1
P + = P TP  P T

3/9/2024 EELU ITF309 Neural Network Lecture 2 23

 Relationship to the Hebb Rule
Hebb Rule
T
W = TP

Pseudoinverse Rule
W = TP +
–1
P + = P TP  P T

If the prototype patterns are orthonormal:

PT P = I

+ T –1 T T
P = P P P = P

3/9/2024 EELU ITF309 Neural Network Lecture 2 24

 Example
+
 –1   1   
    +  –1 1 
p
 1 = ,
1 1 t = –1  p
 2 = ,
1 2 t = 1  W = TP = – 1 1  1 1 
     
 – 1   – 1   –1 –1 

–1
+ T –1 T
P =  P P P = 3 1 – 1 1 – 1 = – 0.5 0.25 – 0.25
13 1 1 –1 0.5 0.25 – 0.25

W = T P+ = – 1 1 – 0.5 0.25 – 0.25 = 1 0 0

0.5 0.25 – 0.25

–1 1
Wp1 = 1 0 0 1 = – 1 Wp2 = 1 0 0 1 = 1
–1 –1

3/9/2024 EELU ITF309 Neural Network Lecture 2 25

Another Pattern Recognition Example
• P7.4 on pp. 7-22 – 7-23
• Given the following three patterns

3/9/2024 EELU ITF309 Neural Network Lecture 2 26

Another Pattern Recognition Example – cont.

3/9/2024 EELU ITF309 Neural Network Lecture 2 27

 Another Pattern Recognition Example – cont.

• For the following test pattern

3/9/2024 EELU ITF309 Neural Network Lecture 2 28

 Associative memory
• Auto-associative memory, often misunderstood
to be only a form of back-propagation or other
neural networks. It is actually a more generic
term that refers to all memories that enable one
to retrieve a piece of data from only a tiny
sample of itself.
• Associative memory is defined as the ability
to learn and remember the relationship
between unrelated items such as the name of
someone we have just met or the aroma of a
particular perfume.
3/9/2024 EELU ITF309 Neural Network Lecture 2 29
Autoassociative & heteroassociative
• That is to say, a system that "associates" two
patterns is one that, when presented only of
these patterns later, the other can be reliably
recalled.
• A memory that reproduces its input pattern as
output is referred to as autoassociative (i.e.
associating patterns with themselves). One that
produces output patterns dissimilar to its inputs
is termed heteroassociative (i.e. associating
patterns with other patterns).

3/9/2024 EELU ITF309 Neural Network Lecture 2 30

 With Hebbian learning (1949) , two
learning methods are possible
• With supervised learning an input is
associated with an output
– If the input and output are the same, we speak
of auto-associative learning
– If they are different it is called hetero-
associative learning
• With unsupervised learning there is no
teacher: the network tries to discern
regularities in the input patterns
3/9/2024 EELU ITF309 Neural Network Lecture 2 31
 Associative memory
Hetero associative Auto associative

A α A A

B β B B

Auto-association (Same Patterns)

A memory
A
Hetero-association (Different Patterns)

Niagara Waterfall
memory
3/9/2024 32
EELU ITF309 Neural Network Lecture 2
 Autoassociative Memory
• Autoassociative memory tries to memorize
the input patterns
– The desired output is the input vector

3/9/2024 EELU ITF309 Neural Network Lecture 2 33

 Autoassociative Memory – cont.
• Linear autoassociator
– For orthogonal patterns with elements to be either 1
or -1, the prototypes are the eigenvectors of the
weight matrix given by the Hebb rule
• The given pattern is not recalled correctly but is amplified
by a constant factor
– See P7.5 (pp. 7-23 – 7-25)

3/9/2024 EELU ITF309 Neural Network Lecture 2 34

 Autoassociative Memory – cont.

3/9/2024 EELU ITF309 Neural Network Lecture 2 35

 Autoassociative Memory

T
p1 = – 1 1 1 1 1 –1 1 –1 –1 –1 – 1 1 1 –1  1 – 1

T T T
W = p1 p1 + p2 p2 + p3 p3

3/9/2024 EELU ITF309 Neural Network Lecture 2 36

 Tests
50% Occluded

67% Occluded

Noisy Patterns (7 pixels)

3/9/2024 EELU ITF309 Neural Network Lecture 2 37

 Autoassociative Memory – cont.
• How can we determine weights and bias for a
network with bias?
– P7.6 (pp. 7-25 --- 7-27)
• How can we determine weight and bias for a
network with desired outputs are 0 and 1’s
instead of -1 and 1?
– P7.7 (pp. 7-27 – 7-28)

3/9/2024 EELU ITF309 Neural Network Lecture 2 38

 Autoassociative Memory – cont.

3/9/2024 EELU ITF309 Neural Network Lecture 2 39

 Variations of Hebbian Learning
• Hebb rule is the most fundamental learning rule and
many learning rules are related to it

ne w old T
Basic Rule: W = W + tq pq

ne w old T
Learning Rate: W = W +  tq pq

ne w old T old old T

Smoothing: W = W +  tq pq –  W = 1 –  W + tq pq

Delta Rule: Wnew = Wol d +  tq – aq pTq

Unsupervised: Wne w = Wold + aqpTq

3/9/2024 EELU ITF309 Neural Network Lecture 2 40