Pattern Classifiers

Hebb Nets
• A single -layer feedforward neural net trained through the Hebb
learning is known as a Hebb net.
• The hebb learning rule is the training procedure for a Hebb net where
a net is trained to implement the logical AND function.
 Hebb Rule
• The Hebb rule is one of the earliest learning rules forANNs. According
to this rule the weight adjustment is computed as
• ∆wi = xi*t
• where t is the target activation
• There are certain points to be kept in mind regarding the Hebb
learning rule.
• First, Hebb rule cannot learn when the target is 0.
• This is because the weight adjustment ∆wi becomes zero when t=0,
irrespective of the value of xi .
 Hebb Rule

• Hence, obviously ,the hebb rule results in better learning if the

input /output both are in bipolar form.
Limitation of Hebb learning:
• The most striking limitation of the Hebb rule is it does not guarantee
to learn a classification instance even if the classes are linearly
separable.
• Hebb net is one of the earliest neural net meant for classification
tasks.
• However, it has very limited capacity.
• A more powerful ANN is the famous perceptrons.
 Procedure Hebb Learning
Step 1. Initialize all weights to 0.
wi =0 for all i=0 to m .
Step 2. For each training vector and target output pair ,
s:t, do steps 3-5.
Step 3. Assign the input vectors to the input layer.
x0 =1, and xi =si for all i=1 to m.
Step 4. Activate the output unit with the target output
y_out=t
Step 5. Adjust the weights
wi (new) = wi(old)+xi *y_out
Step 6. Stop
 Example:Realizing the logical AND
function throughHebb learning.
• To realize a two input AND function we need a net with two input units
and one output unit.
• A bias is also needed.
• Hence the structure of the required neural net should be as shown in
following fig.
• Moreover, the input and output signals must be bipolar form, rather
than the binary form, so that the net may be train properly.
• Considering the truth table of AND operation, and the fact that the bias
is permanently set to 1, we get the training set depicted in Following
Table.
 Structure of a neural net to realize the
AND function

1 X0 W0

X1 Y
x1 W1 y_out= x1^x2

x2 X2 W2
Table: Training set for AND function
Input Patterns Output
X0 X1 X2 t
+1 1 1 1
+1 1 -1 -1
+1 -1 1 -1
+1 -1 -1 -1
Example:Realizing the logical AND function
through Hebb learning.

• During the training process, all weights are initialized to 0.

• Therefore , initially W0=W1=W2=0
• At each training instance, the weights are changed according to the
formula
• W1(new)=Wi(old)+∆Wi
• Where ∆Wi, the increment in Wi , is computed as ∆Wi = xi*t.
• After initialization , the progress of the learning process by the
network is shown in table.
Hebbian learning of AND function
# Training Pattern Target Weight Adjustment Weights
output
x0 x1 x2 t ∆w0 ∆w1 ∆w2 w0 w1 w2
0 0 0 0
1 1 1 1 1 1 1 1 1 1 1
2 1 1 -1 -1 -1 -1 1 0 0 2
3 1 -1 1 -1 -1 1 -1 -1 1 1
4 1 -1 -1 -1 -1 1 1 -2 2 2
Hebbbian training to learn AND function
x2 x2
(-1,1) (1,1) (-1,1) (1,1)

x1
x1
(-1 ,-1) (1,-1) (-1,-1) (1,-1)
x2= -x1-1 x 2= 0

(a) After 1st pattern (1,1) ( b) After 2nd pattern (1,-1)

Hebbbian training to learn AND function

x2 x2

(-1,1) (1,1) (-1,1) (1,1)

x1 x1
x2=-x1+1 x2=-x1+1
( -1,-1) (1,-1) (-1,-1) (1,-1)
(c) After 3rd pattern (-1,1) (d) After 4th pattern (-1,-1)
Example:Realizing the logical AND function through
Hebb learning.

• Hence , on completion of one epoch of training , the weight vector is

W=[w0, w1, w2]=[-2,2,2].
• The progress in learning by thenet can be visualized by observing the
orientation of the decision line after each training instance.
• Putting the values of the interconnection weights in the equation
w 1 w0
• x2=- x 1 -
w 2
w2
Example:Realizing the logical AND function through
Hebb learning.

• We get (i) x2=-x1-1 after the 1st training instance,

(ii) x2=0, after the 2nd training instance,
(iii) x2=-x1+1 after the 3rd training istance, and finally,
(iv) x2=-x1+1 after the 4th training instance.

• The progress in the learning process is depicted in Fig.(a)-(d).

• We see that after training with the first pattern [x0,x1,x2]=[1,1,1],the
ANN learns to classify two patterns (-1,-1) and (1,1) successfully.
• But it fails to classify correctly the other two patterns (-1,1) and (1,-1)
Example:Realizing the logical AND function through Hebb
learning.

• After learning with the second pattern (1,-1) the situation is better.
• Only (-1,1) is still misclassified.
• This is corrected after training with the third pattern (-1,1).
• Now all the -1 producing patterns are i.e., (1,-1), (-1,1) and (-1,-1) are in
the same class and the remaining pattern (1,1) constitutes the other
class.
Structure of the Hebb net

1 x0

w0
x1 x1
w1
y
xi
xi wi y_out

xm
xm wm
Structure of the Hebb net
• It should be noted that the input unit X0 and the associated weight
w0 play the role of the bias.
• The activation X0 is always kept at 1.
• Hence, the expression for the adjustment of w0 becomes
wo(new)= w0(old)+1xy_out=w0(old)+y_out
Perceptrons
 Perceptron Learning Rule
• Let us consider aimple ANN consisting of a single perceptron Y with
m+1 input units X0 ---------------Xm as shown in following fig.
• The corresponding weights of interconnection are w0---------wm.
• The bias is introduced as the weight w0 connected to the input X0
whose activation is fixed at 1.
• For the output unit , it is convenient to use the bipolar activation
function:
1, if y_in>0
y_out={ 0 , if y_in=0
-1, if y_in<0
Structure of a perceptron

1 x0

w0
x1 x1
w1
y
xi
xi wi y_out

xm
xm wm
Perceptron Learning Rule

• Now let X=(x0--------xm) be a trainng vector for which the output of the
perceptron is expected to be t,where t=1,0 or -1.
• The current combination of the weights is given by the weight vector
W= (w0----------wm ).
• If the perceptron produces the desire output , then the weights
w0-----wm need not be changed and they are to be kept unaltered.
• If, however , the perceptron misclassifies X negatively(meaning, it
erroneously produces -1 instead of the desired output +1) then the
weight should be appropriately increased.
Perceptron Learning Rule

• Conversely, the weights areto be decreased in case the perceptron

misclassifies X positively(i.e., it erroneously produces +1 instead of the
desire output -1).

• The learning strategy of the perceptron is summarized in following

table.
Perceptron Learning Rule

Condition Action
1.The perceptron classifies the No change in the current set of
input pattern correctly(y_out=t) weights w0,w1-------,wm
2.The perceptron misclassifies the Increase each wi by ∆wi ,where ∆wi
input pattern negatively(y_out=-1, is proportional to xi for all
but t=+1)
i=0,1,-------m
3.The perceptron misclassifies the Decrease each wi by ∆wi ,where
input pattern positively(y_out=+1, ∆wi is proportional to xi for all
but t=-1)
i=0,1,-------m
Perceptron Learning Rule

• Hence the perceptron learning rule is informally stated as

If the output is erroneous Then adjust the interconnection weights Else
leave the interconnection weights unchanged.

• In more precise terms,

IF y_out≠t Then
FOR i=0 To m DO wi(new)= wi(old)+ɳ*t*xi
ELSE
FOR i=0 to m do wi(new)= wi(old)
Structure of an ANN with several perceptrons

x0 X0 w01 Y1 y_out1
w0n w0j
Yj
xi Xi wi1 wij y_out2
win
Xm wm1 wmj Yn y_outn
xm wmn
Input layer Output layer
Perceptron Learning Rule

• Thus the perceptron learning rule can be formulated as

∆wi = ɳ*(t-y_out)*xi for i=0,1,-------m
• Here ɳ is a constant known as the learning rate.
• It should be noticed that when a training vector is correctly classified
then y_out=t, the weight adjustment ∆w i =0.
• When y_out =-1 but t=+1, i.e., the patern is misclassified negatively,
then t-y_out=+2 so that ∆wi =0 is incremental and is proportional to x i .
• If , however , the input pattern is misclassified positively, the adjustment
is decremental, and obviously, proportional to x i .
Perceptron Learning Rule

• Using the matrix notation, the perceptron learning rule may now be written
as
∆W=ɳ*(t-y_out)*X ---------------------(1)
• Where ∆W and X are the vectors corresponding to the interconnection
weights and the inputs.
• ∆W=[∆w0,∆w1,---------------∆wm]
and X=[x0, x1,----------------xn]
• Equation 1 can be easily extended to a network of several perceptron at the
output layer.
• Such architecture is shown in above Fig of slide no 25.
Perceptron Learning Rule

• The input to the perceptron are

w01 ... ... ... wi1 ... ... ... wm1
x0
. ... ... ... . ... ... ... .
.
. ... ... ... . ... ... ... .
.
y _ in 1 = . ... ... ... . ... ... ... .
X[ . ]
[ w0 j ... ... ... wij ... ... ... wmj ]
... xi
. ... ... ... . ... ... ... .
y _ in j
.
. ... ... ... . ... ... ... .
.. .
. ... ... ... . ... ... ... .
.
y _ in n w0 n ... ... ... win ... ... ... wmn
xm
Perceptron Learning Rule

• or Y_in T =WT*XT
• For such an ANN , the djustment ∆wij of the weight wij is given by
∆wij =ɳ*(tj -y_outj )*xi
• Let us assume the following matrix notations:
w01 w02 .... .... w0 n
w11 w12 ... ... w1n
∆W= ... ... ... ... ..
... ... ... ... ...
wm1 wm 2 ... ... wmn
Perceptron Learning Rule

• T= [t1 ... ... ... tn ]

• X= [ x0 ..... .... .... xm ]

• and Y_out= [ y _ out1 ... ... ... y _ out2 ]

Perceptron Learning Rule

• Then the expresion of the perceptron learning rule becomes

[∆w]T =ɳ*[T-Y_out]T *X
Learning the logical AND function by a
perceptron
• Let us train a perceptron to realize the logical AND function.The training
patterns and the corresponding target outputs for the AND operation
where the input and the outputs are in bipolar form are given in
following table.The structure of the perceptron is same as showin in
following fig.Activation function for the output unit is:

1, if y_in>0
• y_out=g(y_in)={ 0, if y_in=0
-1, if y_in<0
 Structure of a neural net to realize the
AND function

1 X0 W0

X1 Y
x1 W1 y_out= x1^x2

x2 X2 W2
Table: Training set for AND function
Input Patterns Output
X0 X1 X2 t
+1 1 1 1
+1 1 -1 -1
+1 -1 1 -1
+1 -1 -1 -1
Learning the logical AND function by a
perceptron
• The successive steps of computation with the first training pair are
given below.
i) Initialize weights: w0=w1=w2=0 and the learning rate ɳ=1
ii) For the training pair s:t=(1,1,1):(1) , compute the net input y_in
and the activation y_out :
y_in =w0*x0+w1*x1+w2*x2
= 0*1+0*1+0*1
=0
y_out=0
Learning the logical AND function by a perceptron

• iii) Apply the perceptron learning rule to find the weight

adjustment .The rule is:
• If y_out =t, then the weights are not change, otherwise change thw
weights according to the formula wi(new)=wi(old)+∆wi where
• ∆wi =ɳ*t*xi
• In the present case , y_out =0 and the t=1 and y_out≠t.
• Therefore the weights are to be adjusted.
• w0(new)=w0(old)+∆w0
=w0(old)+ɳ*t*x0
=0+1*1*1=1
Learning the logical AND function by a
perceptron
• Similarly, w1(new)=w1(old)+∆w1
=w1(old)+ɳ*t*x1
=0+1*1*1=1
and, w2(new)=1.
• Putting these values of the weights in the formula
w 1 w0
x 1
x2 = - w 2
- w2
Learning the logical AND function by a perceptron

• We get the separating line as x2=-x1-1, which classifies the pattern (1,1) and (-1,-
1), correctly but misclassifies the patterns (-1,1) and (1,-1).
• Following fig represent the above condition.
• A trace of the remaining steps of the first epoch of the training (an epoch of
training consists of successive application of each and every training pair of the
training set for once ) are noted in following table.
• Hence the set of weights obtained at the end of the first epoch of training is (-1,
1,1) which represent the decession line
x2=-x1+1 .
• It can be easily verified that this line ( and the corresponding set of
weights)successfully realizes the AND function for all possible input combinations.
• Hence , the is no need of further training.
Learning the logical AND function by a
perceptron
x2
(-1,1) (1,1)

x1

(-1 ,-1) (1,-1)

x2= -x1-1

After 1st pattern (1,1)

Perceptron learning of AND function
# Input Pattern Net Activation Target Weight Adjustments Weights
input output
x0 x1 x2 y_in y_out t ∆w0 ∆w1 ∆w2 w0 w1 w2
0 0 0 0
1 1 1 1 0 0 1 1 1 1 1 1 1
2 1 1 -1 1 1 -1 -1 -1 1 0 0 2
3 1 -1 1 2 1 -1 -1 1 -1 -1 1 1
4 1 -1 -1 -3 -1 -1 0 0 0 -1 1 1
Procedure Perceptron -Learning
•The notable points regarding procedure perceptron-learning are given below:
1. The input vectors are allowed to be either binary or bipolar. However, the
outputs must be in biploar form.
2. The bias w0 is adjustable but the threshold θ used in the activation function is
fixed.
3. Learning takes place only when the computed output does not match the target
output. Moreover , as ∆wi =ɳ*t*xi the weight adjustment is 0 if xi =0.Hence no
learning take place if either the input is 0,or the computed output matches the
target output.Consequently, as training proceeds,more and more training pattern
yeild correct results and less learning is required by the net.
Procedure Perceptron -Learning
4. The threshold θ of the activation function may be interpreted as a separating band
of width 2θ between the region of positive response and the negative response. The
band itself is undecided in the sense that if the net input falls within the range [-θ,θ],
the activation is neither positive , nor negative .Moreover, changing the value of θ
would change the width of the undecided region, along with the position of the
separating lines.Therefore, for perceptron, the bias and the threshold are no longer
interchangeable.
5.The band separating the regions of the positive response from that of the negative
response is defined by the pair of lines.
w0x0+w1x1+w2x2=θ
w0x0+w1x1+w2x2=-θ
Procedure for perceptron-Learning
Step 1:Initialize all weights w0 ,....... ,wm.
Step 2: set learning rate ɳ such that 0<ɳ≤1, and threshold θ.
Step 3: For each training pair s:t do steps 4-8.
Step 4: Activate the input units, xi=si , for i=0,----------,m.
Step 5: Compute the net input to the output unit
m

y_in=
i  0
w i x i

Step 6: Compute the activation of the output unit using the following function.
Procedure for perceptron-Learning

1, if y_in >θ
y_out= { 0, if y_in -θ≤y_in≤θ
-1, if y_in <-θ

Step 7:If there is an error, i.e., y_out≠t, then adjust the weights as follows
wi(new)=wi(old)+n*t*xi
If, however, no error has occurred, the weights are kept unchanged.
Step 8: If there were no error, i.e., y_out=t, for the entire set of training
pairs,then stop, otherwise, go to step 3.
Any Question?
Thank You