CS407 Neural Computation

Lecture 4:
Single Layer Perceptron (SLP)
Classifiers

Lecturer: A/Prof. M. Bennamoun

 Outline
ƒ What’s a SLP and what’s classification?
ƒ Limitation of a single perceptron.
ƒ Foundations of classification and Bayes Decision making theory
ƒ Discriminant functions, linear machine and minimum distance
classification
ƒ Training and classification using the Discrete perceptron
ƒ Single-Layer Continuous perceptron Networks for linearly
separable classifications
ƒ Appendix A: Unconstrained optimization techniques
ƒ Appendix B: Perceptron Convergence proof
ƒ Suggested reading and references
What is a perceptron and what is
a Single Layer Perceptron (SLP)?
Perceptron
„ The simplest form of a neural network
„ consists of a single neuron with adjustable
synaptic weights and bias
„ performs pattern classification with only two
classes
„ perceptron convergence theorem :
– Patterns (vectors) are drawn from two
linearly separable classes
– During training, the perceptron algorithm
converges and positions the decision
surface in the form of hyperplane between
two classes by adjusting synaptic weights
 What is a perceptron?
m

Bias v = ∑w x +b
k
j =1
kj j k
bk
x1 wk1 Activation

x2 wk2
function
y = ϕ (v )
k k

vk Output
Σ ϕ(.) yk
...

...

xm wkm Summing
junction Discrete Perceptron:
Input Synaptic
weights
ϕ (⋅) = sign (⋅)
signal

Continous Perceptron:
ϕ (⋅) = S − shape
Activation Function of a perceptron

+1
+1

vi vi

-1

Signum Function
(sign) Continous Perceptron:
Discrete Perceptron: ϕ (v) = s − shape
ϕ (⋅) = sign (⋅)
SLP Architecture
Single layer perceptron

Input layer Output layer

Where are we heading? Different
Non-Linearly Separable Problems
http://www.zsolutions.com/light.htm

Types of Exclusive-OR Classes with Most General

Structure Decision Regions Problem Meshed regionsRegion Shapes

Single-Layer Half Plane A B

Bounded By B
Hyperplane A
B A

Two-Layer Convex Open A B

Or B
Closed Regions A
B A

Three-Layer Arbitrary
(Complexity A B
Limited by No. B
A
of Nodes) B A
Review from last lectures:
Implementing Logic Gates with
Perceptrons http://www.cs.bham.ac.uk/~jxb/NN/l3.pdf

ƒ We can use the perceptron to implement the basic logic gates (AND, OR
and NOT).
ƒ All we need to do is find the appropriate connection weights and neuron
thresholds to produce the right outputs for each set of inputs.
ƒ We saw how we can construct simple networks that perform NOT, AND,
and OR.
ƒ It is then a well known result from logic that we can construct any logical
function from these three operations.
ƒ The resulting networks, however, will usually have a much more complex
architecture than a simple Perceptron.
ƒ We generally want to avoid decomposing complex problems into simple
logic gates, by finding the weights and thresholds that work directly in a
Perceptron architecture.
Implementation of Logical NOT, AND, and OR
ƒ In each case we have inputs ini and outputs out, and need to determine
the weights and thresholds. It is easy to find solutions by inspection:
 The Need to Find Weights Analytically
ƒ Constructing simple networks by hand is one thing. But what about
harder problems? For example, what about:

ƒ How long do we keep looking for a solution? We need to be able to

calculate appropriate parameters rather than looking for solutions by trial
and error.
ƒ Each training pattern produces a linear inequality for the output in terms
of the inputs and the network parameters. These can be used to compute
the weights and thresholds.
Finding Weights Analytically for the AND Network

ƒ We have two weights w1 and w2 and the threshold θ, and for each
training pattern we need to satisfy

ƒ So the training data lead to four inequalities:

ƒ It is easy to see that there are an infinite number of solutions. Similarly,

there are an infinite number of solutions for the NOT and OR networks.
Limitations of Simple Perceptrons

ƒ We can follow the same procedure for the XOR network:

ƒ Clearly the second and third inequalities are incompatible with the
fourth, so there is in fact no solution. We need more complex networks,
e.g. that combine together many simple networks, or use different
activation/thresholding/transfer functions.
ƒ It then becomes much more difficult to determine all the weights and
thresholds by hand.
ƒ These weights instead are adapted using learning rules. Hence, need to
consider learning rules (see previous lecture), and more complex
architectures.
E.g. Decision Surface of a Perceptron
x2
x2
+
+ + -
+ -
- x1
x1
+ - +
-
-
Linearly separable Non-Linearly separable

• Perceptron is able to represent some useful functions

• But functions that are not linearly separable (e.g. XOR)
are not representable
The Discrete Perceptron
Discrete Perceptron Training Algorithm
• So far, we have shown that coefficients of linear
discriminant functions called weights can be
determined based on a priori information about sets of
patterns and their class membership.
•In what follows, we will begin to examine neural
network classifiers that derive their weights during the
learning cycle.
•The sample pattern vectors x1, x2, …, xp, called the
training sequence, are presented to the machine along
with the correct response.
 Discrete Perceptron Training Algorithm
- Geometrical Representations http://140.122.185.120
Zurada, Chapter 3

(Intersects the origin

point w=0)
5 prototype patterns in this case: y1, y2, …y5
If dim of augmented pattern vector is > 3, our power of visualization are no longer of assistance. In this case,
the only recourse is to use the analytical approach.
 Discrete Perceptron Training Algorithm
- Geometrical Representations…
•Devise an analytic approach based on the geometrical
representations
– E.g. the decision surface for the training pattern y1
( )
∇ w w t y1 = y1 Gradient
(the direction of
y1 in Class 1 If y1 in steepest increase)
(see previous slide) Class 1:
Weight w ′ = w1 + cy1
Space
If y1 in c controls the
Class 2: size of adjustment

y1 in Class 2 w ′ = w1 − cy1
c (>0) is the correction
Weight increment (is two times the
Space learning constant ρ
introduced before)
(correction in negative gradient direction)
Discrete Perceptron Training Algorithm
- Geometrical Representations…
 Discrete Perceptron Training Algorithm
- Geometrical Representations…

w1t y
cy = y =p
yt y

Note 1: p=distance so >0

Note 2: c is not constant and depends on the current training pattern as expressed by eq. Above.
 Discrete Perceptron Training Algorithm
- Geometrical Representations…
•For fixed correction rule: c=constant, the correction of
weights is always the same fixed portion of the current
training vector
– The weight can be initialised at any value

•For dynamic correction rule: c depends on the distance

from the weight (i.e. the weight vector) to the decision
surface in the weight space. Hence
Current weight Current input
pattern

– The initial weight should be different from 0.

(if w1=0, then cy =0 and w’=w1+cy=0, therefore no possible adjustments).
 Discrete Perceptron Training Algorithm
- Geometrical Representations…
•Dynamic correction rule: Using the value of c from previous slide as
a reference, we devise an adjustment technique which depends on
the length w2-w1 λ=2: Symmetrical reflection w.r.t decision plane

λ=0: No weight adjustment

Νote: λ is the ratio of the distance

between the old weight vector w1
and the new w2, to the distance
from w1 to the pattern hyperplane
 Discrete Perceptron Training Algorithm
- Geometrical Representations…
•Example:

x1 = 1, x3 = 3, d1 = d 3 = 1 : class 1
x2 = −0.5, x4 = −2, d 2 = d 4 = −1 : class 2
•The augmented input vectors are:

1 − 0.5 3  − 2

y1 =  , y 2 =   , y3 =   y4 =  
1  1  1 1
•The decision lines wtyi=0, for i=1, 2, 3, 4 are sketched
on the augmented weight space as follows:
Discrete Perceptron Training Algorithm
- Geometrical Representations…
Discrete Perceptron Training Algorithm
- Geometrical Representations…
For c = 1 and w1 = [− 2.5 1.75]
t

•Using w ' = w ± cy the weight training with each step can

be summarized as follows:
c
∆w = [d k − sgn(w kt y k )]y k
k

2
•We obtain the following outputs and weight updates:
•Step 1: Pattern y1 is input
 1 
o1 = sgn [− 2.5 1.75]    = −1
 1 
d1 − o1 = 2
− 1.5
w =w +y = 
2 1 1

 2.75 
 Discrete Perceptron Training Algorithm
- Geometrical Representations…
•Step 2: Pattern y2 is input
 − 0.5 

o2 = sgn  [− 1.5 2.75]    =1

  1 
d 2 − o2 = −2
 −1 
w = w −y = 
3 2 2

1.75
•Step 3: Pattern y3 is input
 3 
o3 = sgn [− 1 1.75]    = −1
 1 
d 3 − o3 = 2
 2 
w =w +y = 
4 3 3

 2.75
 Discrete Perceptron Training Algorithm
- Geometrical Representations…
• Since we have no evidence of correct classification of
weight w4 the training set consisting of an ordered
sequence of patterns y1 ,y2 and y3 needs to be recycled.
We thus have y4= y1 , y5= y2, etc (the superscript is used
to denote the following training step number).
•Step 4, 5: w6 = w5 = w4 (no misclassification, thus no
weight adjustments).
•You can check that the adjustment following in steps 6
through 10 are as follows:
w 7 = [2.5 1.75]
t

w10 = w 9 = w 8 = w 7
w11 = [3 0.75]
t

w11 is in solution area.

The Continuous Perceptron
Continuous Perceptron Training Algorithm
http://140.122.185.120
Zurada, Chapter 3

•Replace the TLU (Threshold Logic Unit) with the

sigmoid activation function for two reasons:
– Gain finer control over the training procedure
– Facilitate the differential characteristics to enable
computation of the error gradient

(of current
error function)

The factor ½ does not affect the location of

the error minimum
Continuous Perceptron Training Algorithm…

•The new weights is obtained by moving in the direction

of the negative gradient along the multidimensional error
surface

By definition of the steepest descent concept,

each elementary move should be
perpendicular to the current error contour.
Continuous Perceptron Training Algorithm…
•Define the error as the squared difference between the
desired output and the actual output

Training rule of
continous perceptron
∂ (net ) (equivalent to delta
Since net = w t y, we have = yi i = 1,2,..., n + 1 training rule)
∂wi
Continuous Perceptron Training Algorithm…
Continuous Perceptron Training Algorithm…
Same as previous example (of discrete perceptron) but with a
continuous activation function and using the delta rule.

Same training pattern set as

discrete perceptron example
Continuous Perceptron Training Algorithm…
2
1  2 
E k = d k −  − 1
1 + exp(−λ net )  
k
2

2
1  2 
E1 (w ) = 1 −  − 1 
2  1 + exp[− λ ( w1 + w2 )]  

λ = 1 and reducing the terms simplifies this expression to the following form
2
E1 (w ) =
[1 + exp(w1 + w2 )]2
similarly
2
E2 ( w ) =
[1 + exp(0.5w1 − w2 )]2
2 2
E3 (w ) = E4 ( w ) =
[1 + exp(3w1 + w2 )]2 [1 + exp(2w1 − w2 )]2
These error surfaces are as shown on the previous slide.
Continuous Perceptron Training Algorithm…

minimum
Mutlicategory SLP
 Multi-category Single layer Perceptron nets
•Treat the last fixed component of input pattern vector as
the neuron activation threshold…. T=wn+1

yn+1= -1 (irrelevant wheter it

is equal to +1 or –1)
 Multi-category Single layer Perceptron nets…
• R-category linear classifier using R discrete bipolar
perceptrons
– Goal: The i-th TLU response of +1 is indicative of
class i and all other TLU respond with -1
 Multi-category Single layer Perceptron nets…
•Example 3.5

Indecision regions = regions

should be where no class membership of
(-1, - 1, 1) t an input pattern can be
uniquely determined based on
the response of the classifier
(patterns in shaded areas are
not assigned any reasonable
classification. E.g. point Q for
which o=[1 1 –1]t => indecisive
response). However no
patterns such as Q have been
used for training in the
example.
 Multi-category Single layer Perceptron nets…
For c = 1 and w11 = [1 − 2 0] w12 = [0 − 1 2] and w13 = [1 3 − 1]
t t t

•Step 1: Pattern y1 is input

 10  
   
sgn [1 − 2 0] 2   = 1 Since the
  only w12 = w11
  −
 1 incorrect
response is w 22 = w12
 10   provided
    by TLU3,  1  10  − 9
sgn [0 − 1 2] 2   = −1
w 32 =  3  −  2  =  1 
we have
  − 1 
   − 1 − 1  0 
 10  
   
sgn [1 3 − 1] 2   = 1*
  − 1 
  
 Multi-category Single layer Perceptron nets…
•Step 2: Pattern y2 is input

  2 
   
sgn [1 − 2 0]− 5  = 1*
  − 1  
   1  2  − 1
  2  w13 = 2 − − 5 =  3 
   
sgn [0 − 1 2]− 5  = 1 0  − 1  1 
  − 1  
   w 32 = w 22
  2  w 33 = w 32
   
sgn [− 9 1 0]− 5  = −1
  − 1  
  
 Multi-category Single layer Perceptron nets…
•Step 3: Pattern y3 is input 4 One can
w14 = − 2
( )
verify that
sgn w13t y 3 = 1* the only
 2 
sgn (w y ) = −1
adjusted
3t
weights
2 3
w 42 = w 32
sgn (w y ) = 1
from now
3t
on are those
3 3
w 34 = w 33 of TLU1

• During the second cycle:

w15 = w14
 2 w18 = w17
w16 = 3 5
3 w19 = 3
7 5
w17 = − 2
 4 
 Multi-category Single layer Perceptron nets…
•R-category linear classifier using R continuous bipolar
perceptrons
 APPENDIX B

Perceptron Convergence Proof

Perceptron Convergence Proof Haykin, Chapter 3

Consider the following perceptron:

m
v(n) = ∑ wi (n) xi (n)
i =0

= w T ( n) x( n)

w T x > 0 for every input vector x belonging to class C1

w T x ≤ 0 for every input vector x belonging to class C 2
 Perceptron Convergence Proof…
„ The algorithm for the weight adjustment for the
perceptron
– if x(n) is correctly classified no adjustments to w
w (n + 1) = w (n) if w T x(n) ≤ 0 and x(n) belongs to class C 2

w (n + 1) = w (n) if w T x(n) > 0 and x(n) belongs to class C1

– otherwise

w(n + 1) = w(n) − η (n)x(n) if wT x(n) > 0 and x(n) belongs to class C2

w(n + 1) = w(n) + η (n)x(n) if wT x(n) ≤ 0 and x(n) belongs to class C1

– learning rate parameter η (n) controls adjustment

applied to weight vector
Perceptron Convergence Proof
For η (n) = 1 and w (0) = 0
Suppose the perceptron incorrectly classifies the vectors
x(1), x(2),... such that

wT x(n) ≤ 0 so that : w(n + 1) = w(n) + η (n)x(n)

But sinceη = 1 ⇒
w(n + 1) = w(n) + x(n) for x(n) belonging to C1
Since w(0) = 0, iteratively we find w(n + 1)
w(n + 1) = x(1) + x(2) + ... + x(n) (B1)
Since the classes C1 and C2 are assumed to be linearly
separable, there exists a solution w0 for which wTx(n)>0 for
the vectors x(1), …x(n) belonging to the subset H1(subset of
training vectors that belong to class C1).
Perceptron Convergence Proof
For a fixed solution w0, we may then define a positive number
α as
α = min w x(n) T
0 ( B 2)
x ( n )∈H1

Hence equation (B1) above implies

w T0 w(n + 1) = w T0 x(1) + w T0 x(2) + ... + w T0 x(n)
Using equation B2 above, (since each term is greater or equal
than α), we have T
w 0 w(n + 1) ≥ nα
Now we use the Cauchy-Schwartz inequality:
2 2
(a.b) ≤ a b
2
or
2 (a.b) 2 2
a ≥ 2
for b ≠ 0
b
Perceptron Convergence Proof
This implies that:
2 n 2α 2
w(n + 1) ≥ 2
( B3)
w0
Now let’s follow another development route (notice index k)
w(k + 1) = w(k ) + x(k ) for k = 1, ..., n and x(k) ∈ H1
By taking the squared Euclidean norm of both sides, we get:
2 2 2
w(k + 1) = w(k ) + x(k ) + 2wT (k )x(k )
But under the assumption the the perceptron incorrectly
classifies an input vector x(k) belonging to the subset H1, we
have wT (k )x(k ) < 0 and hence :
2 2 2
w(k + 1) ≤ w(k ) + x(k )
Perceptron Convergence Proof
Or equivalently,
2 2 2
w(k + 1) − w(k ) ≤ x(k ) ; k = 1,...n

Adding these inequalities for k=1,…n, and invoking the initial

condition w(0)=0, we get the following inequality:
n
w(n + 1) ≤ ∑ x(k ) ≤ nβ
2 2
( B4)
k =1

Where β is a positive number defined by;

n
β = max ∑ x(k )
2

x ( k )∈H1
k =1
Eq. B4 states that the squared Euclidean norm of w(n+1)
grows at most linearly with the number of iterations n.
Perceptron Convergence Proof
The second result of B4 is clearly in conflict with Eq. B3.
•Indeed, we can state that n cannot be larger than some
value nmax for which Eq. B3 and B4 are both satisfied with
the equality sign. That is nmax is the solution of the eq.
2
nmaxα2
2
= nmax β
w0
•Solving for nmax given a solution w0, we find that
2
β w0
nmax =
α2
We have thus proved that for η(n)=1 for all n, and for w(0)=0,
given that a sol’ vector w0 exists, the rule for adapting the
synaptic weights of the perceptron must terminate after at most
nmax iterations.