0% found this document useful (0 votes)
47 views21 pages

Single Layer Feedforward Networks

This document summarizes perceptrons and Adaline networks. [1] Perceptrons are single-layer feedforward networks that can learn linearly separable patterns using an error-driven learning rule. [2] Adalines use the same architecture but learn using the delta rule to minimize mean squared error. [3] Both can only learn linearly separable problems; multi-layer networks with nonlinear hidden units are required for more complex tasks.
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
47 views21 pages

Single Layer Feedforward Networks

This document summarizes perceptrons and Adaline networks. [1] Perceptrons are single-layer feedforward networks that can learn linearly separable patterns using an error-driven learning rule. [2] Adalines use the same architecture but learn using the delta rule to minimize mean squared error. [3] Both can only learn linearly separable problems; multi-layer networks with nonlinear hidden units are required for more complex tasks.
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 21

Chapter 2

Single Layer Feedforward Networks


Perceptrons
• By Rosenblatt (1962)
– For modeling visual perception (retina)
– A feedforward network of three layers of units:
Sensory, Association, and Response
– Learning occurs only on weights from A units to R units
(weights from S units to A units are fixed).
– Each R unit receives inputs from n A units
– For a given training sample s:t, change weights between A
and R only if the computed output y is different from the
target output t (error driven)
wSA wAR
S A R
Perceptrons
• A simple perceptron
– Structure:
• Sing output node with threshold function
• n input nodes with weights wi, i = 1 – n
– To classify input patterns into one of the two classes
(depending on whether output = 0 or 1)
– Example: input patterns: (x1, x2)
• Two groups of input patterns
(0, 0) (0, 1) (1, 0) (-1, -1);
(2.1, 0) (0, -2.5) (1.6, -1.6)
• Can be separated by a line on the (x1, x2) plane x1 - x2 = 2
• Classification by a perceptron with
w1 = 1, w2 = -1, threshold = 2
Perceptrons

(-1, -1)
(1.6, -1.6)

• Implement threshold
by a node x0
– Constant output 1
– Weight w0 = - threshold
– A common practice in
NN design
Perceptrons
• Linear separability
– A set of (2D) patterns (x1, x2) of two classes is linearly
separable if there exists a line on the (x1, x2) plane
• w0 + w1 x1 + w2 x2 = 0
• Separates all patterns of one class from the other class
– A perceptron can be built with
• 3 input x0 = 1, x1, x2 with weights w0, w1, w2
– n dimensional patterns (x1,…, xn)
• Hyperplane w0 + w1 x1 + w2 x2 +…+ wn xn = 0 dividing the
space into two regions
– Can we get the weights from a set of sample patterns?
• If the problem is linearly separable, then YES (by perceptron
learning)
• Examples of linearly separable classes
- Logical AND function o x
patterns (bipolar) decision boundary
x1 x2 output w1 = 1
-1 -1 -1 w2 = 1 o o
-1 1 -1 w0 = -1
1 -1 -1 x: class I (output = 1)
1 1 1 -1 + x1 + x2 = 0 o: class II (output = -1)
- Logical OR function
x x
patterns (bipolar) decision boundary
x1 x2 output w1 = 1
-1 -1 -1 w2 = 1
-1 1 1 w0 = 1 o x
1 -1 1
1 1 1 1 + x1 + x2 = 0 x: class I (output = 1)
o: class II (output = -1)
Perceptron Learning
• The network
– Input vector ij (including threshold input = 1) n
– Weight vector w = (w0, w1,…, wn ) net=w⋅i j = ∑
w k ik , j
– Output: bipolar (-1, 1) using the sign node functionk =0

output=¿ { 1 if w⋅i j >0 ¿ ¿¿¿


• Training samples
– Pairs (ij , class(ij)) where class(ij) is the correct classification of ij
• Training:
– Update w so that all sample inputs are correctly classified (if
possible)
– If an input ij is misclassified by the current w
class(ij) · w · ij < 0
change w to w + Δw so that (w + Δw) · ij is closer to class(ij)
Perceptron Learning

Where η > 0 is the learning rate


Perceptron Learning
• Justification

( w    xk )  i j  ( w    class(i j )  i j )  i j
 w  i j    class(i j )  i j  i j
since i j  i  0
( w    xk )  i j  w  i j    class(i j )  i j  i j
 0 if class(i j )  1
 0 if class(i )  1
 j

 new net moves toward class(i j )


• Perceptron learning convergence theorem
– Informal: any problem that can be represented by a
perceptron can be learned by the learning rule
¿
– Theorem: If there is a w such that f (i p  w )  class (i p )
*

for all P training sample patterns {i p , class (i p )}, then for


0
any start weight vector w , the perceptron learning rule

will converge to a weight vector w such that for all p
f (i p  w )  class (i p )
( w and w may not be the same.)
*

– Proof: reading for grad students (Sec. 2.4)


Perceptron Learning
• Note:
– It is a supervised learning (class(ij) is given for all sample input ij)
– Learning occurs only when a sample input misclassified (error
driven)
• Termination criteria: learning stops when all samples are correctly
classified
– Assuming the problem is linearly separable
– Assuming the learning rate (η) is sufficiently small
• Choice of learning rate:
– If η is too large: existing weights are overtaken by Δw =   class (i j )  i j
– If η is too small (≈ 0): very slow to converge
– Common choice: η = 1.
• Non-numeric input:
– Different encoding schema
ex. Color = (red, blue, green, yellow). (0, 0, 1, 0) encodes “green”
Perceptron Learning
• Learning quality
– Generalization: can a trained perceptron correctly classify
patterns not included in the training samples?
• Common problem for many NN learning models
– Depends on the quality of training samples selected.
– Also to some extent depends on the learning rate and
initial weights (bad choices may cause learning too slow
to converge to be practical)
– How can we know the learning is ok?
• Reserve a few samples for testing
Adaline
• By Widrow and Hoff (~1960)
– Adaptive linear elements for signal processing
– The same architecture of perceptrons

– Learning method: delta rule (another way of error driven),


also called Widrow-Hoff learning rule
Try to reduce the mean squared error (MSE) between the net
input and the desired out put
Adaline
• Delta rule
– Let ij = (i0,j, i1,j,…, in,j ) be an input vector with desired output dj
– The squared error
• E  (d j  net j ) 2  (d j   wl il , j ) 2
l
• Its value determined by the weights wl
– Modify weights by gradient descent approach
• 

• Change weights in the opposite direction of E / wk


wk   (d j   wl il , j )  ik , j   (d j  net j )  ik , j
l
Adaline Learning Algorithm
Adaline Learning
• Delta rule in batch mode
– Based on mean squared error over all P samples
1 P
E  p
( d  net p ) 2
P p 1

• E is again a function of w = (w0, w1,…, wn )


• the gradient of E:
E 2 P 
  [(d p  net p ) (d p  net p )]
wk P p 1 wk
2 P
   [(d p  net p )  ik , p ]
P p 1
P
E
• Therefore wi      [(d p  net p )  ik , p ]
wi p 1
Adaline Learning
• Notes:
– Weights will be changed even if an input is classified
correctly
– E monotonically decreases until the system reaches a state
with (local) minimum E (a small change of any wi will
cause E to increase).
– At a local minimum E state, E / w i  0 i , but E is not
guaranteed to be zero (netj != dj)
• This is why Adaline uses threshold function rather than
linear function
Linear Separability Again
• Examples of linearly inseparable classes
- Logical XOR (exclusive OR) function x o

patterns (bipolar) decision boundary


x1 x2 output
-1 -1 -1 o x
-1 1 1
1 -1 1 x: class I (output = 1)
1 1 -1 o: class II (output = -1)
No line can separate these two classes, as can be seen from the
fact that the following linear inequality system has no solution

w{ 0−w1−w2<0 (1)¿{w0−w1+w2≥0 (2)¿{w0+w1−w2≥0 (3) ¿ ¿


because we have w0 < 0 from
(1) + (4), and w0 >= 0 from
(2) + (3), which is a
contradiction
Why hidden units must be non-linear?
• Multi-layer net with linear hidden layers is equivalent to a
single layer net
v11
x1 z1 w1 threshold = 0
v12
Y
v21
x2 v22 z2 w2

– Because z1 and z2 are linear unit


z1 = a1* (x1*v11 + x2*v21) + b1
z1 = a2* (x1*v12 + x2*v22) + b2
– nety = z1*w1 + z2*w2
= x1*u1 + x2*u2 + b1+b2 where
u1 = (a1*v11+ a2*v12)w1, u2 = (a1*v21 + a2*v22)*w2
nety is still a linear combination of x1 and x2.
– XOR can be solved by a more complex network with
hidden units

Threshold 
2
x1 z1 2 Threshold 
-2
-2
Y
x2 2 z2 2

(-1, -1) (-1, -1) -1


(-1, 1) (-1, 1) 1
(1, -1) (1, -1) 1
(1, 1) (-1, -1) -1
Summary
• Single layer nets have limited representation power
(linear separability problem)
• Error driven seems a good way to train a net
• Multi-layer nets (or nets with non-linear hidden units)
may overcome linear inseparability problem, learning
methods for such nets are needed
• Threshold/step output functions hinders the effort to
develop learning methods for multi-layered nets

You might also like