Lecture on Pattern

Classification and Pattern

Association.
Pattern Classification
•Classification is the process of
idenifying the class to which a given
pattern belongs
Pattern Classification

• For example, let us considered the set of S of all three bit patterns .We
may divide the patterns of S into two classes A and B where A is the
class of all patterns having more 0’s than 1’s and B the
converse.Therefore,

• S={000, 001, 010, 011, 100, 101, 110, 111}

• A={000, 001, 010, 100}
• B={011, 101, 110, 111}
Pattern Classification
• Now given an arbitary 3-bit pattern, the classification
problem here is to decide whether it belongs to the
class A , or class B.

• In other words, we habe to establish the mapping

shown in fig. below.
Pattern Classification

000
001 A
010
011
100
101 B
110
111

Fig:Classification of 3-bit patterns based on the number of 0s and 1s.

 Pattern Classification

• The simplest way to achieve Row Pattern Class

this is to execute a table 0 0 0 0 A
look-up procedure, as shown 1 0 0 1 A
in table. 2 0 1 0 A
3 0 1 1 B
Table:Classification as 4 1 0 0 A
a table look-up 5 1 0 1 B
procedure. 6 1 1 0 B
7 1 1 1 B
 A procedure for classification of 3-Bit
Pattern
Procedure Classify (x, A, B) End-if
Begin End-for
n0=n1=0; /* initialize counts*/ If n0>n1
/* count 0s and 1s in x*/ Then Return A;
For i← 1 to 3 do Else Return B;
If the ith bit is 0 End if
Then n0++ ; End -procedure
Else n1++ ;
Fig.:An artificial neuron to classify 3- bit binary
patterns based on the number of 0s and 1s.

x1 X1 +1
.
.
Y
X2
x2 +1 y_out

X3
x3 +1
 Pattern Association
3
y_in = xw
i 1
i i =x1w1+x2w2+x3w3

=x1+x2+x3
1 if y_in≥2
y_out=f(y_in)={
0 otherwise
Classification of 3-bit Pattern with two output units
3

y_in1 ==  i 1
xi wi1

=x1w11+x2w21+x3w31=x1+x2+x3
x1 X1 W11=+1 1 if y_in1≥2
y_out1 =f(y_in1)={
-1 Y1 y_out1
0 , otherwise
X2 +1 3

x2 -1 xw
i 1
i i

Y2
y_in2=
+1 y_out2 =x1w12+x2w22+x3w32=-x1-x2-x3
X3
1, if y_in2≥-1
x3 W32=-1 y_out2= f(y_in2) ={
0, otherwise
 Pattern Association
Given an input pattern, and a set of patterns already stored in the
memory , finding the closest match of the input pattern among the
stored patterns and returning it as the output, is known as pattern
association.
 Pattern Association
The basic concept of pattern association is explained below with the
help of a simple illustrative example.The example is inspired by
Hopfield network[1982].
P1
+3 +3 Inactive Unit
P2 +3 P3 Active Unit
-1 -1 -1 -1
P4 +3 P5
+3 +3
P6
The essintial features of Network
The essential features of the network are describe below.
i)PE states :at any instant, a unit may either be in an active or an
inactive state.Moreover, depending on the circumstances , the state
of a unity may change from active to inactive and vice versa.In fig.
An active unit is shown with a black circle and an inactive unit is
indicated by a hollow circle.
ii)Interconnections: All interconnections are bidirectional
magnitude of the weight associated with an interconnection give
the strenght of influence the connected units play on each other.
The essintial features of Network
iii)Signed weights: A negative weight implies that the corresponding units tend to
inhibit, or activate, each other.
iv)Initialization : The network is initialized by making certain units active and keeping
others inactive.The initial combination of active and inactive units is considered as the
input pattern.After initialization, the network passes through a number of
transformations.The transformation take place according to the rules described below.
v)Transformations:At each stage during the sequence of transformations,the next
state of the every unit pi, i= 1,…..,6, is determined. The next state of the every unit pi
is obtained by considering all active neighbours of pi and taking the algebraic sum of
the weights of the paths between pi and the neighbouring actives units.If the sum is
greater than 0, then pi becomes active for the next phase.Otherwise it becomes
inactive. The state of a unit without any active unit in its neighbourhood remains
unaltered.This process is known as parallel relaxation.
 Example with Hopfield Network
• For example, let the network be initialized with the pattern shown in
Fig.
• Initially, all units except p2 and p5 are inactive.
• To find the state of P1 in the next instant, we look for the active
neighbours of p1 and find that p2 is the only active unit connected to p1
through an interconnection link of weight+3.
• Hence p1 becomes active in the next instant.Similarly , for p3,both p2
and p5 are active units in its neighbourhood.
• The sum of the corresponding weights is w23 +w35 =+3-1=+2.Hence p3
also becomes active.
 Example with Hopfield Network
Fig.: Pattern association through parallel relaxation

P1 P1 P1
+3 +3 +3 +3 +3 +3
P2 +3 P3 P2 +3 P3 P2 +3 P3
-1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1
P4 +3 P5 P4 +3 P5P4 +3 P5
+3 +3 +3 +3 +3 +3
P6 P6 P6
(a) (b) (c)
 Example with Hopfield Network

• However, p2 itself becomes inactive because the only active unit in its
vicinity, p5 is connected to it through a negatively link.
• Table shows the details of computations for transformation of the
network from Fig.(a) and (b).
• The configuration of fig.(b) is not stable.
• The network further transforms itself from above fig.(b) to fig.©, which
is a stable state.
• Therefore , we can say that the given network associates the pattern
shown in Fig.(a) to that shown in Fig.©.
Table: Computation of parallel relaxation
on fig.(a)
Unit Present State Active Neighbouring Sum Next State
unit(s)
P1 Inactive P2 +3 Active
P2 Active P5 -1 Inactive
P3 Inactive P2,P5 +3-1=+2 Active

P4 Inactive P2,P5 -1+3=+2 Active

P5 Active P2 -1 Inactive

P6 Inactive P5 +3 Active
 Example with Hopfield Network
Fig.: Non trivial patterns stored in a Hopefield
Network
P1 P1 P1
+3 +3 +3 +3 +3 +3
P2 +3 P 3 P2 +3 P3 P2 +3 P3
-1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1

P4 +3 P5 P4 +3 P5 P4 +3 P5
+3 +3 +3 +3 +3 +3

P6 P6 P6
(a) (b) (c )
 Example with Hopfield Network
• A little investigation reveals that the given network has three non-
trivial stable states as shown in above fig. (a) to (c).
• The trivial state is that where all units are inactive.
• It can be easily verified that if one or more of the units P1,P2, P3 is/ are
active initially while the rest P4, P5, P6 are inactive, the network
converges to the pattern shown in above Fig.(a) .
• Similarly the pattern of fig.(b) is associated with any input pattern
where at one unit of the group {P4, P5,P6} is /are active.Finally, an
input pattern having active units from both the groups {P1,P2,P3} and
{P4, P5,P6}would associate to with the pattern depicted in fig. (c ).
• Hence the given network may be thought of as storing three non-
trivial patterns as discussed above.Such networks are also referred to
as associative memories, or content addressable memories.
 Activation Function
• The output from a processing unit is termed as its activation .
• Activation of a processing unit is a function of the net input to the
processing unit.
• The function that maps the net input value to the output signal value,
i.e. the activation , is known as the activation function of the unit.
• Some common activation functions are presented below.
 Identity Function
• The simplest activation function
is the identity function that y
passes on the incoming signal as g(x)=x
the outgoing signal without any a
change.
• Therefore, the identity activation a
function g(x) is defined as x
g(x)=x

Fig. : The identity activation function

 Step Function
• Another frequently used activation function is the step function.
• The basic step function produces a 1 or 0 depending on whether the
net input is greater than 0 or therwise.
• Mathematically the step function is defined as follows.
1, if x>0
g(x)={
0, otherwise
• The step function is also known as the heaviside function.
Graphical Representation of Basic Step
function

1 g(x)
x
Threshold activation function
• In Basic step function , occasionally, instead of 0 a non-zero threshold
value θ is used. This is known as the threshold step function
and is defined as

1, if x>θ
g(x)={
0, otherwise
• The shape of the threshold function is shown in fig. below.
 Threshold activation function
y

1 g(x)
x
θ

Fig. Threshold step function.

Bipolar Step Function
• The step function discussed so far are binary step functions since
they always evaluates to 0 or 1.
• Occasionally, it is more convenient to work with bipolar data,-1 and
+1, then the binary data.
• If a signal of value 0 is sent through a weighted path, the information
contained in the interconnection weight is lost as it is multiplied by 0.
• To overcome this problem , the binary input is converted to bipolar
form and then a suitable bipolar activation function is employed.
Bipolar Step Function

• Accordingly , binary step functions have their bipolar versions.

• The output of a bipolar step function is -1 or +1, not 0, or 1.
• The bipolar step function and threshold function are shown in fig. below.
• They are deined as follows.

g(x)={ +1, if x>θ

g(x)={ +1, if x>0 -1, otherwise
-1, otherwise
a) Bipolar Step function b) Bipolar threshold function
Bipolar Step Function

y
y

+1
+1 g(x) g(x)

θ
x

-1 -1

Fig.Bipolar sep function Fig. Bipolar Threshold Function

 The Sigmoid function
• As the step function is not differentiable .
• Some ANN training algorithm requires that the activation function be
continuous and differentiable.
• The step function is not suitable for such cases.
• Sigmoid functions have the nice property that they can apporoximate the
step function to the desired extent without losing its differentiablity.
Binary sigmoid, also referred to as the logistic sigmoid is defined by
equations.
1
g(x)= 1  e   x
 Binary Sigmoid Function
• The parameter δ in above equation is known as the steepness
parameter.
• The shape of the sigmoid function is shown in below fig.

1 δ1
0.5 δ2
δ1<δ2

0
Fig:Binary Sigmoid Function
 Binary Sigmoid Function

• The transition from 0 to 1 could be made as steep as desired by

increasing the value of δ to appropriate extent.
• The first derivative of g(x), denoted by g'(x) is expressed as
g'(x)= δ g(x)(1-g(x))
 Bipolar Sigmoid Function
• Depending on the requirement, the binary sigmoid function can be
scaled to any range of values appropriate for a given application.
• The most widely used range is from -1 to +1, and the corresponding
sigmoid function is referred to as the bipolar sigmoid function. The
formulae for the bipolar sigmoid function
 and its first derivative are given
below .
  x
1  e
g(x)= 1  e   x

g'(x)= δ/2(1+g(x))(1-g(x))
 Bipolar Sigmoid Function
• Following Fig. presents its form graphically.

-1

Fig.Bipolar Sigmoid function

 Hyperbolic Tangent Function
• Another bipolar activation function that is widely employed in ANN
applications is the hyperbolic tangent function.
• The function , as well as its first derivative, are expressed by
following equations.
 x
e x
 e
h(x)=  x
e x
 e

h'(x)=(1+h(x))(1-h(x))
 Hyperbolic Tangent Function

• The hyperbolic tangent function is closely related to the bipolar

sigmoid function.
• When the input data is binary and not continuously valued in the range
from 0 to 1, they are generally converted to bipolar form and then a
bipolar sigmoid or hyperbolic tangent activation function is applied on
them by the processing units.
ANY QUESTION?
THANK YOU