Pattern Recognition

P.S.Sastry
[email protected]

Dept. Electrical Engineering

Indian Institute of Science, Bangalore

PR video course p.1/90

Reference Books

R.O.Duda, P.E.Hart and D.G.Stork, Pattern

Classification, Johy Wiley, 2002

C.M.Bishop, Pattern Recognition and Machine

Learning, Springer, 2006.

C. M. Bishop, Neural Networks for Pattern

Recognition, Oxford University Press, Indian Edition,
2003.
R.O.Duda and P.E. Hart, Pattern Classification and
Scene Analysis, Wiley, 1973

PR video course p.2/90

Pattern Recognition
A basic attribute of people categorisation of sensory
input
Pattern PR System Class label
Examples of Pattern Recognition tasks

Reading facial expressions

Recognising Speech

Reading a Document

Identifying a person by fingerprints

Diagnosis from medical images

PR video course p.3/90

Machine Recognition of Patterns

X

pattern feature extractor classifier class

label

Feature extractor makes some measurements on the

input pattern.

X is called Feature Vector. Often, X n .

Classifier maps each feature vector to a class label.

Features to be used are problem-specific.

PR video course p.4/90

Some Examples of PR Tasks

Character Recognition
Pattern Image.
Class identity of character
Features: Binary image, projections (e.g., row and
column sums), Moments etc.

PR video course p.5/90

Some Examples of PR Tasks

Speech Recognition
Pattern 1-D signal (or its sampled version)
Class identity of speech units
Features LPC model of chunks of speech,
spectral info, cepstrum etc.
Pattern can become a sequence of feature vectors.

PR video course p.6/90

Examples contd...

Fingerprint based identity verification

Pattern image plus a identity claim
Class Yes / No
Features: Position of Minutiae, Orientation field of
ridge lines etc.

PR video course p.7/90

Examples contd...

Video-based Surveillance
Pattern video sequence
Class e.g., level of alertness
Features Motion trajectories, Parameters of a
prefixed model etc.

PR video course p.8/90

Examples contd...

Credit Screening
Pattern Details of an applicant (for, e.g., credit
card)
Class Yes / No
Features: income, job history, level of credit, credit
history etc.

PR video course p.9/90

Examples contd...

Imposter detection (of, e.g., credit card)

Pattern A sequence of transactions
Class Yes / No
Features: Amount of money, locations of
transactions, times between transactions etc.

PR video course p.10/90

Examples contd...

Document Classification
Pattern A document and a query
Class Relevant or not (in general, rank)
Features word occurrence counts, word context
etc.
Spam filtering, diagnostics of machinery etc.

PR video course p.11/90

Design of Pattern Recognition Systems

Features depend on the problem. Measure relevant

quantities.

PR video course p.12/90

Design of Pattern Recognition Systems

Features depend on the problem. Measure relevant

quantities.

Some techniques available to extract more relevant

quantities from the initial measurements. (e.g., PCA)

PR video course p.13/90

Design of Pattern Recognition Systems

Features depend on the problem. Measure relevant

quantities.

Some techniques available to extract more relevant

quantities from the initial measurements. (e.g., PCA)

After feature extraction each pattern is a vector

Classifier is a function to map such vectors into class

labels.

PR video course p.14/90

Design of Pattern Recognition Systems

Features depend on the problem. Measure relevant

quantities.

Some techniques available to extract more relevant

quantities from the initial measurements. (e.g., PCA)

After feature extraction each pattern is a vector

Classifier is a function to map such vectors into class

labels.
Many general techniques of classifier design are
available.
Need to test and validate the final system.

PR video course p.15/90

Some notation

Feature Space, X Set of all possible feature vectors.

PR video course p.16/90

Some notation

Feature Space, X Set of all possible feature vectors.

Classifier: a decision rule or a function,

h : X {1, , M }.

Often, X = n . Convenient to take M = 2.

Then we take the labels as {0, 1} or {1, 1}.

PR video course p.17/90

Some notation

Feature Space, X Set of all possible feature vectors.

Classifier: a decision rule or a function,

h : X {1, , M }.

Often, X = n . Convenient to take M = 2.

Then we take the labels as {0, 1} or {1, 1}.

Then, any binary valued function on X is a classifier.

What h to choose? We want correct or optimal
classifier.
How to design classifiers?
How to judge performance?
How to provide performance guarantees?
PR video course p.18/90

We first consider the 2-class problem.

PR video course p.19/90

We first consider the 2-class problem.

Can handle the M > 2 case if we know how to handle

2-class problem.

PR video course p.20/90

We first consider the 2-class problem.

Can handle the M > 2 case if we know how to handle

2-class problem.

Simplest alternative: design M 2-class classifiers.

One Vs Rest

PR video course p.21/90

We first consider the 2-class problem.

Can handle the M > 2 case if we know how to handle

2-class problem.

Simplest alternative: design M 2-class classifiers.

One Vs Rest
There are other possibilities: e.g., Tree structured
classifiers.

PR video course p.22/90

We first consider the 2-class problem.

Can handle the M > 2 case if we know how to handle

2-class problem.

Simplest alternative: design M 2-class classifiers.

One Vs Rest
There are other possibilities: e.g., Tree structured
classifiers.
The 2-class problem is the basic problem.

PR video course p.23/90

We first consider the 2-class problem.

Can handle the M > 2 case if we know how to handle

2-class problem.

Simplest alternative: design M 2-class classifiers.

One Vs Rest
There are other possibilities: e.g., Tree structured
classifiers.
The 2-class problem is the basic problem.

We will also look at M-class classifiesrs.

PR video course p.24/90

A simple PR problem

: Problem: Spot the Right Candidate

: Features:
x1 : Marks based on academic record
x2 : Marks in the interview

PR video course p.25/90

A simple PR problem

: Problem: Spot the Right Candidate

: Features:
x1 : Marks based on academic record
x2 : Marks in the interview

A Classifier: ax1 + bx2 > c Good

PR video course p.26/90

A simple PR problem

: Problem: Spot the Right Candidate

: Features:
x1 : Marks based on academic record
x2 : Marks in the interview

A Classifier: ax1 + bx2 > c Good

Another Classifier: x1 x2 > c Good
(or (x1 + a)(x2 + b) > c).

PR video course p.27/90

A simple PR problem

: Problem: Spot the Right Candidate

: Features:
x1 : Marks based on academic record
x2 : Marks in the interview

A Classifier: ax1 + bx2 > c Good

Another Classifier: x1 x2 > c Good
(or (x1 + a)(x2 + b) > c).

Design of classifier:
We have to choose a specific form for the classifier.
What values to use for parameters such as a, b, c?
PR video course p.28/90

Designing Classifiers

Need to decide how feature vector values determine

the class.
(How different marks reflect goodness of candidate)

PR video course p.29/90

Designing Classifiers

Need to decide how feature vector values determine

the class.
(How different marks reflect goodness of candidate)

In most applications, not possible to design classifier

from physics of the problem.

PR video course p.30/90

Designing Classifiers

Need to decide how feature vector values determine

the class.
(How different marks reflect goodness of candidate)

In most applications, not possible to design classifier

from physics of the problem.

The difficulties are

Lot of variability in patterns of a single class
Variability in feature vector values
Feature vectors of patterns from different classes
can be arbitrarily close.
Noise in measurements
PR video course p.31/90

Designing Classifiers contd...

Often the only information available for the design is

A training set of example patterns.

Training set: {(Xi , yi ), i = 1, . . . , }.

Here Xi is an example feature vector of class yi .

PR video course p.32/90

Designing Classifiers contd...

Often the only information available for the design is

A training set of example patterns.

Training set: {(Xi , yi ), i = 1, . . . , }.

Here Xi is an example feature vector of class yi .

Generation of training set Take representative

patterns of known category (data collection) and
obtain the feature vectors. (choice of feature
measurements).

Now learn an appropriate function h as classifier.

(Model choice)

Test and validate the classifier on more data.

PR video course p.33/90

A simple PR problem

: Problem: Spot the Right Candidate

: Features:
x1 : Marks based on academic record
x2 : Marks in the interview

A Classifier: ax1 + bx2 > c Good

We have chosen a specific form for the classifier.

Design of classifier: What values to use for a, b, c?

Information available: experience history of past

candidates

PR video course p.34/90

Training Set

PR video course p.35/90

Another example problem

Problem: recognize persons of medium build

Features: Height and Weight

The classifier is nonlinear here

PR video course p.36/90

Learning from Examples

Designing a classifier is a typical problem of learning

from examples. (Also called learning with a teacher).

Nature of feedback from teacher determines difficulty

of the learning problem.
PR video course p.37/90

In the context of Pattern Recognition

f eature

V ector Classifier

Class

f eedback

Teacher

Supervised learning Teacher gives the true class

label for each feature vector
Reinforcement Learning noisy assessment of
performance, e.g., correct/incorrect
Unsupervised learning no teacher input (Clustering
problems)
PR video course p.38/90

When the class labels of training patterns as given by

teacher are noisy, we consider it as supervised
learning with label noise or classification noise.

Many classifier design algorithms do supervised

learning.

PR video course p.39/90

Function Learning

Closely related problem. Output is continuous-valued

rather than discrete as in classifiers.
Here training set examples could be
{(Xi , yi ), i = 1, . . . , }, Xi X , yi .

PR video course p.40/90

Function Learning

Closely related problem. Output is continuous-valued

rather than discrete as in classifiers.
Here training set examples could be
{(Xi , yi ), i = 1, . . . , }, Xi X , yi .

The prediction variable, y , is continuous; rather than

taking finitely many values. ( There can be noise in
examples).

Similar learning techniques needed to infer the

underlying functional relationship between X and y .
(Regression function of y on X ).
PR video course p.41/90

Examples of Function Learning

Time series prediction: Given a series x1 , x2 , , find

a function to predict xn .

PR video course p.42/90

Examples of Function Learning

Time series prediction: Given a series x1 , x2 , , find

a function to predict xn .

Based on past values: Find a best function

xn = h(xn1 , xn2 , , xnp )

Predict stock prices, exchange rates etc.

Linear prediction model used in speech analysis

PR video course p.43/90

Examples of Function Learning

Time series prediction: Given a series x1 , x2 , , find

a function to predict xn .

Based on past values: Find a best function

xn = h(xn1 , xn2 , , xnp )

Predict stock prices, exchange rates etc.

Linear prediction model used in speech analysis

More general predictors can use other variables also.

Predict rainfall based on measurements and
(possibly) previous years data.
In general, System Identification. (An application:
smart sensors)
PR video course p.44/90

Examples contd... : Equaliser

Tx x(k) channel Z(k) filter
y(k) Rx
We want y(k) = x(k). Design (or adapt) the filter to
achieve this
We can choose a filter as T
y(k) =

ai Z(k i)

i=1

Find best ai a function learning problem.

PR video course p.45/90

Examples contd... : Equaliser

Tx x(k) channel Z(k) filter
y(k) Rx
We want y(k) = x(k). Design (or adapt) the filter to
achieve this
We can choose a filter as T
y(k) =

ai Z(k i)

i=1

Find best ai a function learning problem.

Training set: {(x(k), Z(k)), k = 1, 2, , N }
How do we know x(k) at the receiver end?
Prior agreements (Protocols)

PR video course p.46/90

Learning from examples

Learning from examples is basic to both classification

and regression.

Training set: {(X1 , y1 ), , (X , y )}.

We essentially want to fit a best function: y = f (X).

PR video course p.47/90

Learning from examples

Learning from examples is basic to both classification

and regression.

Training set: {(X1 , y1 ), , (X , y )}.

We essentially want to fit a best function: y = f (X).

Suppose X . Then this is a familiar curve-fitting

problem.

PR video course p.48/90

Learning from examples

Learning from examples is basic to both classification

and regression.

Training set: {(X1 , y1 ), , (X , y )}.

We essentially want to fit a best function: y = f (X).

Suppose X . Then this is a familiar curve-fitting

problem.

Model choice (choice of form for f ) is very important

here.
If we choose f to be a polynomial, higher the degree
better would be the performance on training data.

But that is not the real performance A polynomial of

degree would give zero error!!
PR video course p.49/90

Learning from examples Generalization

To obtain a classifier (or a regression function) we use

the training set.

We know the class label of patterns (or the values for

prediction variable) in the training set.

PR video course p.50/90

Learning from examples Generalization

To obtain a classifier (or a regression function) we use

the training set.

We know the class label of patterns (or the values for

prediction variable) in the training set.

Errors on the training set do not necessarily tell how

good is the classifier.

PR video course p.51/90

Learning from examples Generalization

To obtain a classifier (or a regression function) we use

the training set.

We know the class label of patterns (or the values for

prediction variable) in the training set.

Errors on the training set do not necessarily tell how

good is the classifier.

Any classifier that amounts to only storing the training

set is useless.
Interested in the generalization abilities how does
our classifier perform on unseen or new patterns.

PR video course p.52/90

Design of Classifiers

The classifier should perform well inspite of inherent

variability of patterns and noise in feature extraction
and/or in class labels as given in training set.

PR video course p.53/90

Design of Classifiers

The classifier should perform well inspite of inherent

variability of patterns and noise in feature extraction
and/or in class labels as given in training set.

Statistical Pattern Recognition An approach where

the variabilities are captured through probabilistic
models.

PR video course p.54/90

Design of Classifiers

The classifier should perform well inspite of inherent

variability of patterns and noise in feature extraction
and/or in class labels as given in training set.

Statistical Pattern Recognition An approach where

the variabilities are captured through probabilistic
models.
There are other approaches, e.g., syntactic pattern
recognition, fuzzy-set based methods etc.

PR video course p.55/90

Design of Classifiers

The classifier should perform well inspite of inherent

variability of patterns and noise in feature extraction
and/or in class labels as given in training set.

Statistical Pattern Recognition An approach where

the variabilities are captured through probabilistic
models.
There are other approaches, e.g., syntactic pattern
recognition, fuzzy-set based methods etc.

In this course we consider classification and

regression (function learning) problems in the
statistical framework.
PR video course p.56/90

Statistical Pattern Recognition

X is the feature space. (We take X = n ). We

Consider a 2-class problem for simplicity of notation.

A given feature vector can come from different

classes with different probabilities.

PR video course p.57/90

Statistical Pattern Recognition

X is the feature space. (We take X = n ). We

Consider a 2-class problem for simplicity of notation.

A given feature vector can come from different

classes with different probabilities.

Let: fi be the probability density function of the

feature vectors from class-i, i = 0, 1.

fi are called class conditional densities.

PR video course p.58/90

Statistical Pattern Recognition

X is the feature space. (We take X = n ). We

Consider a 2-class problem for simplicity of notation.

A given feature vector can come from different

classes with different probabilities.

Let: fi be the probability density function of the

feature vectors from class-i, i = 0, 1.

fi are called class conditional densities.

Let X = (X1 , , Xn ) n represent the feature

vector.
Then fi is the (conditional) joint density of the random
variables X1 , , Xn given that X is from class-i.

PR video course p.59/90

Class conditional densities model the variability in the

feature values.
For example, the two classes can be uniformly
distributed in the two regions as shown. (The two
classes are separable here).

Class 1
Class 2

PR video course p.60/90

When class regions are separable, an important

special case is linear separability.

Class 1
Class 2

Class 1

Class 2

The classes in the left panel above are linearly

separable (can be separated by a line) while those in
the right panel are not linearly separable (though
separable).
PR video course p.61/90

In general, the two class conditional densities can

overlap. (The same value of feature vector can be
from different classes with different probabilities)

Class 1

Class 2

PR video course p.62/90

The statistical viewpoint gives us one way of looking

for optimal classifier.

We can say we want a classifier that has least

probability of misclassifying a random pattern (drawn
from the underlying distributions).

PR video course p.63/90

Let

qi (X) = Prob[class = i|X], i = 0, 1.

qi is called posterior probability (function) for class-i.

PR video course p.64/90

Let

qi (X) = Prob[class = i|X], i = 0, 1.

qi is called posterior probability (function) for class-i.

Consider the classifier

h(X) = 0 if q0 (X) > q1 (X)

= 1 otherwise

PR video course p.65/90

Let

qi (X) = Prob[class = i|X], i = 0, 1.

qi is called posterior probability (function) for class-i.

Consider the classifier

h(X) = 0 if q0 (X) > q1 (X)

= 1 otherwise

q0 (X) > q1 (X) would imply that the feature vector X

is more likely to come from class-0 rather than
class-1.
PR video course p.66/90

Let

qi (X) = Prob[class = i|X], i = 0, 1.

qi is called posterior probability (function) for class-i.

Consider the classifier

h(X) = 0 if q0 (X) > q1 (X)

= 1 otherwise

q0 (X) > q1 (X) would imply that the feature vector X

is more likely to come from class-0 rather than
class-1.
Hence, intuitively, such a classifier should minimize
probability of error in classification.
PR video course p.67/90

Statistical Pattern Recognition

X (= n ) is the feature space. Y = {0, 1} is the set of

class labels

PR video course p.68/90

Statistical Pattern Recognition

X (= n ) is the feature space. Y = {0, 1} is the set of

class labels

H = {h | h : X Y} is the set of classifiers of

interest.

PR video course p.69/90

Statistical Pattern Recognition

X (= n ) is the feature space. Y = {0, 1} is the set of

class labels

H = {h | h : X Y} is the set of classifiers of

interest.
For a feature vector X , let y(X) denote the class
label of X . In general, y(X) would be random.

PR video course p.70/90

Statistical Pattern Recognition

X (= n ) is the feature space. Y = {0, 1} is the set of

class labels

H = {h | h : X Y} is the set of classifiers of

interest.
For a feature vector X , let y(X) denote the class
label of X . In general, y(X) would be random.

Now we want to assign a figure of merit to each

possible classifier in H.

PR video course p.71/90

For example, we can rate different classifiers by

F (h) = Prob[h(X) 6= y(X)]

F (h) is the probability that h misclassifies a random

X.

PR video course p.72/90

For example, we can rate different classifiers by

F (h) = Prob[h(X) 6= y(X)]

F (h) is the probability that h misclassifies a random

X.

Optimal classifier would be one with lowest value of

F.

PR video course p.73/90

For example, we can rate different classifiers by

F (h) = Prob[h(X) 6= y(X)]

F (h) is the probability that h misclassifies a random

X.

Optimal classifier would be one with lowest value of

F.
Given a h we can calculate F (h) only if we know the
probability distributions of classes.

PR video course p.74/90

For example, we can rate different classifiers by

F (h) = Prob[h(X) 6= y(X)]

F (h) is the probability that h misclassifies a random

X.

Optimal classifier would be one with lowest value of

F.
Given a h we can calculate F (h) only if we know the
probability distributions of classes.

Minimizing F is not a straight-forward optimization

problem.
PR video course p.75/90

For example, we can rate different classifiers by

F (h) = Prob[h(X) 6= y(X)]

F (h) is the probability that h misclassifies a random

X.

Optimal classifier would be one with lowest value of

F.
Given a h we can calculate F (h) only if we know the
probability distributions of classes.

Minimizing F is not a straight-forward optimization

problem.

Here we are treating all errors as same. We will

PR video course p.76/90

Statistical PR contd.

Recall that fi (X) denotes the probability density

function of feature vectors of class-i. ( class
conditional densities).

PR video course p.77/90

Statistical PR contd.

Recall that fi (X) denotes the probability density

function of feature vectors of class-i. ( class
conditional densities).

Let pi = Prob[y(X) = i]. Called prior probabilities.

PR video course p.78/90

Statistical PR contd.

Recall that fi (X) denotes the probability density

function of feature vectors of class-i. ( class
conditional densities).

Let pi = Prob[y(X) = i]. Called prior probabilities.

Recall, posterior probabilities,

qi (X) = Prob[y(X) = i | X].

PR video course p.79/90

Statistical PR contd.

Recall that fi (X) denotes the probability density

function of feature vectors of class-i. ( class
conditional densities).

Let pi = Prob[y(X) = i]. Called prior probabilities.

Recall, posterior probabilities,

qi (X) = Prob[y(X) = i | X]. Now, by Bayes theorem

qi (X) = fi (X)pi / Z
where Z = f0 (X)p0 + f1 (X)p1 is the normalising
constant
PR video course p.80/90

Bayes Classifier

Consider the classifier given by

q0 (X)
>1
h(X) = 0 if
q1 (X)
= 1 otherwise

PR video course p.81/90

Bayes Classifier

Consider the classifier given by

q0 (X)
>1
h(X) = 0 if
q1 (X)
= 1 otherwise

This is called the Bayes classifier. Given our statistical

framework, this is the optimal classifier.

PR video course p.82/90

Bayes Classifier

Consider the classifier given by

q0 (X)
>1
h(X) = 0 if
q1 (X)
= 1 otherwise

This is called the Bayes classifier. Given our statistical

framework, this is the optimal classifier.

q0 (X) > q1 (X) is same as p0 f0 (X) > p1 f1 (X).

PR video course p.83/90

Bayes Classifier

Consider the classifier given by

q0 (X)
>1
h(X) = 0 if
q1 (X)
= 1 otherwise

This is called the Bayes classifier. Given our statistical

framework, this is the optimal classifier.

q0 (X) > q1 (X) is same as p0 f0 (X) > p1 f1 (X).

We will prove optimality of Bayes classifier later.

PR video course p.84/90

story so far

We consider PR as a two step process Feature

measurement/extraction and Classification
A classifier is to map feature vectors to Class labels.

The main difficulty in designing classifiers is due to

large variability in feature vector values.

The main information we have for the design is a

training set of examples.

Function learning is a closely related problem.

In both cases we need to learn from (training)

examples.
PR video course p.85/90

story so far

The statistical view is: model the variation in feature

vectors from a given class through probability models.

One objective can be: Find a classifier that has least

probability of error.

For example, Bayes Classifier is the optimal one if we

know class conditional densities.

PR video course p.86/90

Organization of the course

Overview of classifier learning strategies

Nearest Neighbour classification rule

Bayes Classifier for minimizing risk.

Some variations on this theme (e.g.

Neymann-Pearson Classifier)

Techniques for estimating class conditional densities

to implement Bayes classifier.

PR video course p.87/90

Organization of the course

Linear discriminant functions

Learning linear discriminant functions (Perceptron,
LMS, logistic regression)

Linear least-squares regression

Simple overview of statistical learning theory

Empirical risk minimization and VC theory

PR video course p.88/90

Organization of the course

Learning nonlinear classifiers

Neural Networks (Backpropagation, RBF networks)

Support Vector Machines

Kernel-based methods
Feature extraction and dimensionality reduction (PCA)

Boosting and ensemble classifiers (AdaBoost)

PR video course p.89/90

Thank You!

PR video course p.90/90