Introduction to Pattern Recognition

Selim Aksoy

Department of Computer Engineering

Bilkent University
[email protected]

CS 551, Fall 2018

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 1 / 39
 Human Perception

I Humans have developed highly sophisticated skills for

sensing their environment and taking actions according to
what they observe, e.g.,
I recognizing a face,
I understanding spoken words,
I reading handwriting,
I distinguishing fresh food from its smell.
I We would like to give similar capabilities to machines.

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 2 / 39
 What is Pattern Recognition?
I A pattern is an entity, vaguely defined, that could be given a
name, e.g.,
I fingerprint image,
I handwritten word,
I human face,
I speech signal,
I DNA sequence,
I ...
I Pattern recognition is the study of how machines can
I observe the environment,
I learn to distinguish patterns of interest,
I make sound and reasonable decisions about the categories
of the patterns.
CS 551, Fall 2018 2017,
c Selim Aksoy (Bilkent University) 3 / 39
 Human and Machine Perception

I We are often influenced by the knowledge of how patterns

are modeled and recognized in nature when we develop
pattern recognition algorithms.
I Research on machine perception also helps us gain deeper
understanding and appreciation for pattern recognition
systems in nature.
I Yet, we also apply many techniques that are purely
numerical and do not have any correspondence in natural
systems.

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 4 / 39
 Pattern Recognition Applications

Figure 1: English handwriting recognition.

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 5 / 39
 Pattern Recognition Applications

Figure 2: Chinese handwriting recognition.

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 6 / 39
 Pattern Recognition Applications

Figure 3: Fingerprint recognition.

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 7 / 39
 Pattern Recognition Applications

Figure 4: Biometric recognition.

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 8 / 39
 Pattern Recognition Applications

Figure 5: Autonomous navigation.

CS 551, Fall 2018 2017,
c Selim Aksoy (Bilkent University) 9 / 39
 Pattern Recognition Applications

Figure 6: Cancer detection and grading using microscopic tissue data.

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 10 / 39
 Pattern Recognition Applications

Figure 7: Cancer detection and grading using microscopic tissue data. (left)
A whole slide image with 75568 × 74896 pixels. (right) A region of interest with
7440 × 8260 pixels.

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 11 / 39
 Pattern Recognition Applications

Figure 8: Land cover classification using satellite data.

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 12 / 39
 Pattern Recognition Applications

Figure 9: Building and building group recognition using satellite data.

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 13 / 39
 Pattern Recognition Applications

Figure 10: License plate recognition: US license plates.

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 14 / 39
 Pattern Recognition Applications

Figure 11: Clustering of microarray data.

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 15 / 39
 An Example

I Problem: Sorting incoming

fish on a conveyor belt
according to species.
I Assume that we have only
two kinds of fish:
I sea bass,
I salmon.

Figure 12: Picture taken from a

camera.

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 16 / 39
 An Example: Decision Process

I What kind of information can distinguish one species from

the other?
I length, width, weight, number and shape of fins, tail shape,
etc.
I What can cause problems during sensing?
I lighting conditions, position of fish on the conveyor belt,
camera noise, etc.
I What are the steps in the process?
I capture image → isolate fish → take measurements → make
decision

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 17 / 39
 An Example: Selecting Features

I Assume a fisherman told us that a sea bass is generally

longer than a salmon.
I We can use length as a feature and decide between sea
bass and salmon according to a threshold on length.
I How can we choose this threshold?

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 18 / 39
 An Example: Selecting Features

Figure 13: Histograms of the length feature for two types of fish in training
samples. How can we choose the threshold l∗ to make a reliable decision?

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 19 / 39
 An Example: Selecting Features

I Even though sea bass is longer than salmon on the

average, there are many examples of fish where this
observation does not hold.
I Try another feature: average lightness of the fish scales.

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 20 / 39
 An Example: Selecting Features

Figure 14: Histograms of the lightness feature for two types of fish in training
samples. It looks easier to choose the threshold x∗ but we still cannot make a
perfect decision.

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 21 / 39
 An Example: Cost of Error

I We should also consider costs of different errors we make

in our decisions.
I For example, if the fish packing company knows that:
I Customers who buy salmon will object vigorously if they see
sea bass in their cans.
I Customers who buy sea bass will not be unhappy if they
occasionally see some expensive salmon in their cans.
I How does this knowledge affect our decision?

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 22 / 39
 An Example: Multiple Features

I Assume we also observed that sea bass are typically wider

than salmon.
I We can use two features in our decision:
I lightness: x1
I width: x2
I Each fish image is now represented as a point (feature
vector ) !
x1
x=
x2
in a two-dimensional feature space.

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 23 / 39
 An Example: Multiple Features

Figure 15: Scatter plot of lightness and width features for training samples.
We can draw a decision boundary to divide the feature space into two
regions. Does it look better than using only lightness?

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 24 / 39
 An Example: Multiple Features

I Does adding more features always improve the results?

I Avoid unreliable features.
I Be careful about correlations with existing features.
I Be careful about measurement costs.
I Be careful about noise in the measurements.
I Is there some curse for working in very high dimensions?

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 25 / 39
 An Example: Decision Boundaries
I Can we do better with another decision rule?
I More complex models result in more complex boundaries.

Figure 16: We may distinguish training samples perfectly but how can
we predict how well we can generalize to unknown samples?

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 26 / 39
 An Example: Decision Boundaries
I How can we manage the tradeoff between complexity of
decision rules and their performance to unknown samples?

Figure 17: Different criteria lead to different decision boundaries.

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 27 / 39
 More on Complexity

1

−1

0 1

Figure 18: Regression example: plot of 10 sample points for the input
variable x along with the corresponding target variable t. Green curve is the
true function that generated the data.

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 28 / 39
 More on Complexity
 

1

1

0 0

−1 −1

0 1 0 1

(a) 0’th order polynomial (b) 1’st order polynomial

 

1

1

0 0

−1 −1

0 1 0 1

(c) 3’rd order polynomial (d) 9’th order polynomial

Figure 19: Polynomial curve fitting: plots of polynomials having various
orders, shown as red curves, fitted to the set of 10 sample points.

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 29 / 39
 More on Complexity

 

1

1

0 0

−1 −1

0 1 0 1

(a) 15 sample points (b) 100 sample points

Figure 20: Polynomial curve fitting: plots of 9’th order polynomials fitted to
15 and 100 sample points.

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 30 / 39
 Pattern Recognition Systems
Physical environment

Data acquisition/sensing Training data

Pre−processing Pre−processing

Feature extraction Feature extraction/selection

Features Features

Classification Model Model learning/estimation

Post−processing

Decision

Figure 21: Object/process diagram of a pattern recognition system.

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 31 / 39
 Pattern Recognition Systems

I Data acquisition and sensing:

I Measurements of physical variables.
I Important issues: bandwidth, resolution, sensitivity,
distortion, SNR, latency, etc.
I Pre-processing:
I Removal of noise in data.
I Isolation of patterns of interest from the background.
I Feature extraction:
I Finding a new representation in terms of features.

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 32 / 39
 Pattern Recognition Systems

I Model learning and estimation:

I Learning a mapping between features and pattern groups
and categories.
I Classification:
I Using features and learned models to assign a pattern to a
category.
I Post-processing:
I Evaluation of confidence in decisions.
I Exploitation of context to improve performance.
I Combination of experts.

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 33 / 39
 The Design Cycle

Collect Select Select Train Evaluate

data features model classifier classifier

Figure 22: The design cycle.

I Data collection:
I Collecting training and testing data.
I How can we know when we have adequately large and
representative set of samples?

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 34 / 39
 The Design Cycle

I Feature selection:
I Domain dependence and prior information.
I Computational cost and feasibility.
I Discriminative features.
I Similar values for similar patterns.
I Different values for different patterns.
I Invariant features with respect to translation, rotation and
scale.
I Robust features with respect to occlusion, distortion,
deformation, and variations in environment.

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 35 / 39
 The Design Cycle

I Model selection:
I Domain dependence and prior information.
I Definition of design criteria.
I Parametric vs. non-parametric models.
I Handling of missing features.
I Computational complexity.
I Types of models: templates, decision-theoretic or statistical,
syntactic or structural, neural, and hybrid.
I How can we know how close we are to the true model
underlying the patterns?

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 36 / 39
 The Design Cycle

I Training:
I How can we learn the rule from data?
I Supervised learning: a teacher provides a category label or
cost for each pattern in the training set.
I Unsupervised learning: the system forms clusters or natural
groupings of the input patterns.
I Reinforcement learning: no desired category is given but the
teacher provides feedback to the system such as the
decision is right or wrong.

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 37 / 39
 The Design Cycle

I Evaluation:
I How can we estimate the performance with training
samples?
I How can we predict the performance with future data?
I Problems of overfitting and generalization.

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 38 / 39
 Summary

I Pattern recognition techniques find applications in many

areas: machine learning, statistics, mathematics, computer
science, biology, etc.
I There are many sub-problems in the design process.
I Many of these problems can indeed be solved.
I More complex learning, searching and optimization
algorithms are developed with advances in computer
technology.
I There remain many fascinating unsolved problems.

CS 551, Fall 2018 2017,

c Selim Aksoy (Bilkent University) 39 / 39