01 Introduction

Shahid Bahonar university of Kerman
Introduction to Pattern Recognition
Hadis Mohseni
Computer Engineering Department
Shahid Bahonar University of Kerman
September 17, 2019
Hadis Mohseni Statistical Pattern Recognition Course 1 / 33

Outline Shahid Bahonar university of Kerman
Outline
. Text Books & References
. Pattern Recognition: Definition and Applications
. Main approaches for Pattern Recognition
. Statistical Pattern Recognition
. Roadmap

Text Books & References Shahid Bahonar university of Kerman
Outline
. Roadmap

Text Books & References Shahid Bahonar university of Kerman
Text Books & References
. R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification,

John Wiley & Sons, 2000.
. C. M. Bishop, Pattern Recognition and Machine Learning, 2006.
. S. Theodoridis and K. Koutroumbas, Pattern Recognition, 4th

edition, 2008.
. Good papers from PAMI journal, PR journal, ICPR conf, ICIP

conf, and many others.
. Related sites: IAPR association, statistical PR courses, etc.

Pattern Recognition: Definition and Applications Shahid Bahonar university of Kerman
Outline
. Roadmap

What is Pattern?
. Pattern: “A pattern is the opposite of a chaos; it is an entity

vaguely defined, that could be given a name.” [Watanabe]
. Examples of pattern
A face
A handwritten character
A spoken word
A DNA sequence
A fingerprint
An internet attack
. Pattern class: A set of similar patterns

A class can be defined by its training samples

What is Pattern?
http://www.slideshare.net/armandovieira2/pattern-recognition-18230948

What is Pattern?

What is Pattern?

What is Pattern?

What is Pattern?

What is Pattern?

Pattern Recognition: Definition
X “A field whose objective is to assign an object or event to one of a

number of categories, based on features derived to emphasize
commonalities” [Larry O’Gorman, IAPR news-letter 25 (1), 2003].
. Relate a perceived pattern to

previously perceived patterns
. Given an input pattern, make a

decision about the “category” or
“class” of the pattern.
X Related fields: Machine learning, computer vision, data mining

Pattern Recognition: Importance
. “The problem of searching for patterns in data is a

fundamental one and has a long and successful history” [Bishop,
2006]
. “The real power of human thinking is based on

recognizing patterns. The better computers get at pattern
recognition, the more humanlike they will become.” [Ray Kurzweil,
NYTimes, N ov24, 2003]
. Pattern recognition is needed in designing almost all automated

systems.

Pattern Recognition: Applications
. Sensory input
. Audio
Speech recognition
Speaker identification
. Image and Video
Character, fingerprint, face, object recognition
Activity recognition, sports analysis
Human-Computer Interaction (HCI)
Medical image processing
. Other data
Information Retrieval, Document Categorization
Weather forecasting (prediction)
Data mining: Banking and insurance application
Genome Sequence Matching
Pattern Recognition: Applications

Main approaches for Pattern Recognition Shahid Bahonar university of Kerman
Outline
. Roadmap

Main approaches for Pattern Recognition
. Statistical
Describing an object in terms of a feature vector
Neural networks can also be considered as a closely related
approach to statistical pattern recognition
. Structural (Syntactic)
Describing an object by its structure (in terms of simple primitives
and their relationship)
Patterns are constructed from simpler sub-patterns in hierarchical
fashion

Structural Patten Recognition
. Assumption: pattern structure is quantifiable and extractable =⇒

structural similarity of patterns can be assessed.
Formal grammars to describe pattern structures
Parsing to recognition: The test pattern is classified according to
which grammar can produce it
Relational descriptions (graphs, strings, tree, · · ·)

Relational graph (string, tree, · · ·) matching to recognize the
category of the input pattern (e.g. correspondence mapping
between two images)

Pattern Description Using Grammars: Example
[Schalkoff, Pattern Recognition: Statistical, Structural, and Neural Approaches, 1992]

Pattern Description Using Graphs: Example
[http://www.iri.upc.edu/people/sanfeliu/Research.html]

Statistical Pattern Recognition Shahid Bahonar university of Kerman
Outline
. Roadmap

Statistical Pattern Recognition
. d-dimensional feature space

. Each pattern as a point in this space
A feature vector
. Given training patterns from each class, the goal is to find the
category of a new pattern (supervised learning)

Learning a pattern
. Supervised learning: A teacher provides a category label for

each pattern in the training set
. Unsupervised learning: The system forms clusters unlabeled

input patterns or “natural groupings” of the unlabeled input
patterns

Pattern Recognition Example
. Unsupervised
No labels is available
. Supervised
We have training data consisting of a set of pixels and their labels
We need to classify other pixels
Transductive learning: only labels the available unlabeled pixels
Inductive learning: can specify the label of new pixels

Pattern Recognition Goals
. Primary goals: Classification or Clustering
. Basic components of pattern recognition

feature extraction
classification or clustering
. Feature extraction: transformation of original data to a data set

with a reduced number of variables
The new set contains the most discriminatory information
Helps in differentiating between the categories of input patterns
Facilitates decision making

Fish Classification: Salmon vs. Sea Bass
. Problem
Sorting fishes according to
species: sea bass or salmon

Fish Classification System
[https://www.byclb.com/TR/Tutorials/neural˙networks/ch1˙1.htm]


. Sensing
. Preprocessing
Image enhancement
Segmentation: isolate fishes from the background and from other
fishes
. Feature extraction
Discriminative features: What kind of information can distinguish
between these two species e.g length, lightness, width, number of fins
Reduce the data by measuring certain features
. Classification
Training phase: Find a classifier dividing the feature space into
decision regions according to the collected training samples
Testing phase: Use the learned classifier to find a category for a
fish according to its features
. Representation: fish
length feature
Histograms obtained
from training
samples of the two
classes
High overlap of the histograms ⇒Length alone is a poor feature

. Representation: fish
lightness feature
Histograms obtained
from training
samples of the two
classes
Histograms obtained from training

samples of the two classes
Less overlap of the histograms


. Feature Vector
Two-dimensional feature space
Each fish image is now represented as a point

. Two-dimensional Feature Space
Two features together are better than individual features


. Representation: feature vector
Ideal feature vector: discriminates well among samples of different
classes
small intra-class variation
large interclass separation
Ideal feature vector needs a simple decision boundary
Finding an ideal feature vector may be hard.
Divide the load between feature extraction and classification phases.
Practical issues about features
Invariant properties (e.g. Invariant to translation, rotation, scale,
· · ·)
Robustness to noise
Easy to measure (real-time applications)

Feature Selection
Important issues
. Curse of dimensionality: Error rate may increase with too

many features (especially when a small number of training samples
is available)
. How many features are required which subset of features can be

selected!
Some features may be redundant (highly correlated)
features can increase error rate

Complex Decision Boundary
. Generalization ability
Central aim of designing a classifier is to correctly classify new (test)
inputs and not to find the optimal performance on the training data

Boundary with Good Generalization
. Simple decision boundaries are preferred

Decision Boundary Choice

Remember!
Occam’s Razor:
Entities are not to be multiplied without necessity.

William of Occam (1284-1347)

Statistical Pattern Recognition
[Jain et al., Statistical Pattern Recognition:A Review, PAMI 22(11), 2000]

Feature Extraction & Classification
. Feature extraction is usually much more problem and domain

dependent than classification
. However, no particular classification algorithm is also “the best”

for all tasks
No free lunch theorem
”Any two optimization algorithms are equivalent when their
performance is averaged across all possible problems”,
Wolpert and Macready [2005]

Feature Extraction & Classification
. Feature extraction is usually much more problem and domain

dependent than classification
. However, no particular classification algorithm is also “the best”

for all tasks
No free lunch theorem
”Any two optimization algorithms are equivalent when their
performance is averaged across all possible problems”,
Wolpert and Macready [2005]
In practice, we combine prior knowledge about the classification

task with a search process
Some different types of classifiers are developed and compared.

The Design Cycle
. Data collection
. Feature Choice
. Model Choice
. Training
. Evaluation

The Design Cycle
. Data collection
How do we know when we have
collected a sufficient set of training
samples!
. Feature Choice
. Model Choice
. Training
. Evaluation

The Design Cycle
. Data collection
. Feature Choice
What features are better to extract
and select!
. Model Choice
. Training
. Evaluation

The Design Cycle
. Data collection
. Feature Choice
. Model Choice
How are we to know to reject a class
of models and try another one!
. Training
. Evaluation

The Design Cycle
. Data collection
. Feature Choice
. Model Choice
. Training
Use a training procedure to
determine the classifier based on the
chosen model and available training
samples
. Evaluation

The Design Cycle
. Data collection
. Feature Choice
. Model Choice
. Training
. Evaluation
Error rate (or performance)
After evaluation, we may decide to
get back to one of the previous steps
and change our decision (e.g., chosen
features)

System Performance
. Error (or misclassification) rate
. Cost or risk: More general than error rate
. Robustness
. Speed

Computational Complexity
. What is the trade-off between computational efficiency and

performance?
. How does an algorithm scale as a function of the number of

features, training samples, or categories?

Roadmap Shahid Bahonar university of Kerman
Outline
. Roadmap

Roadmap Shahid Bahonar university of Kerman
Road map
. Classification
Classifiers based on Bayesian decision theory
PDF estimation
ML, MAP Bayesian estimation
Non-parametric methods
Discriminant functions
SVM & kernels
Neural Networks
Evaluation of classifiers
. Dimensionality reduction
Feature selection
Feature extraction (linear and non-linear dimensionality reduction)
. Clustering
Partitional & hierarchical clustering
Spectral clustering
. Classification of sequential data (HMM)
. Semi-supervised learning
. Some
Hadis Mohseniadvanced topics and Pattern
Statistical applications
Recognition Course 33 / 33

01 Introduction

Uploaded by

Copyright:

Available Formats

01 Introduction

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

01 Introduction

Uploaded by

Copyright:

Available Formats

Shahid Bahonar university of Kerman

Introduction to Pattern Recognition

September 17, 2019

Hadis Mohseni Statistical Pattern Recognition Course 1 / 33

. Text Books & References

. Pattern Recognition: Definition and Applications

. Main approaches for Pattern Recognition

. Statistical Pattern Recognition

Hadis Mohseni Statistical Pattern Recognition Course 2 / 33

. Text Books & References

. Pattern Recognition: Definition and Applications

. Main approaches for Pattern Recognition

. Statistical Pattern Recognition

Hadis Mohseni Statistical Pattern Recognition Course 3 / 33

Text Books & References

. R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification,

. C. M. Bishop, Pattern Recognition and Machine Learning, 2006.

. S. Theodoridis and K. Koutroumbas, Pattern Recognition, 4th

. Good papers from PAMI journal, PR journal, ICPR conf, ICIP

. Related sites: IAPR association, statistical PR courses, etc.

Hadis Mohseni Statistical Pattern Recognition Course 4 / 33

. Text Books & References

. Pattern Recognition: Definition and Applications

. Main approaches for Pattern Recognition

. Statistical Pattern Recognition

Hadis Mohseni Statistical Pattern Recognition Course 5 / 33

. Pattern: “A pattern is the opposite of a chaos; it is an entity

. Pattern class: A set of similar patterns

Hadis Mohseni Statistical Pattern Recognition Course 6 / 33

Hadis Mohseni Statistical Pattern Recognition Course 6 / 33

Hadis Mohseni Statistical Pattern Recognition Course 6 / 33

Hadis Mohseni Statistical Pattern Recognition Course 6 / 33

Hadis Mohseni Statistical Pattern Recognition Course 6 / 33

Hadis Mohseni Statistical Pattern Recognition Course 6 / 33

Hadis Mohseni Statistical Pattern Recognition Course 6 / 33

Pattern Recognition: Definition

X “A field whose objective is to assign an object or event to one of a

. Relate a perceived pattern to

. Given an input pattern, make a

X Related fields: Machine learning, computer vision, data mining

Hadis Mohseni Statistical Pattern Recognition Course 7 / 33

Pattern Recognition: Importance

. “The problem of searching for patterns in data is a

. “The real power of human thinking is based on

. Pattern recognition is needed in designing almost all automated

Hadis Mohseni Statistical Pattern Recognition Course 8 / 33

Pattern Recognition: Applications

Pattern Recognition: Applications

Hadis Mohseni Statistical Pattern Recognition Course 9 / 33

. Text Books & References

. Pattern Recognition: Definition and Applications

. Main approaches for Pattern Recognition

. Statistical Pattern Recognition

Hadis Mohseni Statistical Pattern Recognition Course 10 / 33

Main approaches for Pattern Recognition

Hadis Mohseni Statistical Pattern Recognition Course 11 / 33

Structural Patten Recognition

. Assumption: pattern structure is quantifiable and extractable =⇒