Introduction to Pattern Recognition Part I

Selim Aksoy Bilkent University Department of Computer Engineering [email protected]

RETINA Pattern Recognition Tutorial, Summer 2005

Human Perception
Humans have developed highly sophisticated skills for sensing their environment and taking actions according to what they observe, e.g., recognizing a face understanding spoken words reading handwriting distinguishing fresh food from its smell We would like to give similar capabilities to machines.

RETINA Pattern Recognition Tutorial, Summer 2005

1/25

What is Pattern Recognition?

A pattern is an entity, vaguely dened, that could be given a name, e.g., ngerprint image speech signal handwritten word DNA sequence human face ... Pattern recognition is the study of how machines can observe the environment learn to distinguish patterns of interest make sound and reasonable decisions about the categories of the patterns
RETINA Pattern Recognition Tutorial, Summer 2005 2/25

Human and Machine Perception

We are often inuenced by the knowledge of how patterns are modeled and recognized in nature when we develop pattern recognition algorithms. Research on machine perception also helps us gain deeper understanding and appreciation for pattern recognition systems in nature. Yet, we also apply many techniques that are purely numerical and do not have any correspondence in natural systems.

RETINA Pattern Recognition Tutorial, Summer 2005

3/25

An Example
Problem: Sorting incoming sh on a conveyor belt according to species Assume that we have only two kinds of sh: sea bass salmon
Figure 1: Picture taken from a camera.

RETINA Pattern Recognition Tutorial, Summer 2005

4/25

An Example: Decision Process

What kind of information can distinguish one species from the other? length, width, weight, number and shape of ns, tail shape, etc. What can cause problems during sensing? lighting conditions, position of sh on the conveyor belt, camera noise, etc. What are the steps in the process? capture image isolate sh take measurements make decision
RETINA Pattern Recognition Tutorial, Summer 2005 5/25

An Example: Selecting Features

Assume a sherman told us that a sea bass is generally longer than a salmon. We can use length as a feature and decide between sea bass and salmon according to a threshold on length. How can we choose this threshold?

RETINA Pattern Recognition Tutorial, Summer 2005

6/25

An Example: Selecting Features

Figure 2: Histograms of the length feature for two types of sh in training samples. How can we choose the threshold l to make a reliable decision?
RETINA Pattern Recognition Tutorial, Summer 2005 7/25

An Example: Selecting Features

Even though sea bass is longer than salmon on the average, there are many examples of sh where this observation does not hold. Try another feature: average lightness of the sh scales.

RETINA Pattern Recognition Tutorial, Summer 2005

8/25

An Example: Selecting Features

Figure 3: Histograms of the lightness feature for two types of sh in training samples. It looks easier to choose the threshold x but we still cannot make a perfect decision.
RETINA Pattern Recognition Tutorial, Summer 2005 9/25

An Example: Cost of Error

We should also consider costs of dierent errors we make in our decisions. For example, if the sh packing company knows that: Customers who buy salmon will object vigorously if they see sea bass in their cans. Customers who buy sea bass will not be unhappy if they occasionally see some expensive salmon in their cans. How does this knowledge aect our decision?

RETINA Pattern Recognition Tutorial, Summer 2005

10/25

An Example: Multiple Features

Assume we also observed that sea bass are typically wider than salmon. We can use two features in our decision: lightness: x1 width: x2 Each sh image is now represented as a point (feature vector ) x1 x= x2 in a two-dimensional feature space.

RETINA Pattern Recognition Tutorial, Summer 2005

11/25

An Example: Multiple Features

Figure 4: Scatter plot of lightness and width features for training samples. We can draw a decision boundary to divide the feature space into two regions. Does it look better than using only lightness?
RETINA Pattern Recognition Tutorial, Summer 2005 12/25

An Example: Multiple Features

Does adding more features always improve the results? Avoid unreliable features. Be careful about correlations with existing features. Be careful about measurement costs. Be careful about noise in the measurements. Is there some curse for working in very high dimensions?

RETINA Pattern Recognition Tutorial, Summer 2005

13/25

An Example: Decision Boundaries

Can we do better with another decision rule? More complex boundaries. models result in more complex

Figure 5: We may distinguish training samples perfectly but how can we predict how well we can generalize to unknown samples?
RETINA Pattern Recognition Tutorial, Summer 2005 14/25

An Example: Decision Boundaries

How can we manage the tradeo between complexity of decision rules and their performance to unknown samples?

Figure 6: Dierent criteria lead to dierent decision boundaries.

RETINA Pattern Recognition Tutorial, Summer 2005

15/25

Pattern Recognition Systems

Physical environment Data acquisition/sensing Training data

Preprocessing

Preprocessing

Feature extraction Features Classification Model

Feature extraction/selection Features Model learning/estimation

Postprocessing Decision

Figure 7: Object/process diagram of a pattern recognition system.

RETINA Pattern Recognition Tutorial, Summer 2005 16/25

Pattern Recognition Systems

Data acquisition and sensing: Measurements of physical variables Important issues: bandwidth, resolution, sensitivity, distortion, SNR, latency, etc. Pre-processing: Removal of noise in data Isolation of patterns of interest from the background Feature extraction: Finding a new representation in terms of features

RETINA Pattern Recognition Tutorial, Summer 2005

17/25

Pattern Recognition Systems

Model learning and estimation: Learning a mapping between features and pattern groups and categories Classication: Using features and learned models to assign a pattern to a category Post-processing: Evaluation of condence in decisions Exploitation of context to improve performance Combination of experts
RETINA Pattern Recognition Tutorial, Summer 2005 18/25

Pattern Recognition Applications

Table 1: Example pattern recognition applications.
Problem Domain Document image analysis Document classication Document classication Multimedia database retrieval Speech recognition Natural language processing Biometric recognition Medical Military Industrial automation Industrial automation Remote sensing Bioinformatics Data mining Application Optical character recognition Internet search Junk mail ltering Internet search Telephone directory assistance Information extraction Personal identication Diagnosis Automatic target recognition Printed circuit board inspection Fruit sorting Forecasting crop yield Sequence analysis Searching for meaningful patterns Input Pattern Document image Text document Email Video clip Speech waveform Sentences Face, iris, ngerprint Microscopic image Optical or infrared image Intensity or range image Images taken on a conveyor belt Multispectral image DNA sequence Points in multidimensional space Pattern Classes Characters, words Semantic categories Junk/non-junk Video genres Spoken words Parts of speech Authorized users for access control Cancerous/healthy cell Target type Defective/non-defective product Grade of quality Land use categories Known types of genes Compact and well-separated clusters

RETINA Pattern Recognition Tutorial, Summer 2005

19/25

The Design Cycle

Collect data Select features Select model Train classifier Evaluate classifier

Figure 8: The design cycle.

Data collection: Collecting training and testing data How can we know when we have adequately large and representative set of samples?

RETINA Pattern Recognition Tutorial, Summer 2005

20/25

The Design Cycle

Feature selection: Domain dependence and prior information Computational cost and feasibility Discriminative features Similar values for similar patterns Dierent values for dierent patterns Invariant features with respect to translation, rotation and scale Robust features with respect to occlusion, distortion, deformation, and variations in environment

RETINA Pattern Recognition Tutorial, Summer 2005

21/25

The Design Cycle

Model selection: Domain dependence and prior information Denition of design criteria Parametric vs. non-parametric models Handling of missing features Computational complexity Types of models: templates, decision-theoretic or statistical, syntactic or structural, neural, and hybrid How can we know how close we are to the true model underlying the patterns?

RETINA Pattern Recognition Tutorial, Summer 2005

22/25

The Design Cycle

Training: How can we learn the rule from data? Supervised learning: a teacher provides a category label or cost for each pattern in the training set Unsupervised learning: the system forms clusters or natural groupings of the input patterns Reinforcement learning: no desired category is given but the teacher provides feedback to the system such as the decision is right or wrong

RETINA Pattern Recognition Tutorial, Summer 2005

23/25

The Design Cycle

Evaluation: How can we estimate the performance with training samples? How can we predict the performance with future data? Problems of overtting and generalization

RETINA Pattern Recognition Tutorial, Summer 2005

24/25

Summary
Pattern recognition techniques nd applications in many areas: machine learning, statistics, mathematics, computer science, biology, etc. There are many sub-problems in the design process. Many of these problems can indeed be solved. More complex learning, searching and optimization algorithms are developed with advances in computer technology. There remain many fascinating unsolved problems.

RETINA Pattern Recognition Tutorial, Summer 2005

25/25