PATTERN RECOGNITION

Lecture Notes on Pattern Recognition and Bayes Decision Theory

Overview: This lecture provided a comprehensive overview of the
challenges in pattern recognition, highlighting the diverse problem
domains and the increasing complexity of decision boundaries needed
to address them. It covered fundamental concepts such as feature
representation, feature space, and class separability, which are essential
for understanding classification problems. Additionally, the discussion
introduced neural networks and hyperbox classifiers as effective tools
for handling complex classification scenarios, particularly when
traditional linear and nonlinear classifiers fall short.
Learning Objectives: By the end of this lecture, students should be
able to:
1. Illustrate the mapping of patterns to feature space and how
decision boundaries are established.
2. Describe the concept of linear and non-linear separability in
classification problems.
3. Explain the role of decision boundaries in classification and their
implications.
4. Differentiate between single-layer and multi-layer perceptrons in
classification tasks.
5. Understand how neural networks approximate complex decision
boundaries.
Introduction: Good morning. Today, we will begin our discussion on
pattern recognition problems, with a particular focus on Bayes decision
theory, which serves as the foundation of statistical pattern recognition.
Before delving into the specifics of Bayes decision theory, let us briefly
recapitulate the fundamental aspects of pattern recognition.
Feature Extraction in Pattern Recognition: In pattern recognition,
an object can be described using various features. These features may
be derived from:
Boundary Features: Shape descriptors or shape features.
Region Features: Derived from the enclosed region, such as texture,
color, or intensity.

AOT/ME/ESC-CSBS-601/SKL Page 1
 PATTERN RECOGNITION

No single feature alone is sufficient to accurately describe an object.

Instead, multiple features must be considered together, forming a
feature vector. The feature vector consists of components from
different feature categories (e.g., boundary features, shape features, and
region features), arranged in a predefined order. The same order must
be maintained throughout the process of pattern modelling and
recognition.
The dimensionality of the feature vector depends on the complexity of
the problem being addressed. It can range from as low as two or three
dimensions to as high as 500 dimensions. Increasing the dimensionality
generally enhances the accuracy of pattern representation.
Feature Space Representation: Once a pattern is described using a
feature vector, it is mapped to a point in the feature space:
A two-dimensional feature vector maps to a point in a two-
dimensional space.
A three-dimensional feature vector maps to a point in a three-
dimensional space.
An n-dimensional feature vector maps to a point in an n-dimensional
feature space.
As the dimensionality increases, the complexity of visualization also
increases, making it difficult to represent patterns graphically.
Classification and Decision Boundaries: In classification problems,
patterns belong to different categories, denoted as ω1 and ω2. Each
category's patterns form a distinct cluster (point cloud) in the feature
space:
Patterns belonging to class ω1 are closely clustered together.
Patterns belonging to class ω2 form another distinct cluster.
The two clusters are separated by a decision boundary.
For a simple case where the two classes are linearly separable, the
decision boundary is a straight line in two dimensions, a plane in three
dimensions, or a hyperplane in higher dimensions. If a new feature
vector falls on one side of the boundary, it is classified as belonging to
ω1, and if it falls on the other side, it belongs to ω2.

AOT/ME/ESC-CSBS-601/SKL Page 2
 PATTERN RECOGNITION

Supervised Learning and Linear Separability

A classifier is trained using a set of training samples, where the class
labels are known. This is called supervised learning. If the two classes
can be separated by a linear decision boundary, they are linearly
separable. In higher dimensions:
A straight line separates the two classes in a two-dimensional space.
A plane separates them in a three-dimensional space.
A hyperplane separates them in an n-dimensional space.
Non-Linearly Separable Classes
In cases where the classes are not linearly separable, a more complex
non-linear decision boundary is required. The most common non-
linear classifiers include:
Quadratic Classifier: Uses a quadratic function to separate the classes.
Cubic Classifier: Uses a cubic function for classification.
Higher-Order Classifiers: Rarely used due to complexity in designing
the decision boundary.
If the decision boundary is highly complex, analytical methods may not
be feasible. Instead, neural networks are used, where decision
boundaries are learned through weight matrices.
Neural Network Classifiers
Neural networks provide a flexible approach to pattern classification
by approximating non-linear decision boundaries. A neural network
consists of:
Input Layer: Receives the feature vector.
Hidden Layers: Intermediate layers that refine feature representations.
Output Layer: Produces the classification result.
Each connection in the neural network has an associated weight, which
is learned during training. The neural network adjusts these weights to
approximate the optimal decision boundary. The simplest form of a
neural network is the single-layer perceptron, which can only handle
linearly separable problems. For non-linear decision boundaries, multi-
layer perceptrons (MLPs) are used, incorporating one or more hidden
layers.

AOT/ME/ESC-CSBS-601/SKL Page 3
 PATTERN RECOGNITION

The neural network learns a piecewise linear approximation of a

complex decision boundary, making it effective for problems where an
analytical expression for the decision boundary is not feasible.
Complex Class Distributions:
 Intertwined Classes: When classes are highly intertwined (e.g.,

spiral-shaped distributions), traditional linear and non-linear

classifiers may fail.
 Hyperbox Classifiers: One approach for such cases is to define

hyperboxes (rectangles in 2D, hyper-rectangles in nD) around

regions of the feature space belonging to each class. These
hyperboxes can then be grouped under "umbrellas" representing
each class.
 Min-Max Classifiers: Hyperboxes can be represented by their

minimum and maximum points.

 Fuzzy Min-Max Classifiers: Incorporate fuzzy set theory to

handle uncertainty in class membership.

Conclusion: Pattern recognition relies on feature extraction, feature
vector representation, and classification based on decision boundaries.
When classes are linearly separable, a simple hyperplane-based
classifier suffices. For non-linearly separable problems, neural
networks provide a robust solution by approximating complex decision
boundaries. In the next lecture, we will explore Bayes decision theory
in greater depth and examine its application in statistical pattern
recognition.
Multiple Choice Questions (MCQs)
1. What is the basis of statistical pattern recognition?
a) Decision Trees
b) Bayes Decision Theory
c) Neural Networks
d) Genetic Algorithms
2. Which of the following is NOT considered a type of feature
in pattern recognition?
a) Shape features

AOT/ME/ESC-CSBS-601/SKL Page 4
 PATTERN RECOGNITION

b) Texture features
c) Intensity features
d) Algorithmic features
3. What is a feature vector?
a) A single feature representing an object
b) A collection of features arranged in a specific order
c) A decision boundary separating classes
d) A neural network output
4. What determines the dimensionality of the feature vector?
a) The color of the object
b) The number of categories in classification
c) The complexity of the pattern recognition problem
d) The speed of computation
5. If an object is represented by a three-dimensional feature
vector, where is it mapped?
a) A two-dimensional feature space
b) A three-dimensional feature space
c) A one-dimensional feature space
d) A hyperplane
6. Why is a two-dimensional feature space commonly used for
illustrations?
a) It is the most accurate representation of real-world data
b) It simplifies visualization and explanation
c) It ensures a higher classification accuracy
d) It requires fewer computations
7. What does a decision boundary do in pattern recognition?
a) Divides feature space into regions corresponding to
different classes
b) Computes the mean of all feature vectors
c) Determines the exact shape of the object
d) Eliminates unnecessary features from the dataset
8. When two classes are linearly separable, how can they be
separated?

AOT/ME/ESC-CSBS-601/SKL Page 5
 PATTERN RECOGNITION

a) By a curved boundary
b) By a straight line or hyperplane
c) By clustering algorithms
d) By neural networks only
9. What happens when two classes are NOT linearly
separable?
a) A straight-line decision boundary can still be used
b) The problem must be redefined
c) A non-linear decision boundary is needed
d) Pattern recognition is not possible
10. Which classifier is commonly used for non-linearly
separable classes?
a) Linear classifier
b) Quadratic or cubic classifier
c) Decision tree classifier
d) Single-layer perceptron
11. What is the role of neural networks in pattern
recognition?
a) They find the mean feature value
b) They form a collection of straight-line boundaries to
model complex decision boundaries
c) They always provide a linear solution
d) They eliminate the need for feature extraction
12. What is a single-layer perceptron?
a) A network with one input and one output layer, but no
hidden layers
b) A network with multiple hidden layers
c) A neural network used only for image classification
d) A system with only one neuron
13. What is a multilayer perceptron (MLP)?
a) A classifier that only works for binary classification
b) A neural network with one or more hidden layers

AOT/ME/ESC-CSBS-601/SKL Page 6
 PATTERN RECOGNITION

c) A decision tree model

d) A purely statistical model
14. What is the main advantage of adding more hidden
layers in a neural network?
a) It reduces computation time
b) It allows encoding more complex decision boundaries
c) It makes the network linear
d) It eliminates training samples
15. What is the role of neural networks in classification?
a) To create handcrafted rules for classification
b) to approximate non-linear decision boundaries using
piecewise linear segments
c) to reduce the dimensionality of the feature vector
d) to remove irrelevant data from the dataset

AOT/ME/ESC-CSBS-601/SKL Page 7