Pattern Recognition and

Computer Vision NOTES

UNIT - 1

Induction algorithms
Induction algorithms are a class of algorithms used in machine learning to create
models based on observed data. They work by generalizing from specific
examples to broader rules or patterns. Here are some key points about induction
algorithms:

Induction Process: The process involves taking a set of training examples and
deriving a general rule that can be applied to new, unseen instances. This is
often done through methods like decision trees, where the algorithm learns to
classify data based on features.

Types of Induction Algorithms: There are various types of induction

algorithms, including:

Decision Trees: These algorithms create a model that predicts the value of
a target variable based on several input variables. They split the data into
subsets based on the value of input features.

Rule-Based Systems: These systems use a set of "if-then" rules to make

predictions or decisions based on input data.

Neural Networks: These are inspired by the human brain and consist of
interconnected nodes (neurons) that process data in layers.

Learning from Data: Induction algorithms learn from data by identifying

patterns and relationships within the training set. The quality of the induction
depends on the amount and quality of the training data.

Pattern Recognition and Computer Vision NOTES 1

 Applications: Induction algorithms are widely used in various fields, including
finance for credit scoring, healthcare for disease diagnosis, and marketing for
customer segmentation.

Challenges: One of the main challenges with induction algorithms is

overfitting, where the model becomes too complex and captures noise in the
training data rather than the underlying pattern. Techniques like cross-
validation and pruning are often used to mitigate this issue.

Induction algorithms are fundamental to the field of machine learning and are
essential for building predictive models that can generalize well to new data.

Other Induction Methods

Other induction methods encompass various techniques used for classification
and pattern recognition beyond traditional algorithms. Here are some key points
regarding these methods:

Non-parametric Estimation: There are two overarching approaches for

pattern classification:

One approach estimates densities and uses them for classification,

exemplified by methods like Parzen windows and probabilistic neural
networks.

The other approach directly chooses categories, as seen in k-nearest-

neighbor methods and relaxation networks.

Nearest-Neighbor Methods: These methods are foundational in classification,

where the category of a test pattern is determined based on the categories of
its nearest neighbors in the training data. The error rate of the nearest-
neighbor method is bounded by twice the Bayes error rate in the limit of
infinite training data.

Relaxation Methods: These create "basins of attraction" around training

prototypes, allowing for easy identification of category labels for test patterns
that lie within these basins.

Dimensionality Reduction: Techniques like Fisher's linear discriminant are

used to reduce the dimensionality of the feature space, aiming to find a
subspace where categories are best separated.

Pattern Recognition and Computer Vision NOTES 2

 Rule-Based Methods: These methods utilize general relationships among
entities to build classifiers based on rules. They are integral to expert systems
in artificial intelligence, although their use in pattern recognition has been
modest.

Grammatical Inference: This involves learning grammars from data, which

can be used for classification tasks. It is a more specialized approach that can
be tailored based on the type of grammar being inferred.

In summary, other induction methods include a variety of techniques that leverage

different principles, such as non-parametric estimation, nearest-neighbor
classification, relaxation methods, dimensionality reduction, rule-based systems,
and grammatical inference, to enhance classification performance.

Rule Induction
Rule induction is a method used in machine learning to create classifiers based on
rules that describe relationships among entities. Here are some key points about
rule induction:

Rule-Based Methods: These methods are integral to expert systems in

artificial intelligence. They focus on a broad class of if-then rules for
representing and learning relationships. For example, a simple rule could be:

IF Swims(x) AND HasScales(x) THEN Fish(x), which indicates that if an

object x has the properties of swimming and having scales, then it is
classified as a fish.

Learning Rules: The process of learning rules can involve various algorithms.
For instance, decision trees can be trained using methods like CART, ID3, or
C4.5, and then simplified to extract rules. The learning process often involves
identifying the best simple rule that describes the largest number of training
examples and iterating to refine these rules.

Sequential Covering: This approach involves learning a single rule, deleting

the examples it explains, and iterating this process. This leads to a disjunctive
set of rules that cover the training data. The designer must specify predicates
and functions based on prior knowledge of the problem domain.

Types of Rules: There are two main types of if-then rules:

Pattern Recognition and Computer Vision NOTES 3

 Propositional Rules: These describe specific instances without variables.

First-Order Logic Rules: These allow for variables and can express more
general relationships.

Applications and Challenges: Rule induction is used in various applications,

but it can face challenges such as high noise levels in data, which complicates
the use of rules. Additionally, the choice of predicates and their evaluation can
be difficult tasks.

In summary, rule induction is a powerful technique for building classifiers that can
be easily interpreted and applied in various domains, although it requires careful
design and consideration of the underlying data.

Decision Trees
Decision trees are a method used in machine learning for classification tasks.
They classify patterns through a sequence of questions, where the next question
depends on the answer to the current one. This approach is particularly useful for
non-metric data, as all questions can be framed in a "yes/no" or "true/false"
format. Here are some key points about decision trees:

Structure: A decision tree consists of nodes connected by branches. The root

node is at the top, and it asks for the value of a specific property of the
pattern. Each link from the root corresponds to a possible value, leading to
subsequent nodes until reaching terminal or leaf nodes, which have no further
links. Each leaf node is assigned a category label, and the test pattern is
classified based on the label of the leaf node reached.

Classification Process: The classification begins at the root node, where a

property is queried. Depending on the answer, the appropriate link is followed
to a descendant node. This process continues until a leaf node is reached,
which provides the final classification.

Interpretability: One of the advantages of decision trees is their

interpretability. The decisions made for any test pattern can be easily
understood as a conjunction of decisions along the path to the corresponding
leaf node. For example, if the properties are {taste, color, shape, size}, a
pattern might be classified based on specific conditions related to these
properties.

Pattern Recognition and Computer Vision NOTES 4

 Growing the Tree: The process of creating a decision tree involves recursively
splitting the training data into smaller subsets based on the properties that
best separate the categories. The goal is to create pure subsets, where all
samples in a subset belong to the same category. If a node is impure, the tree
may continue to split until a stopping criterion is met.

Pruning: To avoid overfitting, decision trees can be pruned. This involves

removing branches that have little importance or that do not contribute
significantly to the classification accuracy. Pruning helps simplify the model
and improve its generalization to new data.

Applications: Decision trees are widely used in various fields, including

finance, healthcare, and marketing, due to their ability to handle both
numerical and categorical data effectively.

In summary, decision trees are a powerful and interpretable method for

classification tasks, utilizing a structured approach to decision-making based on
the properties of the data.

Bayesian Methods
Bayesian methods are a class of statistical techniques that incorporate prior
knowledge or beliefs, along with observed data, to make inferences about
unknown parameters. Here are some key points about Bayesian methods:

Conceptual Framework: Unlike maximum likelihood methods, which treat

parameters as fixed but unknown, Bayesian methods view parameters as
random variables with a known prior distribution. Observing data updates this
prior to form a posterior distribution, reflecting our revised beliefs about the
parameters after seeing the data.

Posterior Distribution: The posterior distribution is computed using Bayes'

theorem, which states:

P(θ | D) = (P(D | θ) * P(θ)) / P(D)

where (P(\theta|D)) is the posterior, (P(D|\theta)) is the likelihood, (P(\theta)) is

the prior, and (P(D)) is the evidence.

Learning from Data: As more data is observed, the posterior distribution

typically sharpens, concentrating around the true parameter values. This

Pattern Recognition and Computer Vision NOTES 5

 phenomenon is known as Bayesian learning.

Model Selection: Bayesian methods allow for model selection by comparing

the posterior probabilities of different models, taking into account the prior
distributions over the models.

Bias and Variance Tradeoff: Bayesian methods explicitly address the tradeoff
between bias and variance, which is crucial for effective estimation and
generalization.

Applications: Bayesian methods are widely used in various fields, including

machine learning, pattern recognition, and decision-making, due to their ability
to incorporate prior knowledge and handle uncertainty effectively.

Challenges: One of the main challenges with Bayesian methods is the need to
specify prior distributions, which can be subjective and may not always be
easy to determine. Additionally, computational complexity can arise, especially
in high-dimensional parameter spaces.

In summary, Bayesian methods provide a robust framework for statistical

inference that combines prior beliefs with observed data, allowing for a
comprehensive understanding of uncertainty in model parameters.

Other Bayesian Methods

Bayesian methods often involve a weighted average of models, reflecting the
remaining uncertainty about the possible models. This approach uses the full
distribution ( p(\theta|D) ) to utilize more information brought to the problem than
do maximum likelihood methods. Bayesian methods are particularly useful when
there is strong asymmetry in the distribution ( p(\theta|D) ), as they can exploit this
information to provide better results.
General Bayesian methods with a "flat" or uniform prior are equivalent to
maximum likelihood methods when there is a large amount of data leading to a
strongly peaked ( p(\theta|D) ). However, when ( p(\theta|D) ) is broad or
asymmetric around ( \theta ), Bayesian methods and maximum likelihood methods
are likely to yield different distributions ( p(x|D) ).
Bayesian methods also make explicit the crucial problem of bias and variance
tradeoffs, which is essential for designing classifiers by determining the posterior
densities for each category and classifying a test point by the maximum posterior.

Pattern Recognition and Computer Vision NOTES 6

 Naïve Bayes
Naïve Bayes is a classification technique based on applying Bayes' theorem with
strong (naïve) independence assumptions between the features. Here are some
key points about Naïve Bayes:

Independence Assumption: The fundamental assumption of Naïve Bayes is

that the features are conditionally independent given the class label. This
means that the presence of a particular feature does not affect the presence
of any other feature, given the class label. Mathematically, this can be
expressed as:

P(ω_k | x) ∝ ∏_{i=1}^{d} P(x_i | ω_k)

where (P(\omega_k|x)) is the posterior probability of class (\omega_k) given
the features (x), and (P(x_i|\omega_k)) is the likelihood of feature (x_i) given
class (\omega_k).

Bayes' Theorem: The classification is based on Bayes' theorem, which allows

us to compute the posterior probability of each class given the features. The
formula is:
P(ω_k | x) = (P(x | ω_k) * P(ω_k)) / P(x)
where (P(x|\omega_k)) is the likelihood, (P(\omega_k)) is the prior probability
of class (\omega_k), and (P(x)) is the evidence.

Practical Use: Despite the strong independence assumption, Naïve Bayes

often performs surprisingly well in practice, especially in text classification
tasks such as spam detection and sentiment analysis. This is because the
independence assumption simplifies the computation of the likelihoods,
making the model efficient and easy to implement.

Types of Naïve Bayes Classifiers: There are several variations of Naïve Bayes
classifiers, including:

Gaussian Naïve Bayes: Assumes that the features follow a Gaussian

distribution.

Multinomial Naïve Bayes: Suitable for discrete counts, commonly used in

document classification.

Bernoulli Naïve Bayes: Similar to multinomial but assumes binary features.

Pattern Recognition and Computer Vision NOTES 7

 Applications: Naïve Bayes classifiers are widely used in various applications,
including:

Text classification (e.g., spam filtering, sentiment analysis)

Medical diagnosis

Recommendation systems.

In summary, Naïve Bayes is a simple yet effective classification method that

leverages Bayes' theorem and the assumption of feature independence to make
predictions based on observed data.

Naive Bayes Induction for Numeric Attributes

Naïve Bayes induction for numeric attributes involves applying the Naïve Bayes
classification technique while considering that the features are continuous rather
than discrete. Here are some key points regarding this approach:

Independence Assumption: The Naïve Bayes classifier assumes that the

features are conditionally independent given the class label. This means that
the presence of a particular feature does not affect the presence of any other
feature, given the class label.

Class-Conditional Probability Densities: For numeric attributes, the class-

conditional probability densities (p(x|ω_j)) can be modeled using distributions
such as Gaussian. This allows the computation of the likelihood of the
observed numeric features given each class.

Gaussian Naïve Bayes: A common approach is to assume that the numeric

features follow a Gaussian distribution. The parameters of the Gaussian (mean
and variance) are estimated from the training data for each class.

Classification Process: The classification is performed by calculating the

posterior probability for each class using Bayes' theorem, which incorporates
the likelihood of the features given the class and the prior probability of the
class.

Applications: This method is widely used in various applications, including

text classification, medical diagnosis, and recommendation systems, where
numeric attributes are prevalent.

Pattern Recognition and Computer Vision NOTES 8

 In summary, Naïve Bayes induction for numeric attributes leverages the
independence assumption and models the numeric features using probability
distributions to classify instances effectively.

Correction to the Probability Estimation

Correction to the probability estimation involves addressing the errors that arise
from estimating probabilities based on finite samples. Here are some key points
regarding this topic:

Estimation Error: This error arises because the parameters are estimated from
a finite sample. The best way to reduce this error is by increasing the training
data, which can lead to more accurate estimates of the underlying
probabilities.

Bayes Error: This is the error due to overlapping densities (p(x|\omega_i)) for
different values of (i). It is an inherent property of the problem and cannot be
eliminated.

Model Error: This error occurs when the model used for estimation does not
accurately represent the true data-generating process. It can only be
eliminated if the designer specifies a model that includes the true model.

Maximum Likelihood vs. Bayesian Estimation: Maximum likelihood estimation

treats parameters as fixed but unknown, aiming to maximize the probability of
observing the given samples. In contrast, Bayesian estimation treats
parameters as random variables with a prior distribution, updating this
distribution based on observed data.

Practical Implications: In practice, the results from maximum likelihood and

Bayesian estimation are often similar, but the Bayesian approach provides a
more robust framework for incorporating prior knowledge and handling
uncertainty.

In summary, correction to probability estimation is crucial for improving

classification accuracy and involves understanding the sources of error, including
estimation, model, and Bayes errors.

Neural Networks

Pattern Recognition and Computer Vision NOTES 9

 Neural networks are a powerful tool for building classifiers and are based on the
structure and function of biological neurons. Here are some key points about
neural networks:

Structure: A typical neural network consists of an input layer, one or more

hidden layers, and an output layer. Each layer is made up of units (or neurons)
that are interconnected by modifiable weights. The input layer receives the
feature vector, while the output layer produces the classification results.

Feedforward Operation: During the feedforward operation, input patterns are

presented to the network, and each unit computes its activation based on the
weighted sum of its inputs. The output units then emit signals that serve as
discriminant functions for classification.

Backpropagation Algorithm: One of the most popular methods for training

neural networks is the backpropagation algorithm, which uses gradient
descent to minimize the error between the predicted outputs and the actual
target values. This method is simple and effective, allowing for the adjustment
of weights throughout the network.

Expressive Power: Neural networks, particularly multilayer networks, have the

capability to approximate any continuous function, making them extremely
powerful for various applications. The number of hidden layers and units can
be adjusted based on the complexity of the problem.

Regularization and Complexity Adjustment: It is crucial to manage the

complexity of the network to avoid overfitting. Techniques such as weight
decay and pruning can help in adjusting the complexity of the model.

Applications: Neural networks are widely used in various fields, including

pattern recognition, speech recognition, optical character recognition, and
more. Their ability to learn from data and generalize well makes them suitable
for complex tasks.

In summary, neural networks are a versatile and powerful method for classification
tasks, leveraging a structured approach to learning from data through
interconnected layers of neurons.

Genetic Algorithms

Pattern Recognition and Computer Vision NOTES 10

 Genetic algorithms are a class of optimization techniques inspired by the process
of natural selection. Here are some key points about genetic algorithms:

Representation: In basic genetic algorithms, each classifier is represented by

a binary string known as a chromosome. The mapping from the chromosome
to the features and other aspects of the classifier depends on the problem
domain, allowing designers flexibility in specifying this mapping.

Fitness Evaluation: The performance of each chromosome is evaluated based

on a score, which is typically a monotonic function of accuracy on a dataset.
This score is referred to as fitness, and it is used to rank the chromosomes.

Genetic Operators: The algorithm employs several genetic operators to

produce offspring for the next generation:

Replication: A chromosome is reproduced unchanged.

Crossover: Two chromosomes are mixed to create new offspring. A split

point is chosen randomly, and segments from both parents are combined
to form new chromosomes.

Mutation: Each bit in a chromosome has a small probability of being

changed, introducing variation into the population.

Algorithm Process: The basic genetic algorithm follows these steps:

1. Initialize parameters such as fitness threshold, crossover probability, and

mutation probability.

2. Evaluate the fitness of each chromosome.

3. Rank the chromosomes based on their fitness scores.

4. Select the best chromosomes for reproduction.

5. Apply crossover and mutation to create a new generation of

chromosomes.

6. Repeat the process until a chromosome exceeds the desired fitness

threshold.

Search Capability: Genetic algorithms are particularly effective in complex,

discontinuous search spaces where traditional optimization techniques may

Pattern Recognition and Computer Vision NOTES 11

 struggle. They can explore a wide range of potential solutions and converge
on high-quality classifiers.

Challenges: The performance of genetic algorithms can depend on various

factors, including the number of bits in the chromosomes, population size, and
the rates of crossover and mutation. Balancing these parameters is crucial for
effective search and optimization.

In summary, genetic algorithms utilize principles of natural selection and genetic

variation to optimize classifiers, making them suitable for complex classification
problems.

Instance‐based Learning
Instance-based learning is a type of learning method in machine learning that
relies on specific instances of training data to make predictions. Here are some
key points about instance-based learning:

Basic Concept: Instance-based learning algorithms store instances of the

training data and use them directly for classification or regression tasks. When
a new instance needs to be classified, the algorithm compares it to the stored
instances to find the most similar ones.

Similarity Measures: The effectiveness of instance-based learning heavily

depends on the choice of similarity measures, such as Euclidean distance or
Manhattan distance, to determine how closely related the new instance is to
the stored instances.

k-Nearest Neighbors (k-NN): One of the most common instance-based

learning algorithms is k-NN, which classifies a new instance based on the
majority class of its k nearest neighbors in the training data. The value of k can
significantly affect the performance of the classifier.

Memory Usage: Instance-based learning can require significant memory

since it needs to store all training instances. This can be a limitation when
dealing with large datasets.

Adaptability: This approach is highly adaptable to new data, as it does not

require retraining a model; instead, it simply adds new instances to the
dataset.

Pattern Recognition and Computer Vision NOTES 12

 Applications: Instance-based learning is widely used in various applications,
including pattern recognition, recommendation systems, and any domain
where similarity-based reasoning is beneficial.

In summary, instance-based learning is a flexible and intuitive approach that

leverages stored instances to make predictions, relying on similarity measures to
classify new data points effectively.

Support Vector Machines

Support Vector Machines (SVMs) are a powerful classification technique that
relies on finding the optimal hyperplane to separate data points from different
categories. Here are some key points about SVMs:

High-Dimensional Mapping: SVMs preprocess the data to represent patterns

in a high-dimensional space, typically much higher than the original feature
space. This is achieved through a nonlinear mapping function ( \phi() ) that
transforms the input data into a higher-dimensional space where it can be
linearly separated.

Linear Discriminant: In the transformed space, a linear discriminant is defined

as ( g(y) = a^T y ), where ( a ) is the weight vector and ( y ) is the transformed
pattern vector. The goal is to find a separating hyperplane such that ( z_k
g(y_k) \geq 1 ) for all training patterns, where ( z_k ) indicates the category of
each pattern.

Maximizing the Margin: The SVM aims to find the hyperplane that maximizes
the margin, which is the distance between the hyperplane and the nearest
data points from either category (the support vectors). A larger margin is
expected to lead to better generalization of the classifier.

Support Vectors: The support vectors are the training samples that are
closest to the hyperplane and are critical in defining the optimal separating
hyperplane. They are the most informative patterns for the classification task.

Expected Error Rate: The expected value of the generalization error rate is
bounded by the number of support vectors, which is independent of the
dimensionality of the transformed space. This means that even in high-
dimensional spaces, the complexity of the classifier is characterized by the
number of support vectors.

Pattern Recognition and Computer Vision NOTES 13

 Training Process: The training of an SVM involves solving a constrained
optimization problem to minimize the magnitude of the weight vector while
ensuring correct classification of the training data. This is often done using
methods like Lagrange multipliers and quadratic programming.

Applications: SVMs are widely used in various applications, including text

classification, image recognition, and bioinformatics, due to their effectiveness
in handling high-dimensional data and their robustness against overfitting.

In summary, Support Vector Machines are a sophisticated classification method

that leverages high-dimensional mapping and margin maximization to achieve
effective separation of data points from different categories.

UNIT - 2
Statistical Pattern Recognition
Statistical pattern recognition focuses on the statistical properties of patterns,
generally expressed in probability densities. Here are some key points about
statistical pattern recognition:

Classification Task: At its core, classification is about recovering the model

that generated the patterns. Different classification techniques are useful
depending on the type of candidate models themselves.

Statistical Properties: In statistical pattern recognition, the model for a pattern

may consist of a specific set of features, although the actual pattern sensed
may be corrupted by random noise. This approach emphasizes the statistical
properties of the patterns.

Representation: Achieving a good representation of the patterns is crucial.

This representation should reveal the structural relationships among the
components and express the true (unknown) model of the patterns. Patterns
can be represented in various forms, such as vectors of real-valued numbers
or ordered lists of attributes.

Noise and Features: The presence of noise can complicate the classification
process. It is essential to choose features carefully to enable successful

Pattern Recognition and Computer Vision NOTES 14

 pattern classification.

Comparison with Other Methods: Statistical pattern recognition differs from

syntactic pattern recognition, which uses crisp logical rules to describe
decisions. For example, classifying an English sentence as grammatical or not
would rely on rules rather than statistical descriptions.

Applications: Statistical pattern recognition techniques are applied across

various fields, including image processing, speech recognition, and more,
where understanding the underlying statistical properties of the data is
essential.

In summary, statistical pattern recognition is a method that leverages statistical

properties and probability densities to classify patterns effectively, emphasizing
the importance of representation and feature selection in the process.

Classification and regression

Classification and regression are two fundamental tasks in statistical pattern
recognition and machine learning. Here are some key points regarding these
tasks:

Classification: This task involves assigning a category label to a given input

pattern based on its features. The goal is to recover the model that generated
the patterns, which can be achieved through various classification techniques.
Different methods are useful depending on the type of candidate models. In
statistical pattern recognition, the focus is on the statistical properties of the
patterns, often expressed in probability densities. The model for a pattern may
consist of a specific set of features, although the actual pattern sensed may
be corrupted by random noise.

Regression: This task involves predicting a continuous output value based on

input features. Unlike classification, which assigns discrete labels, regression
aims to estimate a function that maps input variables to a continuous output.
The performance of regression models is often evaluated using metrics such
as mean squared error, which measures the average squared difference
between predicted and actual values.

Learning from Examples: Both classification and regression typically involve

learning from examples, where a set of patterns with known categories or

Pattern Recognition and Computer Vision NOTES 15

 output values is used to train the model. The effectiveness of these methods
often relies on the quality and quantity of the training data.

Decision Boundaries: In classification, the decision boundary is the surface

that separates different classes in the feature space. The goal is to find an
optimal decision boundary that minimizes classification errors. In regression,
the focus is on fitting a curve or line that best represents the relationship
between input features and the output variable.

Bias and Variance: Both tasks are influenced by the bias-variance tradeoff,
where a model with high bias may underfit the data, while a model with high
variance may overfit. Balancing these aspects is crucial for achieving good
generalization performance.

In summary, classification focuses on assigning discrete labels to input patterns,

while regression aims to predict continuous values. Both tasks involve learning
from examples and require careful consideration of model complexity and
evaluation metrics.

Features, Feature Vectors, and Classifiers

Features, feature vectors, and classifiers are fundamental concepts in pattern
recognition and machine learning. Here are some key points regarding these
concepts:

Features: Features are individual measurable properties or characteristics of

the data being analyzed. They serve as the input variables for classifiers. The
choice of features is crucial, as they should effectively represent the
underlying patterns in the data. For example, in classifying fish, features might
include length, lightness, and width.

Feature Vectors: A feature vector is a representation of an instance in a multi-

dimensional space, where each dimension corresponds to a specific feature.
For instance, if we have two features, lightness and width, a fish can be
represented as a feature vector ( x = [x_1, x_2] ), where ( x_1 ) is the lightness
and ( x_2 ) is the width. The feature vector allows for the visualization and
analysis of data points in a structured manner.

Classifiers: Classifiers are algorithms or models that use feature vectors to

assign labels or categories to instances. They evaluate the evidence

Pattern Recognition and Computer Vision NOTES 16

 presented by the feature vector and make decisions based on learned
patterns from training data. The classifier's goal is to partition the feature
space into regions corresponding to different classes, effectively separating
the data points based on their features.

In summary, features are the individual characteristics of the data, feature vectors
are the structured representation of these features, and classifiers are the models
that utilize these vectors to categorize instances effectively. The success of a
classifier often depends on the quality and relevance of the features selected.

Pre‐processing and feature extraction

Pre-processing and feature extraction are critical steps in the pattern
classification process. Here are some key points regarding these concepts:

Pre-processing: This step involves preparing the raw data for analysis by
simplifying subsequent operations without losing relevant information. For
instance, in image processing, pre-processing might include operations like
segmentation, where different objects (e.g., fish) are isolated from one another
and the background. This helps in reducing noise and improving the reliability
of the feature values measured.

Feature Extraction: The purpose of feature extraction is to reduce the data by

measuring certain properties or features of the patterns. This involves
transforming the raw data into a set of features that can be used for
classification. The features should effectively represent the underlying
patterns while being fewer in number than the original data to avoid
information overload. The choice of features is often domain-dependent,
meaning that a feature extractor suitable for one application (like sorting fish)
may not be effective for another (like identifying fingerprints).

Importance of Features: The features selected should reveal the structural

relationships among the components of the data and express the true model
of the patterns. A good feature extractor can significantly simplify the
classification task, making it easier for the classifier to make accurate
decisions.

In summary, pre-processing prepares the data for analysis by reducing noise and
isolating relevant patterns, while feature extraction focuses on identifying and

Pattern Recognition and Computer Vision NOTES 17

 measuring the most informative properties of the data to facilitate effective
classification.

The curse of dimensionality

The curse of dimensionality refers to various phenomena that arise when
analyzing and organizing data in high-dimensional spaces. Here are some key
points regarding the curse of dimensionality:

Sample Size Requirement: As the dimensionality increases, the volume of the

space increases exponentially, making it necessary to have exponentially
more samples to maintain the same density. For instance, if a certain number
of samples is needed in one dimension, a significantly larger number is
required in higher dimensions to achieve the same density.

Distance Concentration: In high-dimensional spaces, the distance between

points becomes less meaningful. Most points tend to be located near the
edges of the space, and the difference in distances between the nearest and
farthest points diminishes. This phenomenon can lead to difficulties in
clustering and classification tasks, as the concept of "closeness" becomes
less reliable.

Increased Complexity: High-dimensional functions can be much more

complicated than their low-dimensional counterparts, making it harder to
discern patterns and relationships within the data. This complexity can lead to
overfitting, where models become too tailored to the training data and fail to
generalize well to new data.

Computational Challenges: The computational complexity of algorithms often

increases with dimensionality, making it more difficult to process and analyze
high-dimensional data efficiently. This can lead to longer processing times and
higher resource requirements.

Mitigation Strategies: To overcome the curse of dimensionality, techniques

such as dimensionality reduction (e.g., principal component analysis) and
feature selection can be employed. These methods aim to reduce the number
of features while retaining the essential information needed for effective
analysis and classification.

Pattern Recognition and Computer Vision NOTES 18

 In summary, the curse of dimensionality presents significant challenges in data
analysis, requiring careful consideration of sample sizes, distance metrics, and
computational strategies to effectively manage high-dimensional data.

Polynomial curve fitting

Polynomial curve fitting is a method used to model the relationship between a set
of data points by fitting a polynomial function to them. Here are some key points
regarding polynomial curve fitting:

Data Points and Noise: In polynomial curve fitting, data points are often
obtained from a true underlying function, which may be corrupted by noise.
For example, data points can be generated by adding zero-mean, independent
noise to a polynomial function, such as a parabola.

Choosing the Polynomial Degree: The degree of the polynomial chosen for
fitting is crucial. A higher-degree polynomial can fit the training data perfectly,
but it may not generalize well to new data. For instance, a tenth-degree
polynomial might fit the data points exactly, but a lower-degree polynomial,
like a second-order function, might provide better predictions for future
samples.

Overfitting and Generalization: Overfitting occurs when the model is too

complex relative to the amount of data available. Reliable interpolation or
extrapolation typically requires that the number of data points exceeds the
number of parameters in the polynomial model. This means that for effective
fitting, the solution should be overdetermined, with more data points than
polynomial coefficients.

Model Simplification: One approach to improve generalization is to start with

a high-order polynomial and then simplify the model by removing the highest-
order terms. This can lead to greater error on the training data but may
improve generalization to unseen data.

In summary, polynomial curve fitting involves selecting an appropriate polynomial

degree to model the relationship between data points while balancing the risk of
overfitting and ensuring that the model generalizes well to new data.

Model complexity

Pattern Recognition and Computer Vision NOTES 19

 Model complexity refers to the intricacy of a model in terms of its structure and
the number of parameters it contains. Here are some key points regarding model
complexity:

Trade-off: There is a fundamental trade-off between model complexity and

performance. A more complex model may fit the training data very well, but it
risks overfitting, which means it may not generalize well to unseen data.
Conversely, a simpler model may not capture all the nuances of the training
data but can perform better on new data.

Overfitting: Overfitting occurs when a model is too complex relative to the

amount of training data available. This can lead to perfect classification of the
training samples but poor performance on novel patterns. It is crucial to adjust
the complexity of the model to avoid this issue.

Regularization: Techniques such as regularization are often employed to

manage model complexity. Regularization adds a penalty for complexity to the
loss function, encouraging simpler models that are less likely to overfit.

Minimum Description Length Principle: This principle suggests that the best
model is one that minimizes the sum of the model's complexity and the
description of the training data given to that model. This approach helps in
selecting models that are not only accurate but also simple.

Algorithmic Complexity: The complexity of a model can also be described in

terms of algorithmic complexity, which quantifies the inherent complexity of a
model based on the length of the shortest program that can describe it.
Simpler models are preferred as they are easier to understand and implement.

Model Selection: Determining the appropriate model complexity is a critical

aspect of model selection. Designers must balance the need for a model that
can explain the data without being overly complex.

In summary, model complexity is a crucial factor in the design of classifiers,

requiring careful consideration to achieve a balance between fitting the training
data and generalizing to new data.

Multivariate non‐linear functions

Multivariate non-linear functions are mathematical functions that involve multiple
variables and exhibit non-linear relationships among them. Here are some key

Pattern Recognition and Computer Vision NOTES 20

 points regarding multivariate non-linear functions:

General Form: A multivariate non-linear function can be expressed in various

forms, often involving combinations of the input variables raised to different
powers or combined in non-linear ways. For example, a quadratic function can
be represented as ( g(x) = a_1 + a_2 x + a_3 x^2 ), where ( x ) is a variable and
( a_1, a_2, a_3 ) are coefficients.

Mapping and Decision Regions: The mapping from input variables to output
can create complex decision regions in the feature space. For instance, when
a linear decision boundary is applied to a non-linear function, the resulting
decision regions can be non-convex and intricate, making classification tasks
more challenging.

Expressive Power: Non-linear functions, particularly when implemented in

neural networks, can approximate any continuous function from input to
output, given a sufficient number of hidden units and appropriate non-
linearities. This capability allows for the modeling of complex relationships in
data.

Applications: Multivariate non-linear functions are widely used in various

fields, including machine learning, where they help in creating models that can
capture intricate patterns in high-dimensional data. Techniques such as
generalized additive models and multivariate adaptive regression splines
(MARS) utilize these functions to enhance predictive performance.

In summary, multivariate non-linear functions are essential for modeling complex

relationships in data, providing the flexibility needed to capture non-linear patterns
effectively.

Bayes' theorem
Bayes' theorem is a fundamental concept in probability theory and statistics that
describes how to update the probability of a hypothesis based on new evidence.
Here are some key points regarding Bayes' theorem:

Formula: Bayes' theorem can be expressed mathematically as:

P(ω_j | x) = (p(x | ω_j) * P(ω_j)) / p(x)
where:

Pattern Recognition and Computer Vision NOTES 21

 ( P(\omega_j | x) ) is the posterior probability of the hypothesis ( \omega_j
) given the evidence ( x ).

( p(x | \omega_j) ) is the likelihood of observing the evidence ( x ) given

that the hypothesis ( \omega_j ) is true.

( P(\omega_j) ) is the prior probability of the hypothesis ( \omega_j )

before observing the evidence.

( p(x) ) is the marginal likelihood of the evidence, which can be computed

as:

p(x) = ∑_j p(x | ω_j) * P(ω_j)

Interpretation: The theorem allows us to invert the conditional probabilities,

turning the likelihood ( p(x | \omega_j) ) into the posterior probability (
P(\omega_j | x) ). This is particularly useful when we want to determine the
probability of a cause (hypothesis) given an observed effect (evidence).

Application: Bayes' theorem is widely used in various fields, including

machine learning, medical diagnosis, and decision-making, as it provides a
systematic way to update beliefs in light of new data.

Prior and Posterior: The prior probability reflects our initial belief about the
hypothesis before seeing the evidence, while the posterior probability
represents our updated belief after considering the evidence. The theorem
emphasizes the importance of both prior knowledge and new evidence in
shaping our understanding of uncertainty.

In summary, Bayes' theorem provides a powerful framework for reasoning about

uncertainty and making informed decisions based on evidence, allowing for the
integration of prior knowledge with new observations.

Decision boundaries
Decision boundaries are surfaces in the feature space that separate different
classes in a classification problem. Here are some key points regarding decision
boundaries:

Definition: A decision boundary is defined as the surface where the classifier

is indifferent between two or more classes. It is the locus of points where the

Pattern Recognition and Computer Vision NOTES 22

 discriminant functions for different classes are equal, leading to ties in
classification decisions.

Types of Decision Boundaries: The nature of the decision boundary can vary
based on the underlying model:

Linear Decision Boundaries: For linear classifiers, the decision boundary

is a hyperplane. This is typically represented by a linear equation in the
feature space.

Non-linear Decision Boundaries: More complex models, such as those

using quadratic discriminant functions, can produce non-linear decision
boundaries, such as hyperquadrics (e.g., ellipsoids or hyperboloids).

Influence of Features: The shape and position of the decision boundary are
influenced by the features used in the model. For instance, if the features are
independent and normally distributed, the decision boundary can take on
specific forms based on the means and covariances of the distributions.

Visualization: In two-dimensional feature spaces, decision boundaries can

often be visualized as curves or lines that separate different regions
corresponding to different classes. For example, in a Gaussian distribution
scenario, the decision boundary might consist of hyperbolas.

Ambiguous Regions: There can be regions where the decision boundary is

ambiguous, meaning that the classifier may not confidently assign a class
label to points in those areas.

In summary, decision boundaries are critical in classification tasks as they define

how different classes are separated in the feature space, and their characteristics
depend on the underlying statistical properties of the data and the chosen
classification model.

Parametric methods
Parametric methods are statistical techniques that assume a specific form for the
underlying probability distribution of the data. Here are some key points regarding
parametric methods:

Assumption of Distribution: Parametric methods rely on the assumption that

the data follows a known distribution, such as normal, exponential, or binomial

Pattern Recognition and Computer Vision NOTES 23

 distributions. This assumption simplifies the problem of estimating the
underlying density functions.

Parameter Estimation: The main goal of parametric methods is to estimate the

parameters of the assumed distribution. For example, in the case of a normal
distribution, the parameters would be the mean (( \mu )) and the variance ((
\sigma^2 )). The estimation can be performed using techniques like maximum
likelihood estimation or Bayesian estimation.

Advantages: One of the primary advantages of parametric methods is their

computational efficiency. Since they rely on a fixed number of parameters,
they often require less data to make reliable inferences compared to
nonparametric methods, which may need a larger sample size to estimate the
underlying distribution accurately.

Limitations: However, parametric methods can be limited by their

assumptions. If the true distribution of the data does not match the assumed
model, the results can be misleading. For instance, classical parametric forms
are often unimodal, while many real-world problems involve multimodal
distributions.

Model Selection: In practice, it is crucial to validate the assumptions of the

parametric model. Techniques such as cross-validation can be used to assess
the model's performance and ensure that it generalizes well to new data.

In summary, parametric methods provide a structured approach to statistical

modeling by assuming a specific form for the data distribution and estimating the
associated parameters, but they require careful consideration of the underlying
assumptions to avoid potential pitfalls.

Sequential parameter estimation

Sequential parameter estimation is a method used in statistics and machine
learning to update the estimates of parameters as new data becomes available.
Here are some key points regarding sequential parameter estimation:

Recursive Bayes Approach: In sequential parameter estimation, the posterior

density of the parameters is updated recursively as new samples are
observed. The relationship is given by the equation:
p(θ | D_n) = (p(x_n | θ) * p(θ | D_{n-1})) / ∫ p(x_n | θ) * p(θ | D_{n-1}) dθ

Pattern Recognition and Computer Vision NOTES 24

 This shows how the posterior density at step ( n ) depends on the likelihood of
the new data point ( x_n ) and the previous posterior density ( p(\theta|D_{n-1})
).

Incremental Learning: This method is also referred to as incremental or online

learning, where the model continuously learns and updates as new data points
are collected. The sequence of posterior densities converges to a Dirac delta
function centered around the true parameter value as more samples are
observed, indicating that the estimates become more precise.

Sufficient Statistics: In some cases, it is not necessary to retain all previous

data points; instead, only a few parameters known as sufficient statistics can
encapsulate all the information needed to update the estimates. This can
simplify the computation significantly.

Applications: Sequential parameter estimation is particularly useful in

scenarios where data arrives in streams or where it is impractical to store all
past data. It allows for real-time updates to the model, making it suitable for
dynamic environments.

In summary, sequential parameter estimation is a powerful technique that enables

continuous updating of parameter estimates using new data, leveraging recursive
Bayesian methods and sufficient statistics for efficient computation.

Linear discriminant functions

Linear discriminant functions are mathematical models used in pattern recognition
and classification tasks. Here are some key points regarding linear discriminant
functions:

Definition: A linear discriminant function can be expressed as ( g(x) = w^T x +

w_0 ), where ( w ) is the weight vector and ( w_0 ) is the bias or threshold
weight. This function is a linear combination of the input features ( x ).

Decision Rule: The decision rule for a two-category linear classifier is to

assign the input ( x ) to class ( \omega_1 ) if ( g(x) > 0 ) and to class ( \omega_2
) if ( g(x) < 0 ). If ( g(x) = 0 ), the assignment can be left undefined.

Geometric Interpretation: The linear discriminant function divides the feature

space into two half-spaces, with the decision boundary defined by the

Pattern Recognition and Computer Vision NOTES 25

 hyperplane where ( g(x) = 0 ). The orientation of this hyperplane is determined
by the normal vector ( w ).

Multicategory Classification: For multicategory problems, multiple linear

discriminant functions can be defined. One approach is to create ( c ) linear
discriminant functions, where each function separates points assigned to one
class from those not assigned to that class. This can lead to complex decision
regions.

Properties: Linear discriminant functions are relatively easy to compute and

can be optimal under certain conditions, such as when the underlying
distributions are cooperative (e.g., Gaussian distributions with equal
covariance). However, they may not perform well in cases with complex,
multimodal distributions unless appropriate nonlinear mappings are applied.

Training: The process of finding the optimal linear discriminant function often
involves minimizing a criterion function, such as the sample risk or training
error, which measures the average loss incurred in classifying the training
samples.

In summary, linear discriminant functions are a fundamental tool in classification

tasks, providing a straightforward method for separating classes in the feature
space through linear decision boundaries. Their effectiveness can depend on the
underlying data distribution and the choice of features.

Feed‐forward network
A feedforward network is a type of artificial neural network where the connections
between the nodes do not form cycles. Here are some key points regarding
feedforward networks:

Structure: A typical feedforward network consists of an input layer, one or

more hidden layers, and an output layer. Each layer is made up of units (or
neurons) that are interconnected by modifiable weights. The input layer
receives the feature vector, while the output layer produces the classification
results.

Feedforward Operation: During the feedforward operation, input patterns are

presented to the network, and each unit computes its activation based on the

Pattern Recognition and Computer Vision NOTES 26

 weighted sum of its inputs. The output units then emit signals that serve as
discriminant functions for classification.

Activation Functions: Each unit in the hidden and output layers typically
applies a non-linear activation function to its net input, which is the weighted
sum of its inputs. Common activation functions include sigmoid and ReLU.

Learning Process: The learning process in feedforward networks often

involves the backpropagation algorithm, which adjusts the weights based on
the error between the predicted output and the actual target values. This
process is repeated for multiple iterations to minimize the error.

Expressive Power: Feedforward networks, especially those with multiple

hidden layers, have the capability to approximate any continuous function,
making them powerful tools for various applications in pattern recognition and
classification.

In summary, feedforward networks are structured neural networks that process

information in one direction, from input to output, and are trained using algorithms
like backpropagation to optimize their performance in classification tasks.

Pattern Recognition and Computer Vision NOTES 27