Unit 1: Basics of Pattern Recognition

• Definition: Pattern Recognition is the process of classifying input data into objects or
categories based on key features.

• Applications: Handwriting recognition, fingerprint identification, speech recognition,

• Types of Pattern Recognition:

o Statistical: Based on statistical information (e.g., Bayesian classifier).

o Syntactic: Based on grammar rules and structure (e.g., image pattern from
pixels).

o Neural: Uses models inspired by the human brain (e.g., neural networks).

• Learning Approaches:

o Supervised Learning: Training data with known labels is used to build models.

o Unsupervised Learning: Only input data is available (no labels); goal is to find
structure (e.g., clustering).

o Reinforcement Learning: Learns through feedback and rewards.

• System Architecture:

1. Sensing: Device captures data.

2. Preprocessing: Normalize, denoise, enhance features.

3. Feature Extraction: Extract meaningful and discriminative attributes (e.g., edge,

4. Classification: Assign input to the most probable class.

Unit 2: Bayesian Decision Theory

Classifiers, Discriminant Functions, Decision Surfaces

• Classifier: Algorithm that assigns an input to one of the predefined classes.

• Discriminant Function: Maps a feature vector to a value; the class with the highest
value is chosen.

o gi(x)>gj(x)g_i(x) > g_j(x)gi(x)>gj(x) implies class i is preferred.

• Decision Surface: A boundary in feature space that separates different classes (e.g.,
line for 2D data).
 Normal Density and Discriminant Functions

• Normal Distribution: Continuous probability distribution defined by mean μ\muμ

• PDF (1D):

p(x)=12πσ2e−(x−μ)22σ2p(x) = \frac{1}{\sqrt{2\pi\sigma^2}} e^{-\frac{(x -

• In classification, we compute likelihood of feature vector belonging to a class using

Discrete Features

• When input features are discrete (e.g., binary or categorical).

• Probability mass function is used.

• Bayes theorem:

P(ωi∣x)=P(x∣ωi)P(ωi)P(x)P(\omega_i|x) = \frac{P(x|\omega_i)P(\omega_i)}{P(x)}P(ωi
∣x)=P(x)P(x∣ωi)P(ωi)

• Maximum a Posteriori (MAP) classifier selects class with highest posterior

Unit 3: Parameter Estimation Methods

Maximum Likelihood Estimation (MLE)

• Estimate parameters such that the likelihood of observed data is maximized.

• For Gaussian:

o Mean: μ=1n∑xi\mu = \frac{1}{n} \sum x_iμ=n1∑xi

o Variance: σ2=1n∑(xi−μ)2\sigma^2 = \frac{1}{n} \sum (x_i - \mu)^2σ2=n1∑(xi

Gaussian Mixture Models (GMM)

• Models data as a mixture of multiple Gaussians.

• Each component represents a cluster or subpopulation.

• Useful in real-world problems like speaker identification.

Expectation-Maximization (EM)
• Algorithm for parameter estimation in models like GMM.

1. E-step: Estimate hidden variables (posterior probabilities).

2. M-step: Update parameters to maximize expected likelihood.

Bayesian Estimation

• Incorporates prior knowledge into parameter estimation.

• Uses Bayes’ theorem to update belief about parameters after seeing data.

• More robust when data is sparse or uncertain.

Unit 4: Hidden Markov Models (HMMs)

Discrete HMMs

• Markov process where state is hidden but generates observable symbols.

• Defined by:

o A: state transition probabilities

o B: emission probabilities (output given state)

o π: initial state distribution

• Applications: Speech recognition, bioinformatics.

Continuous HMMs

• Observations are continuous, not discrete.

• Emission probability modeled using Gaussian or GMM.

Unit 5: Dimension Reduction Methods

Fisher’s Linear Discriminant

• Projects data to a lower dimension to maximize class separability.

• Maximizes the ratio of between-class scatter to within-class scatter.

Principal Component Analysis (PCA)

• Projects data to new axes (principal components) to capture maximum variance.

• Steps:
 1. Mean normalization

2. Compute covariance matrix

3. Compute eigenvectors

4. Select top k components

Parzen Window

• Non-parametric way to estimate the PDF of a random variable.

• Uses kernels (e.g., Gaussian) placed on each data point.

• Good for visualizing data distribution without assuming a specific shape.

K-Nearest Neighbours (KNN)

• For a new point, find the k closest training samples and vote for the class.

• Simple and effective; sensitive to the choice of k and distance metric.

Unit 6: Non-Parametric Techniques for Density Estimation

• No assumption about the form of the distribution.

• Techniques:

o Histogram: Divide data range into bins.

o KNN Density: Density at a point is inverse of volume needed to enclose k

o Parzen Window: Smooth version using kernel functions.

• Useful when underlying distribution is unknown or multimodal.

Unit 7: Linear Discriminant Function Based Classifier

Perceptron

• Linear binary classifier: y=wTx+by = w^T x + by=wTx+b

• Learning rule: Adjust weights based on classification error.

• Converges if data is linearly separable.

Support Vector Machine (SVM)

• Finds optimal hyperplane that maximizes margin between classes.

• Solves convex optimization problem.

Unit 8: Non-Metric Methods for Pattern Classification

Non-Numeric (Nominal) Data

• Feature values are labels (e.g., male/female).

• Require non-metric classifiers (not based on distances).

Decision Trees

• Recursive structure of nodes and branches.

• Internal nodes → feature tests.

• Leaves → class labels.

• Split criteria: Information gain, Gini index, Chi-square.

• Advantages:

o Easy to understand

o Handles both categorical and numerical data

• Disadvantage: Prone to overfitting (can be solved using pruning).

Unit 9: Unsupervised Learning and Clustering

Criterion Functions

• Measure quality of clustering:

o Intra-cluster distance: Should be small.

o Inter-cluster distance: Should be large.

o Silhouette score, SSE, etc.

Clustering Algorithms:

• K-means:

1. Choose k centers randomly

2. Assign points to nearest center

4. Repeat until convergence

• Hierarchical Clustering:

o Agglomerative: Merge closest clusters (bottom-up)

o Divisive: Split until only single clusters remain (top-down)

• Other Methods:

o DBSCAN: Density-based clustering, handles noise

o Mean-Shift: Moves points to high-density regions