Pattern Recognition

(ECECE26)
Practical File

Department of
Electronics and Communication Engineering,
Netaji Subhas University of Technology

Name: Manas Pratim Das

Roll No.: 2022UEC2697
Class: ECE-3
Semester: 6 (2024-25)
 LIST OF EXPERIMENTS

S.NO. DATE EXPERIMENT SIGN.

1 21/01/2025 To write a MATLAB/Python function
that computes the value of the Gaus-
sian distribution N (m, S), at a given
vector x. Hence, plot the effect of vary-
ing mean and variance to the normal
distributions.
2 28/01/2025 To write a MATLAB/Python function
that will take as inputs: (a) the mean
vectors, (b) the covariance matrices of
the class distributions of a c-class prob-
lem, (c) the a priori probabilities of the
c classes, and (d) a matrix X contain-
ing column vectors that stem from the
above classes. It will give as output an
N-dimensional vector whose ith compo-
nent contains the class where the corre-
sponding vector is assigned, according
to the Bayesian classification rule.
3 04/02/2025 To write a MATLAB/Python function
that will take as inputs: (a) the mean
vectors, and (b) a matrix X contain-
ing column vectors that stem from the
above classes. It will give as output an
N-dimensional vector whose ith compo-
nent contains the class where the cor-
responding vector is assigned, accord-
ing to the minimum Euclidean distance
classifier.
S.NO. DATE EXPERIMENT SIGN.
4 11/02/2025 To write a MATLAB/Python function
that will take as inputs: (a) the mean
vectors,(b) the covariance matrix of the
class distributions of a c-class prob-
lem, and (c) a matrix X containing col-
umn vectors that stem from the above
classes. It will give as output an N-
dimensional vector whose ith compo-
nent contains the class where the cor-
responding vector is assigned according
to the minimum Mahalanobis distance
classifier.
5 18/02/2025 To write a MATLAB/Python function
that takes as inputs: (a) a set of N1
vectors packed as columns of a matrix
Z,(b) an N1 -dimensional vector con-
taining the classes where each vector in
Z belongs, (c) the value for the parame-
ter k of the classifier,(d) a set of N vec-
tors packed as columns in the matrix
X. It returns an N-dimensional vector
whose ith component contains the class
where the corresponding vector of X
is assigned, according to the k-nearest
neighbour classifier.
6 11/03/2025 To write a MATLAB/Python function
that will take as inputs: (a) an N-
dimensional vector, each component of
which contains the class where the cor-
responding data vector belongs and (b)
a similar N-dimensional vector each
component of which contains the class
where the corresponding data vector is
assigned from a certain classifier. Its
output will be the percentage of the
places where the two vectors differ (i.e.,
the classification error of the classifier).
S.NO. DATE EXPERIMENT SIGN.
7 18/03/2025 To write a MATLAB/Python function
for the perceptron algorithm. This will
take as inputs: (a) a matrix X contain-
ing N1 -dimensional column vectors, (b)
an N-dimensional row vector y, whose
ith component contains the class (-1 or
+1) where the corresponding vector be-
longs, and (c) an initial value vector
wini for the parameter vector. It re-
turns the estimated parameter vector.
8 01/04/2025 To design a three-layer FFN, using gra-
dient descent to perform the {x1 ,x2 }
→ {y1 ,y2 } mapping. The activation
function for all the nodes is the hy-
perbolic tangent one. For training,
one may select one of the following
algorithms: a) the standard gradient
descent backpropagation algorithm, (
b) the backpropagation algorithm with
momentum , and (c) the backpropaga-
tion algorithm with adaptive learning
rate.
9 08/04/2025 To write MATLAB/Python function
to compute the principal components
of the covariance matrix of an l×N-
dimensional data matrix X as well as
the corresponding variances. Hence,
write a MATLAB/Python function
that evaluates the performance of the
PCA method when applied on a data
matrix X.
 EXPERIMENT-1
Aim:
To write a MATLAB/Python function that computes the value of the Gaus-
sian distribution N (m, S), at a given vector x. Hence, plot the effect of
varying mean and variance to the normal distributions.

Theory:
The Gaussian distribution, also known as the normal distribution, is a fun-
damental probability distribution in statistics and machine learning. It is
widely used due to its natural occurrence in many real-world phenomena.
The probability density function (PDF) of a Gaussian distribution for a ran-
dom variable x is given by:
 
1 1 T −1
f (x) = p exp − (x − m) S (x − m) (1)
(2π)d |S| 2
where:

• x is the d-dimensional input vector,

• m is the mean vector,

• S is the covariance matrix (for multivariate Gaussian),

• |S| denotes the determinant of S,

• S −1 is the inverse of the covariance matrix,

• π is the mathematical constant approximately equal to 3.14159.

For the univariate (one-dimensional) case, the probability density func-

tion simplifies to:

(x − µ)2
 
1
f (x) = √ exp − (2)
σ 2π 2σ 2
where:

• µ (mean) determines the center of the distribution,

• σ 2 (variance) determines the spread.

 The shape of the normal distribution is primarily affected by:

• Mean (µ): Determines the central location of the peak. Changing µ

shifts the distribution along the x-axis without altering its shape.

• Variance (σ 2 ): Determines the spread of the distribution. A higher

variance leads to a wider and flatter distribution, while a lower variance
results in a steeper and narrower distribution.

These effects can be visualized by plotting multiple normal distributions

with varying mean and variance.

Code:
1 import numpy as np
2 import matplotlib . pyplot as plt
3

4 def gaussian (x , mean , variance ) :

5 sigma = np . sqrt ( variance )
6 coefficient = 1 / ( np . sqrt (2 * np . pi * variance ) )
7 exponent = np . exp ( -(( x - mean ) ** 2) / (2 * variance ) )
8 return coefficient * exponent
9

10 x = np . linspace ( -10 , 10 , 400)

11 means = [0 , 0 , -2]
12 variances = [1 , 4 , 1]
13 labels = [ " mean =0 , var =1 " , " mean =0 , var =4 " , " mean = -2 , var =1 " ]
14
15 plt . figure ( figsize =(8 , 6) )
16 for mean , variance , label in zip ( means , variances , labels ) :
17 y = gaussian (x , mean , variance )
18 plt . plot (x , y , label = label )
19
20 plt . xlabel ( " x " )
21 plt . ylabel ( " Gaussian Distribution " )
22 plt . title ( " Effect of Varying Mean and Variance on Normal
Distributions " )
23 plt . legend ()
24 plt . grid ()
25 plt . show ()
Observation:

Figure 1: Gaussian PDF for varying mean and variance

Result:
Successfully plotted the Gaussian distribution and studied the effect of mean
and variance.
 EXPERIMENT-2
Aim:
To write a MATLAB/Python function that will take as inputs: (a) the mean
vectors, (b) the covariance matrices of the class distributions of a c-class
problem, (c) the a priori probabilities of the c classes, and (d) a matrix X
containing column vectors that stem from the above classes. It will give as
output an N-dimensional vector whose ith component contains the class where
the corresponding vector is assigned, according to the Bayesian classification
rule.

Theory:
Bayesian classification is a probabilistic approach to classifying a given set
of input vectors based on prior knowledge of class distributions. It is derived
from Bayes’ theorem, which provides a mathematical framework for updating
prior beliefs with observed data. The goal of this experiment is to implement
a classifier that assigns each input vector to one of the given c classes based
on the Bayesian decision rule. For a given feature vector x, the probability
of belonging to class Cj is given by:

P (x | Cj )P (Cj )
P (Cj | x) = (3)
P (x)
where:

• P (Cj | x) is the posterior probability of class Cj given the observa-

tion x.

• P (x | Cj ) is the likelihood, i.e., the probability density function of

class Cj .

• P (Cj ) is the prior probability of class Cj .

• P (x) is the evidence (a normalizing constant ensuring that probabil-

ities sum to 1).

In practice, we classify x into the class Cj that maximizes the posterior

probability:
 Code:
1 import numpy as np
2 import matplotlib . pyplot as plt
3 from scipy . stats import m ul t i va r i at e _ no r m al
4

5 def b ay e s i an _ c la s s if i e r ( mean_vectors , cov_matrices , priors , X

):
6 c = len ( mean_vectors ) # Number of classes
7 N = X . shape [1] # Number of input vectors
8 posterior_probs = np . zeros (( c , N ) )
9

10 # Compute posterior probabilities for each class

11 for i in range ( c ) :
12 mean = mean_vectors [ i ]
13 cov = cov_matrices [ i ]
14 prior = priors [ i ]
15

16 # Compute likelihood using multivariate normal

distribution
17 likelihood = m u l ti v a ri a t e_ n o rm a l . pdf ( X .T , mean = mean ,
cov = cov )
18 posterior_probs [i , :] = prior * likelihood
19

20 # Assign each vector to the class with the highest

posterior probability
21 return np . argmax ( posterior_probs , axis =0)
22
23 def v i s u a l i z e _ c l a s s i f i c a t i o n ( mean_vectors , cov_matrices , X ,
class_labels ) :
24 colors = [ ’ red ’ , ’ blue ’]
25 markers = [ ’o ’ , ’s ’]
26
27 # Create a grid of points for decision boundary
visualization
28 x_min , x_max = X [0 , :]. min () - 1 , X [0 , :]. max () + 1
29 y_min , y_max = X [1 , :]. min () - 1 , X [1 , :]. max () + 1
30 xx , yy = np . meshgrid ( np . linspace ( x_min , x_max , 100) , np .
linspace ( y_min , y_max , 100) )
31 grid_points = np . c_ [ xx . ravel () , yy . ravel () ]. T
32
33 # Get class predictions for the grid
34 grid_labels = ba y e si a n _c l a ss i f ie r ( mean_vectors ,
cov_matrices , priors , grid_points )
35 grid_labels = grid_labels . reshape ( xx . shape )
36
37 # Plot decision boundary
38 plt . contourf ( xx , yy , grid_labels , alpha =0.3 , cmap = ’
coolwarm ’)
39

40 # Plot data points

41 for i , label in enumerate ( np . unique ( class_labels ) ) :
42 plt . scatter ( X [0 , class_labels == label ] , X [1 ,
class_labels == label ] ,
43 color = colors [ i ] , marker = markers [ i ] , label
= f ’ Class { label } ’)
44
45 # Plot class means
46 for mean , color in zip ( mean_vectors , colors ) :
47 plt . scatter (* mean , color = color , marker = ’x ’ , s =100 ,
linewidth =3 , label = ’ Mean ’)
48

49 plt . xlabel ( ’ X1 ’)
50 plt . ylabel ( ’ X2 ’)
51 plt . legend ()
52 plt . title ( ’ Bayesian Classification and Decision Boundary ’
)
53 plt . show ()
54
55 # Define means , covariance matrices , and priors
56 mean_vectors = [ np . array ([0 , 0]) , np . array ([3 , 3]) ]
57 cov_matrices = [ np . array ([[1 , 0.5] , [0.5 , 1]]) , np . array ([[1 ,
-0.5] , [ -0.5 , 1]]) ]
58 priors = [0.5 , 0.5]
59
60 # Example input vectors
61 X = np . array ([[1 , 2 , 3] , [1 , 2 , 3]])
62
63 # Perform classification
64 class_labels = b ay e s ia n _ cl a s si f i er ( mean_vectors , cov_matrices
, priors , X )
65 print ( " Class assignments : " , class_labels )
66
67 # Visualize classification
68 v i s u a l i z e _ c l a s s i f i c a t i o n ( mean_vectors , cov_matrices , X ,
class_labels )

Observation:
Class assignments: [0 1 1]
 Figure 2: Gaussian PDF for varying mean and variance

Result:
Successfully understood the Bayesian classification rule.
 EXPERIMENT-3
Aim:
To write a MATLAB/Python function that will take as inputs: (a) the mean
vectors, and (b) a matrix X containing column vectors that stem from the
above classes. It will give as output an N-dimensional vector whose ith com-
ponent contains the class where the corresponding vector is assigned, accord-
ing to the minimum Euclidean distance classifier.

Theory:
The minimum Euclidean distance classifier assigns an input vector to the
class whose mean vector is closest to it in terms of Euclidean distance.
The Euclidean distance between two points x = (x1 , x2 , . . . , xn ) and m =
(m1 , m2 , . . . , mn ) in an n-dimensional space is given by:
v
u n
uX
d(x, m) = t (xi − mi )2 (4)
i=1

In this experiment, each vector in the input matrix X is compared with

the given mean vectors using the Euclidean distance formula. The vector is
then assigned to the class corresponding to the mean vector with the smallest
Euclidean distance.

Code:
1 import numpy as np
2 import matplotlib . pyplot as plt
3
4 def c l a s s i f y _ m i n _ e u c l i d e a n ( mean_vectors , X ) :
5 # Compute the squared Euclidean distances between each
vector in X and each mean vector
6 distances = np . linalg . norm ( X [: , : , np . newaxis ] -
mean_vectors [: , np . newaxis , :] , axis =0)
7 # Assign each vector in X to the class with the minimum
distance
8 return np . argmin ( distances , axis =1)
9
10 # Define mean vectors and data points
11 mean_vectors = np . array ([[1 , 4] , [1 , 4]]) # Two mean vectors
( for two classes )
12 X = np . array ([[2 , 3 , 5] , [2 , 3 , 5]]) # Three vectors to
classify
13
14 # Classify points
15 classifications = c l a s s i f y _ m i n _ e u c l i d e a n ( mean_vectors , X )
16
17 # Visualization
18 plt . figure ( figsize =(6 , 6) )
19 colors = [ ’ red ’ , ’ blue ’] # Colors for two classes
20
21 # Plot mean vectors
22 plt . scatter ( mean_vectors [0] , mean_vectors [1] , c = ’ black ’ ,
marker = ’x ’ , s =100 , label = ’ Mean Vectors ’)
23
24 # Plot data points , color - coded by classification
25 unique_classes = np . unique ( classifications )
26 for cls in unique_classes :
27 indices = np . where ( classifications == cls )
28 plt . scatter ( X [0 , indices ] , X [1 , indices ] , c = colors [ cls ] ,
label = f ’ Class { cls } ’)
29
30 plt . xlabel ( ’ Feature 1 ’)
31 plt . ylabel ( ’ Feature 2 ’)
32 plt . legend ()
33 plt . title ( ’ Data Classification using Minimum Euclidean
Distance ’)
34 plt . grid ()
35 plt . show ()
36

37 print ( " Class assignments : " , classifications )

Observation:
Class assignments: [0 1 1]
 Figure 3: Gaussian PDF for varying mean and variance

Result:
Successfully implemented and understood the minimum Euclidean distance
classifier.
 EXPERIMENT-4
Aim:
To write a MATLAB/Python function that will take as inputs: (a) the mean
vectors, (b) the covariance matrix of the class distributions of a c-class prob-
lem, and (c) a matrix X containing column vectors that stem from the above
classes. It will give as output an N-dimensional vector whose ith component
contains the class where the corresponding vector is assigned according to
the minimum Mahalanobis distance classifier.

Theory:
The Mahalanobis distance is a measure of the distance between a point and a
distribution, taking into account the correlations of the dataset. It is defined
as:
p
dM (x, m) = (x − m)T S−1 (x − m) (5)
where:
• x is the input vector,

• m is the mean vector of the class,

• S−1 is the inverse of the covariance matrix.

Unlike the Euclidean distance, the Mahalanobis distance accounts for the
variance and correlations in the dataset, making it a more reliable metric for
classification when features are correlated.

Code:
1 import numpy as np
2 import matplotlib . pyplot as plt
3 from scipy . spatial . distance import mahalanobis
4
5 # Mahalanobis classifier function
6 def m a h a l a n o b i s _ c l a s s i f i e r ( mean_vectors , cov_matrix , X ) :
7 c , N = mean_vectors . shape # Number of classes and
feature dimension
8 M = X . shape [1] # Number of samples to classify
 9

10 inv_cov = np . linalg . inv ( cov_matrix ) # Precompute inverse

covariance matrix
11 classifications = np . zeros (M , dtype = int )
12
13 for i in range ( M ) :
14 x = X [: , i ] # Extract the i - th sample
15 distances = [ mahalanobis (x , mean_vectors [ j ] , inv_cov )
for j in range ( c ) ]
16 classifications [ i ] = np . argmin ( distances ) # Assign
to class with minimum distance
17
18 return classifications
19
20 # Define mean vectors and covariance matrix
21 mean_vectors = np . array ([[1 , 2] , [4 , 5]]) # Two - class
problem in 2 D
22 cov_matrix = np . array ([[1 , 0.5] , [0.5 , 2]]) # Covariance
matrix
23 X = np . array ([[1 , 3 , 5] , [2 , 4 , 6]]) # Three sample vectors
to classify
24
25 # Classify samples
26 assigned_classes = m a h a l a n o b i s _ c l a s s i f i e r ( mean_vectors ,
cov_matrix , X )
27 print ( " Assigned Classes : " , assigned_classes )
28
29 # Visualization
30 plt . figure ( figsize =(8 , 6) )
31 colors = [ ’ blue ’ , ’ red ’]
32 markers = [ ’o ’ , ’s ’]
33
34 # Plot mean vectors
35 for i , mean in enumerate ( mean_vectors ) :
36 plt . scatter ( mean [0] , mean [1] , c = colors [ i ] , marker = ’X ’ , s
=200 , label = f ’ Class { i } Mean ’)
37
38 # Plot sample points
39 for i in range ( X . shape [1]) :
40 plt . scatter ( X [0 , i ] , X [1 , i ] , c = colors [ assigned_classes [ i
]] , marker = markers [ assigned_classes [ i ]] , s =100)
41 plt . text ( X [0 , i ] + 0.1 , X [1 , i ] + 0.1 , f ’ Sample { i } ’ ,
fontsize =12)
42
43 # Set plot labels and legend
44 plt . xlabel ( ’ Feature 1 ’)
45 plt . ylabel ( ’ Feature 2 ’)
46 plt . title ( ’ Mahalanobis Classifier Visualization ’)
47 plt . legend ()
48 plt . grid ()
49 plt . show ()

Observation:
Assigned Classes: [0 1 1]

Figure 4: Gaussian PDF for varying mean and variance

Result:
Successfully implemented and understood the minimum Mahalanobis dis-
tance classifier.
 EXPERIMENT-5
Aim:
To write a MATLAB/Python function that takes as inputs: (a) a set of
N1 vectors packed as columns of a matrix Z, (b) an N1 -dimensional vector
containing the classes where each vector in Z belongs, (c) the value for the
parameter k of the classifier, (d) a set of N vectors packed as columns in the
matrix X. It returns an N-dimensional vector whose ith component contains
the class where the corresponding vector of X is assigned, according to the
k-nearest neighbour classifier.

Theory:
The k-Nearest Neighbors (k-NN) classifier is a simple, non-parametric algo-
rithm that assigns a class to a given input vector based on the majority class
among its k-nearest neighbors in the feature space. The Euclidean distance
is typically used to determine the closeness of neighbors. The classification
process involves:

• Computing the distance between the test sample and all training sam-
ples.

• Selecting the k closest training samples.

• Assigning the most frequent class among these k neighbors to the test
sample.

This approach is particularly effective in pattern recognition and classifi-

cation problems where decision boundaries are complex and non-linear.

Code:
1 import numpy as np
2 import matplotlib . pyplot as plt
3 from scipy . stats import mode
4
5 def knn_classifier (Z , labels , k , X ) :
6 distances = np . linalg . norm ( Z [: , : , None ] - X [: , None , :] ,
axis =0)
 7 knn_indices = np . argsort ( distances , axis =0) [: k ]
8 knn_labels = labels [ knn_indices ]
9 predicted_labels = mode ( knn_labels , axis =0) . mode . squeeze
()
10 return predicted_labels , knn_indices
11
12 # More data points
13 Z = np . array ([[1.0 , 2.0 , 3.0 , 4.5 , 2.2 , 3.8] , [1.0 , 2.0 , 3.0 ,
1.5 , 3.1 , 2.5]]) # 2 D feature vectors
14 labels = np . array ([0 , 1 , 1 , 0 , 1 , 0]) # Class labels
15 X = np . array ([[2.5 , 1.5 , 3.5] , [2.5 , 1.5 , 2.0]]) # 2 D test
vectors
16 k = 3 # Updated k to 3
17
18 predicted_classes , knn_indices = knn_classifier (Z , labels , k ,
X)
19
20 # Define class colors
21 class_colors = {0: ’ blue ’ , 1: ’ red ’}
22
23 # Plot the training points
24 for label in np . unique ( labels ) :
25 plt . scatter ( Z [0 , labels == label ] , Z [1 , labels == label ] ,
color = class_colors [ label ] , marker = ’o ’ , edgecolors = ’k ’ ,
label = f ’ Class { label } ’)
26
27 # Plot the test points
28 for i , x in enumerate ( X . T ) :
29 plt . scatter ( x [0] , x [1] , color = ’ gold ’ , marker = ’s ’ ,
edgecolors = ’k ’ , label = f ’ Test Point { i +1} ( Pred : {
pre dicted _class es [ i ]}) ’)
30
31 # Highlight the nearest neighbors
32 for i , x in enumerate ( X . T ) :
33 neighbors = knn_indices [: , i ]
34 for neighbor in neighbors :
35 plt . plot ([ Z [0 , neighbor ] , x [0]] , [ Z [1 , neighbor ] , x
[1]] , ’k - - ’ , alpha =0.6 , label = ’ Nearest Neighbor ’ if i == 0
and neighbor == neighbors [0] else " " )
36
37 plt . legend ()
38 plt . xlabel ( ’ Feature 1 ’)
39 plt . ylabel ( ’ Feature 2 ’)
40 plt . title ( ’k - NN Classification Visualization ’)
41 plt . show ()
42
43 print ( " Predicted Classes : " , predi cted_c lasses )
Observation:
Predicted Classes: [1 1 0]

Figure 5: Gaussian PDF for varying mean and variance

Result:
Successfully implemented and understood the k-Nearest Neighbors (k-NN)
classifier.
 EXPERIMENT-6
Aim:
To write a MATLAB/Python function that will take as inputs: (a) an N-
dimensional vector, each component of which contains the class where the
corresponding data vector belongs and (b) a similar N-dimensional vector
each component of which contains the class where the corresponding data
vector is assigned from a certain classifier. Its output will be the percentage
of the places where the two vectors differ (i.e., the classification error of the
classifier).

Theory:
Classification error is a key metric in evaluating the performance of a classi-
fier. It quantifies the percentage of instances where the predicted class labels
differ from the true class labels. The classification error can be mathemati-
cally expressed as:
PN
I(yi ̸= ŷi )
E = i=1 × 100 (6)
N
where:

• E is the classification error percentage.

• N is the total number of test samples.

• yi represents the actual class label of the ith sample.

• ŷi represents the predicted class label of the ith sample.

• I(yi ̸= ŷi ) is an indicator function that is 1 if yi ̸= ŷi and 0 otherwise.

In this experiment, we generate a synthetic dataset and use a Support

Vector Machine (SVM) classifier to predict class labels. The classification
error is computed as the proportion of misclassified samples relative to the
total number of samples in the test set.
The classification error is influenced by several factors, including:

• The complexity of the dataset and the separability of the classes.

• The choice of features used for training the classifier.

 • The effectiveness of the classifier and its hyperparameters.

• The presence of noise or overlapping class distributions in the dataset.

A lower classification error indicates that the classifier performs well in
correctly assigning labels to test samples, while a higher error suggests that
the classifier requires further tuning, additional training data, or a more
suitable model selection.

Code:
1 import numpy as np
2 import matplotlib . pyplot as plt
3 from sklearn . svm import SVC
4 from sklearn . model_selection import train_test_split
5 from sklearn . datasets import m ak e _ cl a s si f i ca t i on
6 from sklearn . decomposition import PCA
7
8 def c l a s s i f i c a t i o n _ e r r o r ( true_labels , predicted_labels ) :
9 true_labels = np . array ( true_labels )
10 predicted_labels = np . array ( predicted_labels )
11

12 if true_labels . shape != predicted_labels . shape :

13 raise ValueError ( " Both input vectors must have the
same shape . " )
14
15 mismatches = np . sum ( true_labels != predicted_labels )
16 error_percentage = ( mismatches / len ( true_labels ) ) * 100
17
18 return error_percentage
19
20 # Generate a synthetic dataset
21 X , y = ma k e _ cl a s si f i ca t i on ( n_samples =200 , n_features =5 ,
n_classes =2 , random_state =42)
22
23 # Reduce dimensions for visualization
24 pca = PCA ( n_components =2)
25 X_pca = pca . fit_transform ( X )
26
27 # Split dataset into training and testing sets
28 X_train , X_test , y_train , y_test = train_test_split (X , y ,
test_size =0.2 , random_state =42)
29
30 # Train SVM classifier
31 svm_classifier = SVC ( kernel = ’ linear ’ , random_state =42)
32 svm_classifier . fit ( X_train , y_train )
33
34 # Make predictions on test set
35 y_pred = svm_classifier . predict ( X_test )
36
37 # Compute classification error
38 error = c l a s s i f i c a t i o n _ e r r o r ( y_test , y_pred )
39 print ( f " Classification Error : { error :.2 f }% " )
40

41 # Visualize the dataset

42 plt . figure ( figsize =(10 , 5) )
43
44 # Plot original data
45 plt . subplot (1 , 2 , 1)
46 plt . scatter ( X_pca [: , 0] , X_pca [: , 1] , c =y , cmap = ’ coolwarm ’ ,
edgecolors = ’k ’)
47 plt . title ( " Original Data ( PCA - reduced ) " )
48 plt . xlabel ( " Principal Component 1 " )
49 plt . ylabel ( " Principal Component 2 " )
50
51 # Plot classified data
52 X_test_pca = pca . transform ( X_test )
53 plt . subplot (1 , 2 , 2)
54 plt . scatter ( X_test_pca [: , 0] , X_test_pca [: , 1] , c = y_pred ,
cmap = ’ coolwarm ’ , edgecolors = ’k ’)
55 plt . title ( " SVM Classification Results " )
56 plt . xlabel ( " Principal Component 1 " )
57 plt . ylabel ( " Principal Component 2 " )
58
59 plt . show ()

Observation:
Classification Error: 10.00%
 Figure 6: Gaussian PDF for varying mean and variance

Result:
Successfully implemented the classification error calculation for evaluating
classifier performance.
 EXPERIMENT-7
Aim:
To write a MATLAB/Python function for the perceptron algorithm. This
will take as inputs: (a) a matrix X containing N1 -dimensional column vectors,
(b) an N-dimensional row vector y, whose ith component contains the class
(-1 or +1) where the corresponding vector belongs, and (c) an initial value
vector wini for the parameter vector. It returns the estimated parameter
vector.

Theory:
The perceptron algorithm is a fundamental method in machine learning used
for binary classification of linearly separable data. It is an iterative process
that updates a weight vector to find a linear decision boundary that correctly
separates the given classes. The perceptron learning rule updates the weight
vector whenever a misclassification occurs.
Given a feature matrix X ∈ RN ×d where each row represents a d-dimensional
feature vector, a label vector y ∈ {−1, +1}N containing class labels, and an
initial weight vector wini ∈ Rd , the perceptron algorithm updates the weight
vector w using the following rule:

w = w + y i Xi if yi (w · Xi ) ≤ 0 (7)
where Xi is the ith input vector, yi is the corresponding class label, and
w · Xi is the dot product between the weight vector and the input vector.
The algorithm iterates over all data points and updates w whenever a
misclassified point is encountered. The process continues until no misclassi-
fications occur or a predefined number of iterations is reached.
The perceptron algorithm is guaranteed to converge if the data is linearly
separable. If the data is not linearly separable, the algorithm will continue
indefinitely unless a maximum iteration limit is imposed. The final weight
vector defines a hyperplane that separates the two classes.
In this experiment, the perceptron successfully learns the weight vector
that defines a linear decision boundary separating the input samples based
on their class labels.
 Code:
1 import numpy as np
2 import matplotlib . pyplot as plt
3
4 def plot_data (X , y , w = None ) :
5 # Plot one point for each class in legend
6 plt . scatter ( X [ y == 1][0 , 0] , X [ y == 1][0 , 1] , color = ’ blue
’ , label = ’ Class 1 ’)
7 plt . scatter ( X [ y == -1][0 , 0] , X [ y == -1][0 , 1] , color = ’
red ’ , label = ’ Class -1 ’)
8

9 # Plot all data points without additional legend entries

10 plt . scatter ( X [ y == 1][: , 0] , X [ y == 1][: , 1] , color = ’ blue
’)
11 plt . scatter ( X [ y == -1][: , 0] , X [ y == -1][: , 1] , color = ’
red ’)
12

13 if w is not None :
14 x_min , x_max = min ( X [: , 0]) - 1 , max ( X [: , 0]) + 1
15 x_vals = np . linspace ( x_min , x_max , 100)
16 if w [1] != 0:
17 y_vals = - ( w [0] * x_vals + w [2]) / w [1] #
Corrected decision boundary equation
18 plt . plot ( x_vals , y_vals , ’g - - ’ , label = ’ Decision
Boundary ’)
19
20 plt . xlabel ( ’ Feature 1 ’)
21 plt . ylabel ( ’ Feature 2 ’)
22 plt . legend ()
23 plt . show ()
24
25 def perceptron (X , y , w_ini , max_iter =1000) :
26 w = np . copy ( w_ini )
27 N = X . shape [0]
28

29 for _ in range ( max_iter ) :

30 misclassified = False
31 for i in range ( N ) :
32 if y [ i ] * np . dot (w , X [ i ]) <= 0: #
Mis classi ficati on check
33 w += y [ i ] * X [ i ] # Update rule
34 misclassified = True
35
36 if not misclassified :
37 break # Stop if no m isc la ss if ic at io ns
38
39 return w
40
41 # Feature matrix with bias term
42 X = np . array ([[2 , 3 , 1] , [1 , 1 , 1] , [5 , 1 , 1] , [6 , 4 , 1]])
43 # Class labels
44 y = np . array ([1 , -1 , 1 , 1])
45 # Initial weight vector
46 w_ini = np . zeros ( X . shape [1])
47

48 # Initial visualization
49 plot_data ( X [: , :2] , y )
50
51 # Train perceptron
52 w_final = perceptron (X , y , w_ini )
53 print ( " Estimated parameter vector : " , w_final )
54
55 # Final visualization
56 plot_data ( X [: , :2] , y , w_final )

Observation:
Estimated parameter vector: [ 2 1 -5]

Figure 7: Gaussian PDF for varying mean and variance

Result:
Successfully implemented the perceptron algorithm to compute the estimated
parameter vector.
 EXPERIMENT-8
Aim:
To design a three-layer FFN, using gradient descent to perform the {x1,x2} →
{y1,y2} mapping. The activation function for all the nodes is the hyperbolic
tangent one. For training, one may select one of the following algorithms: (a)
the standard gradient descent backpropagation algorithm, (b) the backprop-
agation algorithm with momentum, and (c) the backpropagation algorithm
with adaptive learning rate.

Theory:
A Feedforward Neural Network (FFN) is a fundamental type of artificial
neural network where connections between the nodes do not form a cycle.
A three-layer FFN consists of an input layer, a hidden layer, and an out-
put layer. The network maps inputs to outputs using a series of weighted
connections and activation functions.
The activation function used for all the nodes in this experiment is the
hyperbolic tangent function (tanh), defined as:

ex − e−x
tanh(x) = (8)
ex + e−x
which produces outputs in the range of (-1,1), allowing better gradient
propagation compared to sigmoid functions.
The network is trained using gradient descent, which updates the weights
to minimize the mean squared error (MSE) between predicted and actual
outputs. The weight updates follow:
∂L
W =W −η (9)
∂W
where η is the learning rate and L is the loss function.
Three training algorithms can be used:

• Standard Gradient Descent: Updates weights in the direction of

the negative gradient.

• Backpropagation with Momentum: Adds a fraction of the pre-

vious update to the current update to accelerate learning and escape
local minima.
 • Backpropagation with Adaptive Learning Rate: Adjusts the
learning rate dynamically based on performance to ensure optimal con-
vergence.

The decision boundary is visualized to assess the network’s performance.

Code:
1 import numpy as np
2 import matplotlib . pyplot as plt
3
4 def tanh ( x ) :
5 return np . tanh ( x )
6
7 def tanh_derivative ( x ) :
8 return 1.0 - np . tanh ( x ) **2
9
10 # Generate training data ( XOR - like pattern with additional
points )
11 X = np . array ([[0 , 0] , [0 , 1] , [1 , 0] , [1 , 1] ,
12 [0.1 , 0] , [0.9 , 1] , [1 , 0.1] , [0 , 0.9] ,
13 [0.2 , 0.8] , [0.8 , 0.2] , [0.3 , 0.7] , [0.7 ,
0.3]])
14

15 y = np . array ([[0 , 0] , [1 , 0] , [1 , 0] , [0 , 1] ,
16 [0 , 0] , [1 , 0] , [1 , 0] , [0 , 1] ,
17 [0 , 1] , [1 , 0] , [0 , 1] , [1 , 0]])
18
19 # Initialize network parameters
20 input_dim = 2
21 hidden_dim = 4
22 output_dim = 2
23 np . random . seed (42)
24 W1 = np . random . randn ( input_dim , hidden_dim )
25 B1 = np . zeros ((1 , hidden_dim ) )
26 W2 = np . random . randn ( hidden_dim , output_dim )
27 B2 = np . zeros ((1 , output_dim ) )
28
29 # Training parameters
30 learning_rate = 0.1
31 epochs = 5000
32 losses = []
33
34 # Training loop
35 for epoch in range ( epochs ) :
36 # Forward pass
37 Z1 = np . dot (X , W1 ) + B1
38 A1 = tanh ( Z1 )
39 Z2 = np . dot ( A1 , W2 ) + B2
40 A2 = tanh ( Z2 )
41
42 # Compute loss
43 loss = np . mean (( y - A2 ) ** 2)
44 losses . append ( loss )
45
46 # Backpropagation
47 dA2 = ( A2 - y ) * tanh_derivative ( Z2 )
48 dW2 = np . dot ( A1 .T , dA2 )
49 dB2 = np . sum ( dA2 , axis =0 , keepdims = True )
50

51 dA1 = np . dot ( dA2 , W2 . T ) * tanh_derivative ( Z1 )

52 dW1 = np . dot ( X .T , dA1 )
53 dB1 = np . sum ( dA1 , axis =0 , keepdims = True )
54
55 # Update weights and biases
56 W1 -= learning_rate * dW1
57 B1 -= learning_rate * dB1
58 W2 -= learning_rate * dW2
59 B2 -= learning_rate * dB2
60
61 if epoch % 1000 == 0:
62 print ( f " Epoch { epoch }: Loss = { loss :.4 f } " )
63
64 # Plot loss curve
65 plt . plot ( losses )
66 plt . xlabel ( " Epochs " )
67 plt . ylabel ( " Loss " )
68 plt . title ( " Loss Curve for Extended XOR - like FFN " )
69 plt . show ()
70
71 # Decision boundary visualization
72 def p l o t _ d e c i s i o n _ b o u n d a r y ( W1 , B1 , W2 , B2 ) :
73 xx , yy = np . meshgrid ( np . linspace ( -0.1 , 1.1 , 100) , np .
linspace ( -0.1 , 1.1 , 100) )
74 grid = np . c_ [ xx . ravel () , yy . ravel () ]
75
76 # Forward pass
77 Z1 = np . dot ( grid , W1 ) + B1
78 A1 = tanh ( Z1 )
79 Z2 = np . dot ( A1 , W2 ) + B2
80 A2 = tanh ( Z2 )
81
82 predictions = np . argmax ( A2 , axis =1) . reshape ( xx . shape )
83
84 plt . contourf ( xx , yy , predictions , alpha =0.3)
85 plt . scatter ( X [: , 0] , X [: , 1] , c = np . argmax (y , axis =1) ,
 edgecolors = ’k ’ , s =100)
86 plt . xlabel ( " x1 " )
87 plt . ylabel ( " x2 " )
88 plt . title ( " Decision Boundary of Extended FFN " )
89 plt . show ()
90
91 p l o t _ d e c i s i o n _ b o u n d a r y ( W1 , B1 , W2 , B2 )

Observation:
Epoch 0: Loss = 1.8093
Epoch 1000: Loss = 0.0774
Epoch 2000: Loss = 0.0653
Epoch 3000: Loss = 0.0453
Epoch 4000: Loss = 0.0235

Figure 8: Loss Curve for the Three-Layer FFN

 Figure 9: Decision Boundary Visualization

Result:
Successfully designed and trained a three-layer feedforward neural network
using gradient descent, demonstrating correct mapping from input to output
with the hyperbolic tangent activation function.
 EXPERIMENT-9
Aim:
Write MATLAB/Python function to compute the principal components of
the covariance matrix of an l × N dimensional data matrix X as well as
the corresponding variances. Hence, write a MATLAB/Python function that
evaluates the performance of the PCA method when applied on a data matrix
X.

Theory:
Principal Component Analysis (PCA) is a widely used dimensionality reduc-
tion technique that transforms high-dimensional data into a lower-dimensional
form, preserving as much variance as possible. It is based on an eigendecom-
position of the covariance matrix of the data.
Given a data matrix X ∈ Rl×N , where each column represents a sample,
the steps to compute PCA are:

• Centering the Data: Subtract the mean of each row from the data.

• Covariance Matrix: Compute the covariance matrix of the centered

data.

• Eigendecomposition: Compute eigenvectors (principal components)

and eigenvalues (variances).

• Projection: Project the data onto the top-k eigenvectors to reduce

dimensionality.

The variance explained by each component is given by:

Si
Explained Variance Ratioi = Pl (10)
j=1 Sj

where Si is the ith eigenvalue.

This experiment also includes visual evaluation via a scree plot (cumu-
lative variance explained) and comparison of the original and reconstructed
data (if 2D or 3D).
 Code:
1 import numpy as np
2 import matplotlib . pyplot as plt
3
4 def compute_pca ( X ) :
5 X_mean = np . mean (X , axis =1 , keepdims = True )
6 X_centered = X - X_mean
7 cov_matrix = np . cov ( X_centered )
8 S , U = np . linalg . eigh ( cov_matrix ) # Use eigh since
cov_matrix is symmetric
9 idx = np . argsort ( S ) [:: -1] # Descending sort
10 S = S [ idx ]
11 U = U [: , idx ]
12 return U , S , X_mean
13
14 def evaluate_pca (X , num_components =2) :
15 U , S , X_mean = compute_pca ( X )
16 X_centered = X - X_mean
17 U_reduced = U [: , : num_components ]
18
19 # Projection and reconstruction
20 Z = U_reduced . T @ X_centered
21 X_reconstructed = U_reduced @ Z + X_mean
22
23 # Explained variance
24 e x p l a i n e d _ v a r i a n c e _ r a t i o = S [: num_components ] / np . sum ( S )
25
26 # Visualizations
27 plt . figure ( figsize =(16 , 6) )
28
29 # Scree plot
30 plt . subplot (1 , 2 , 1)
31 plt . plot ( np . cumsum ( S ) / np . sum ( S ) , marker = ’o ’)
32 plt . axvline ( x = num_components -1 , color = ’r ’ , linestyle = ’ -- ’
)
33 plt . title ( ’ Cumulative Explained Variance ’)
34 plt . xlabel ( ’ Number of Components ’)
35 plt . ylabel ( ’ Explained Variance Ratio ’)
36
37 # Reconstruction comparison ( only if data is 2 D or 3 D )
38 if X . shape [0] in [2 , 3]:
39 from mpl_toolkits . mplot3d import Axes3D
40 ax = plt . subplot (1 , 2 , 2 , projection = ’3 d ’ if X . shape
[0] == 3 else None )
41 ax . set_title ( ’ Original vs Reconstructed ’)
42
43 if X . shape [0] == 2:
44 ax . plot ( X [0] , X [1] , ’bo - ’ , label = ’ Original ’)
45 ax . plot ( X_reconstructed [0] , X_reconstructed [1] , ’
ro - - ’ , label = ’ Reconstructed ’)
46 else :
47 ax . plot3D ( X [0] , X [1] , X [2] , ’bo - ’ , label = ’
Original ’)
48 ax . plot3D ( X_reconstructed [0] , X_reconstructed [1] ,
X_reconstructed [2] , ’ro - - ’ , label = ’ Reconstructed ’)
49
50 ax . legend ()
51
52 plt . tight_layout ()
53 plt . show ()
54

55 return {
56 "U": U,
57 "S": S,
58 " e x p l a i n e d _ v a r i a n c e _ r a t i o " : explained_variance_ratio ,
59 " X_reconstructed " : X_reconstructed
60 }
61
62 # Generate example 3 D data
63 np . random . seed (0)
64 mean = np . array ([[0] , [0] , [0]])
65 cov = np . array ([[3 , 1 , 0.5] , [1 , 2 , 0.3] , [0.5 , 0.3 , 1]])
66 X = np . random . m u l ti v a ri a t e_ n o rm a l ( mean . flatten () , cov , 100) . T
67
68 # Evaluate PCA
69 result = evaluate_pca (X , num_components =2)
Observation:

Figure 10: Scree Plot for PCA

Figure 11: Original vs Reconstructed Data

Result:
Successfully implemented a PCA function that computes principal compo-
nents and their variances. Also evaluated PCA performance using cumulative
explained variance and reconstructed data, demonstrating effective dimen-
sionality reduction.