5.
Classification Algorithm Unit-2: Supervised Learning
Classification algorithms are a subset of machine learning techniques used to predict group membership based
on data features. These algorithms can be evaluated and optimized based on various metrics and techniques
discussed below.
5.1. Evaluating a Classification Model
Evaluation metrics are crucial for assessing the performance of classification models, ensuring that they are
both accurate and applicable to real-world scenarios.
5.1.1. Log Loss or Cross Entropy Loss
• Definition: A performance metric that measures the uncertainty of a classification model's predictions
based on how much the predicted probabilities deviate from the actual class labels.
• Calculation: Log Loss penalizes false classifications by taking into account the uncertainty of each
prediction.
• Formula:
1 #
− $ [𝑦! log(𝑦+) +" )]
" + (1 − 𝑦! )log (1 − 𝑦
𝑁 !$%
• where 𝑦! is the actual class label and 𝑦+" is the predicted probability of the class label.
5.1.2. Confusion Matrix
• Components: Includes True Positives (TP), True Negatives (TN), False Positives (FP), and False
Negatives (FN).
• Utility: Helps visualize the performance of an algorithm. Allows calculation of various statistics like
accuracy, precision, recall, and F1-score.
5.1.3. AUC-ROC Curve
• AUC (Area Under Curve): Represents degree or measure of separability. It tells how much the model
is capable of distinguishing between classes.
AL3451- Machine Learning Unit-2: Supervised Learning Page 1 of 3
Classification Algorithm
• ROC (Receiver Operating Characteristics) Curve: A graphical plot that illustrates the diagnostic
ability of a binary classifier system as its discrimination threshold is varied.
• Significance: Higher the AUC, better the model is at predicting 0s as 0s and 1s as 1s.
5.2. Linear Discriminant Analysis (LDA)
A technique used in statistics, pattern recognition, and machine learning to find a linear combination of
features that characterizes or separates two or more classes of objects or events.
5.2.1. Advantages of LDA
• Simplicity: Easy to implement, interpret, and very efficient to train.
• Performance: Often outperforms other classifiers, especially when the assumptions of similar class
covariances and normally distributed data hold.
5.2.2. Drawbacks of LDA
• Assumption of normality: Assumes that the input variables follow a normal distribution.
• Equal covariance: Assumes the same covariance matrix for all classes which might not always hold
true.
• Dimensionality: Does not perform well when there are more features than observations.
5.2.3. Applications of LDA
• Face Recognition: LDA's ability to model the difference between classes (faces) makes it useful in
facial recognition technologies.
• Customer Identification: Useful in predicting customer categories based on buying behavior.
5.2.4. Variants of LDA
• Quadratic Discriminant Analysis (QDA): A variant of LDA which assumes that each class has its
own covariance matrix.
• Regularized Discriminant Analysis (RDA): Modifies LDA to include regularization, improving
performance when assumptions of LDA are violated.
AL3451- Machine Learning Unit-2: Supervised Learning Page 2 of 3
Classification Algorithm
5.3. Principal Component Analysis (PCA)
A statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly
correlated variables into a set of values of linearly uncorrelated variables called principal components.
5.3.1. Advantages of PCA
• Dimensionality Reduction: Efficiently reduces the number of variables.
• Visualization: Helps in visualizing high-dimensional data.
• Noise Reduction: Can eliminate noise by retaining only the most significant features.
5.3.2. Drawbacks of PCA
• Data Scaling: Highly sensitive to the scaling of variables.
• Loss of Information: Some important information might be lost in the process of reducing
dimensions.
• Interpretability: Principal components are less interpretable than original data.
5.3.3. Difference between LDA & PCA
• Purpose: LDA is used to maximize the separability among known categories. PCA, on the other hand,
does not consider the class labels.
• Direction of Maximizing Variance: PCA projects data in directions that maximize the variance
without considering any class labels, whereas LDA tries to project the data in a way that maximizes
the separation between multiple classes.
Understanding the nuances of different classification algorithms and their evaluation metrics is crucial for
developing effective machine learning models. Each method has its unique characteristics, advantages, and
limitations, making them suitable for specific types of data and analysis.
AL3451- Machine Learning Unit-2: Supervised Learning Page 3 of 3
