Presentation on
Classification(2)
Section 8.5.1-8.5.3
K Saiveer – 30121101.
8.5.1 Metrics for Evaluating Classifier Performance
• Focus on the predictive capability of a model.
• Rather than how fast it takes to classify or build models, scalability, etc.
• Model Evaluation Metrix:
• Accuracy (also known as recognition rate)
• Sensitivity (or recall)
• Specificity
• Precision
• F-measure
• F1, and Fβ
Confusion Matrix : A table that is often used to describe the
performance of a classification model (or classifier) on a set of test data
for which the true values are known.
• Confusion matrix is a table with size m by m.
• An entry, CMi,j in thr first m rows and m columns
Indicates the number of tuples if class i that were
Labeled by the classifier as class j.
Before we discuss the various measures, we need
to become comfortable with some terminology.
• True Positives (TP): Positive tuples that were correctly labelled by the
classifier.
• True Negatives (TN) : Negative tuples that were correctly labelled by
the classifier
• False Positives (FP) : Negative tuples that were incorrectly labelled as
positive
• False negatives (FN) : Positive tuples that were mislabelled as negative
• Most widely used metric:
• Classifier Accuracy, or recognition rate: percentage of test set tuples
that are correctly classified.
• Error rate, or misclassification rate: Error rate = 1 – accuracy, or
Limitation of Accuracy
• Consider a 2-class problem
• Number of class 0 examples = 9990
• Number of class 1 examples = 10
• If model predicts everything to be class 0:
• Accuracy is 9990/10000 = 99.9%.
• Accuracy is mis leading because model does not detect any class 1 example.
• Class Imbalance Problem:
• One class may be rare ,eg. Fraud or cancer.
• Significant majority of the negative class and minority of the positive class
• Sensitivity is also referred to as the true positive (recognition) rate
(TPR).
• The proportion of positive tuples that are correctly identified
• Specificity is the true negative rate (TNR)
• The proportion of negative tuples that are correctly identified
• It can be shown that accuracy is a function of sensitivity and specificity.
• Precision: Can be thought of as measure of exactness
• What percentage of tuples labelled as positive are actually positive.
• Recall: is a measure od completeness
• What percentage of positive tuples are labelled as positive?
• Perfect score is 1.0
• Inverse relationship between precision and recall
• If recall seems familiar. That’s because it is the same as sensitive (or the true positive rate).
• F measure (F1 or F-Score): harmonic mean of precision and recall,
• It gives equal weight to precision and recall.
• Fβ : weighted measure of precision and recall
• Assigns β times as much weight to recall as to precicion
• Commonly used Fβ measures are
• F2: which weights recall twice as much as precision
• F0.5 which weights precision twice as much as recall
Classifier Evaluation Metrics Example
8.5.2 Holdout Method and Random
Subsampling
• Holdout Method :
• This is a basic concept of estimating a prediction.
• Given a dataset, it is partitioned into two disjoint sets called training set and testing set.
• Classifier is learned based on the training set and get evaluated with testing set.
• Proportion of training and testing sets is at the discretion of analyst; typically 1:1 or 2:1, and
there is a trade-off between these sizes of these two sets.
• If the training set is too large, then model may be good enough, but estimation may be less
reliable due to small testing set and vice-versa.
Random Subsampling
• It is a variation of Holdout method to overcome the drawback of over-
presenting a class in one set thus under-presenting it in the other set and vice-
versa.
• In this method, Holdout method is repeated k times, and in each time, two
disjoint sets are chosen at random with a predefined sizes.
• Overall estimation is taken as the average of estimations obtained from each
iteration.
8.5.3 Cross-Validation
• The main drawback of Random subsampling is, it does not have control over the number of times
each tuple is used for training and testing.
• Cross-validation is proposed to overcome this problem.
• There are two variations in the cross-validation method.
• k-fold cross-validation
• N-fold cross-validation
k-fold Cross-Validation
• Dataset consisting of N tuples is divided into k (usually, 5 or 10) equal,
mutually exclusive parts or folds (, and if N is not divisible by k, then the last
part will have fewer tuples than other (k-1) parts.
• A series of k runs is carried out with this decomposition, and in ith iteration is
used as test data and other folds as training data
• Thus, each tuple is used same number of times for training and once for testing.
• Overall estimate is taken as the average of estimates obtained from each
iteration.
D1 Fold 1
Learning
Di technique
Fold i
Data set
Dk
Fold k
CLASSIFIER
Accuracy Performance
N-fold Cross-Validation
• In k-fold cross-validation method, part of the given data is used in training
with k-tests.
• N-fold cross-validation is an extreme case of k-fold cross validation, often
known as “Leave-one-out’’ cross-validation.
• Here, dataset is divided into as many folds as there are instances; thus, all
most each tuple forming a training set, building N classifiers.
• In this method, therefore, N classifiers are built from N-1 instances, and
each tuple is used to classify a single test instances.
• Test sets are mutually exclusive and effectively cover the entire set (in
sequence). This is as if trained by entire data as well as tested by entire data
set.
• Overall estimation is then averaged out of the results of N classifiers.
N-fold Cross-Validation : Issue
• So far the estimation of accuracy and performance of a classifier model is
concerned, the N-fold cross-validation is comparable to the others we have
just discussed.
• The drawback of N-fold cross validation strategy is that it is
computationally expensive, as here we have to repeat the run N times; this
is particularly true when data set is large.
• In practice, the method is extremely beneficial with very small data set
only, where as much data as possible to need to be used to train a classifier.
Thank
you.
