Classification

ML Team
Outline:
1- Introduction
2- Classification:
- Basic concepts
- Process
3- Classification Methods Evaluation:
- Confusion Matrix
- Accuracy , Recall, Precision and F1 Score
- ROC Curve
- AUC
Introduction:
 What is Classification ?

• Classification is a machine learning task of predicting

the value of a labeled categorical class.
Identifying and assigning for a given observation (
data point ) the corresponding class ( label).

• Considered as an instance of Supervised Learning (

Known as Clustering in Unsupervised Learning ).
• Classification quality Evaluation (Error rate).
Data and Goal:

• Data: A set of data records (also called examples,

instances or cases) described by
– k attributes: A1, A2, … Ak.
– a class: Each example is labelled with a pre-defined
class.
• Goal: To learn a classification model from the data that
can be used to predict the classes of new (future, or test)
cases/instances.
Classification Problems: Examples
1) Email Spam
2) Handwritten Digit Recognition
3) Customer behavior prediction:
4) Image classification
5) Anomaly detection problems such as fraud detection
6) Etc
Classification Process:

Step 1: Construction of the model from the learning set

(training set).

Step 2: The use of the model:

-Test the accuracy of the model (test set).
- Use it in predicting new features ( observations).
Step 1: Building the classification Model

• Step 1: Dividing labeled data into a set of training

data ( Learning process) and a set of testing data (
Evaluation Process)
Data Repartition
Commonly:
1) Training set: represents 80% ( 75%).
2) Testing set: represents 20% ( 25%).
Cross Validation
• Cross validation consists of training and then
validating our model on several possible sections of
the train set.
- Divide the training data into k subsets.
- Use k = 1 fold as training data and a subset as test
data.
k-fold cross-validation
• Using k-fold cross-validation for hyper-parameter tuning is
common when the size of the training data is small
– It also leads to a better and less noisy estimate of the model
performance by averaging the results across several folds
• E.g., 5-fold cross-validation (see the figure on the next slide)
1. Split the train data into 5 equal folds
2. First use folds 2-5 for training and fold 1 for validation
3. Repeat by using fold 2 for validation, then fold 3, fold 4, and
fold 5
4. Average the results over the 5 runs (for reporting purposes)
5. Once the best hyper-parameters are determined, evaluate the
model on the test data
Step 1: Building the classification Model

• The chosen machine learning algorithm creates a

predictive model.
Step 2: Evaluation of the classification Model

• Testing data are used to evaluated the created model.

 Step 2: Evaluation of the classification Model

• Based on the obtained results, modifications may be

done to the constructed model or training it on new data.
Step 2: Evaluation of the classification Model
Evaluation of classification methods
• Evaluating the performance of a classification model is essential:
• To know if the model is significant globally: My model does it
really reflect causality?
• To get an idea of deployment performance: What will be the
reliability (the associated costs) when I use my model?
• To compare several candidate models: Which among several
models will be the most efficient given my objectives?

 The measurement and evaluation of the performance of a

classification model is always done on the test sample: It is
necessary to test the performance of the model on data which
have not and e used to build the classification model.
Confusion Matrix:
• To evaluate the performance of a classification model
we present four indicators which are calculated from
the confusion matrix:

1) Accuracy rate
2) The recall
3) The precision
4) F1 Score
Example:
• We have a database of customers who have subscribed
to a service.
• Customers who are still subscribers.
• Customers who have canceled the service.
Example:
• We build a churn score: for each customer, we predict if he
will cancel or keep their subscription the following month.
• What is the performance of this score?
• How much can I trust him to predict future terminations?
Confusion Matrix:
Accuracy:
Precision:

• The precision answers the following question:

What proportion of positive identifications was actually
correct?
 A classification model producing no false positives at a
precision of 1.0.
Recall:
The recall is the number of correctly classified positive
examples divided by the total number of actual positive
examples in the test set.

• The recall (recall) allows you to answer the following

question:
What proportion of true positives was identified correctly?
• It gives an indication of the proportion of false negatives.
 A model producing no false negatives has a recall of 1.0.
An example

• This confusion matrix gives

– precision p = 100% and
– recall r = 1%
because we only classified one positive example correctly and no
negative examples wrongly.
• Note: precision and recall only measure classification on
the positive class.

24
Precision and recall measures

• To evaluate the performance of a model in a complete

way: It is necessary to analyze both the precision and
the recall.
• Precision and recall are frequently in tension:
improving precision usually comes at the expense of
recall and reciprocally.
• Various tools have been created to simultaneously
assess precision and recall. The F-score is one of them
F1-score
• It is hard to compare two classifiers using two measures. F1
score combines precision and recall into one measure

• The harmonic mean of two numbers tends to be closer to the

smaller of the two.
• For F1-value to be large, both p and r much be large.
Receive Operating Characteristics curve
• It is commonly called the ROC curve.
• It is a plot of the true positive rate (TPR)
against the false positive rate (FPR).

• True positive rate:

• False positive rate:

Roc Curve: Sensitivity and Specificity
• In statistics, there are two other evaluation
measures:
– Sensitivity: Same as TPR
– Specificity: Also called True Negative Rate (TNR)

• Then we have
 Receive Operating Characteristics curve
• The ROC curve is a tool for evaluating and comparing
models:
- Independently of confusion matrices of misassignment;
It allows to know if a model M1 will be better than the
model M2 regardless of the confusion matrix.
- Operational even in the case of very unbalanced
distributions: Without the perverse effects of the
confusion matrix linked to the need to perform an
assignment.
- A graphical tool that visualizes performance: Only one a
glance should allow us to see the most suitable model for
our interest.
Example ROC curves
Drawing an ROC curve
Area under the curve (AUC)
• Which classifier is better, C1 or C2?
– It depends on which region you talk about.
• Can we have one measure?
– Yes, we compute the area under the curve (AUC)
• If AUC for Ci is greater than that of Cj, it is said that Ci is
better than Cj.
– If a classifier is perfect, its AUC value is 1
– If a classifier makes all random guesses, its AUC value is
0.5.
ROC curve Comparison
ROC curve Comparison
AUC Evaluation

Perfect Case:
• The curve of M1 is always above
that of M2:
 There cannot exist a situation
where M2 would be a better
classification model.
AUC Evaluation

Possible Case: Overlap between ROC

curves
• M1 is dominated by all models, it
can be eliminated.
• In our example, the convex hull is
formed by the curves of M3 and
M2.
• M4 may be better than M3 in
some cases, but in those cases
there, it will be less good than M2
 M4 can be eliminated.
 Mann-Whitney test
• A rank-based statistics used to
show that two distributions are
different.
• In our context, show that the +
present (in average) higher scores
than the _.
• A statistical test can be derived
from the Sum of the ranks of the +:
S+.

• Mann-Whitney Measure:
Classification Algorithms
• k-Nearest Neighbour (k-NN).
• Decision Tree
• Support vector Machine (SVM)
• Naive Bayes
• Logistic Regression.