Scribd
Upload
38 views38 pages

Chap3 Part1 Classification

The document discusses classification in machine learning, including basic concepts, processes, and evaluation methods. It covers classification tasks, algorithms, building classification models, and evaluating model performance using metrics like accuracy, precision, recall, F1 score, ROC curves, and AUC.

Uploaded by

houcem.swissi
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
38 views38 pages

Chap3 Part1 Classification

The document discusses classification in machine learning, including basic concepts, processes, and evaluation methods. It covers classification tasks, algorithms, building classification models, and evaluating model performance using metrics like accuracy, precision, recall, F1 score, ROC curves, and AUC.

Uploaded by

houcem.swissi
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 38

Classification

ML Team
Outline:
1- Introduction
2- Classification:
- Basic concepts
- Process
3- Classification Methods Evaluation:
- Confusion Matrix
- Accuracy , Recall, Precision and F1 Score
- ROC Curve
- AUC
Introduction:
What is Classification ?

• Classification is a machine learning task of predicting


the value of a labeled categorical class.
Identifying and assigning for a given observation (
data point ) the corresponding class ( label).

• Considered as an instance of Supervised Learning (


Known as Clustering in Unsupervised Learning ).
• Classification quality Evaluation (Error rate).
Data and Goal:

• Data: A set of data records (also called examples,


instances or cases) described by
– k attributes: A1, A2, … Ak.
– a class: Each example is labelled with a pre-defined
class.
• Goal: To learn a classification model from the data that
can be used to predict the classes of new (future, or test)
cases/instances.
Classification Problems: Examples
1) Email Spam
2) Handwritten Digit Recognition
3) Customer behavior prediction:
4) Image classification
5) Anomaly detection problems such as fraud detection
6) Etc
Classification Process:

Step 1: Construction of the model from the learning set


(training set).

Step 2: The use of the model:


-Test the accuracy of the model (test set).
- Use it in predicting new features ( observations).
Step 1: Building the classification Model

• Step 1: Dividing labeled data into a set of training


data ( Learning process) and a set of testing data (
Evaluation Process)
Data Repartition
Commonly:
1) Training set: represents 80% ( 75%).
2) Testing set: represents 20% ( 25%).
Cross Validation
• Cross validation consists of training and then
validating our model on several possible sections of
the train set.
- Divide the training data into k subsets.
- Use k = 1 fold as training data and a subset as test
data.
k-fold cross-validation
• Using k-fold cross-validation for hyper-parameter tuning is
common when the size of the training data is small
– It also leads to a better and less noisy estimate of the model
performance by averaging the results across several folds
• E.g., 5-fold cross-validation (see the figure on the next slide)
1. Split the train data into 5 equal folds
2. First use folds 2-5 for training and fold 1 for validation
3. Repeat by using fold 2 for validation, then fold 3, fold 4, and
fold 5
4. Average the results over the 5 runs (for reporting purposes)
5. Once the best hyper-parameters are determined, evaluate the
model on the test data
Step 1: Building the classification Model

• The chosen machine learning algorithm creates a


predictive model.
Step 2: Evaluation of the classification Model

• Testing data are used to evaluated the created model.


Step 2: Evaluation of the classification Model

• Based on the obtained results, modifications may be


done to the constructed model or training it on new data.
Step 2: Evaluation of the classification Model
Evaluation of classification methods
• Evaluating the performance of a classification model is essential:
• To know if the model is significant globally: My model does it
really reflect causality?
• To get an idea of deployment performance: What will be the
reliability (the associated costs) when I use my model?
• To compare several candidate models: Which among several
models will be the most efficient given my objectives?

 The measurement and evaluation of the performance of a


classification model is always done on the test sample: It is
necessary to test the performance of the model on data which
have not and e used to build the classification model.
Confusion Matrix:
• To evaluate the performance of a classification model
we present four indicators which are calculated from
the confusion matrix:

1) Accuracy rate
2) The recall
3) The precision
4) F1 Score
Example:
• We have a database of customers who have subscribed
to a service.
• Customers who are still subscribers.
• Customers who have canceled the service.
Example:
• We build a churn score: for each customer, we predict if he
will cancel or keep their subscription the following month.
• What is the performance of this score?
• How much can I trust him to predict future terminations?
Confusion Matrix:
Accuracy:
Precision:

• The precision answers the following question:


What proportion of positive identifications was actually
correct?
 A classification model producing no false positives at a
precision of 1.0.
Recall:
The recall is the number of correctly classified positive
examples divided by the total number of actual positive
examples in the test set.

• The recall (recall) allows you to answer the following


question:
What proportion of true positives was identified correctly?
• It gives an indication of the proportion of false negatives.
 A model producing no false negatives has a recall of 1.0.
An example

• This confusion matrix gives


– precision p = 100% and
– recall r = 1%
because we only classified one positive example correctly and no
negative examples wrongly.
• Note: precision and recall only measure classification on
the positive class.

24
Precision and recall measures

• To evaluate the performance of a model in a complete


way: It is necessary to analyze both the precision and
the recall.
• Precision and recall are frequently in tension:
improving precision usually comes at the expense of
recall and reciprocally.
• Various tools have been created to simultaneously
assess precision and recall. The F-score is one of them
F1-score
• It is hard to compare two classifiers using two measures. F1
score combines precision and recall into one measure

• The harmonic mean of two numbers tends to be closer to the


smaller of the two.
• For F1-value to be large, both p and r much be large.
Receive Operating Characteristics curve
• It is commonly called the ROC curve.
• It is a plot of the true positive rate (TPR)
against the false positive rate (FPR).

• True positive rate:

• False positive rate:


Roc Curve: Sensitivity and Specificity
• In statistics, there are two other evaluation
measures:
– Sensitivity: Same as TPR
– Specificity: Also called True Negative Rate (TNR)

• Then we have
Receive Operating Characteristics curve
• The ROC curve is a tool for evaluating and comparing
models:
- Independently of confusion matrices of misassignment;
It allows to know if a model M1 will be better than the
model M2 regardless of the confusion matrix.
- Operational even in the case of very unbalanced
distributions: Without the perverse effects of the
confusion matrix linked to the need to perform an
assignment.
- A graphical tool that visualizes performance: Only one a
glance should allow us to see the most suitable model for
our interest.
Example ROC curves
Drawing an ROC curve
Area under the curve (AUC)
• Which classifier is better, C1 or C2?
– It depends on which region you talk about.
• Can we have one measure?
– Yes, we compute the area under the curve (AUC)
• If AUC for Ci is greater than that of Cj, it is said that Ci is
better than Cj.
– If a classifier is perfect, its AUC value is 1
– If a classifier makes all random guesses, its AUC value is
0.5.
ROC curve Comparison
ROC curve Comparison
AUC Evaluation

Perfect Case:
• The curve of M1 is always above
that of M2:
 There cannot exist a situation
where M2 would be a better
classification model.
AUC Evaluation

Possible Case: Overlap between ROC


curves
• M1 is dominated by all models, it
can be eliminated.
• In our example, the convex hull is
formed by the curves of M3 and
M2.
• M4 may be better than M3 in
some cases, but in those cases
there, it will be less good than M2
 M4 can be eliminated.
Mann-Whitney test
• A rank-based statistics used to
show that two distributions are
different.
• In our context, show that the +
present (in average) higher scores
than the _.
• A statistical test can be derived
from the Sum of the ranks of the +:
S+.

• Mann-Whitney Measure:
Classification Algorithms
• k-Nearest Neighbour (k-NN).
• Decision Tree
• Support vector Machine (SVM)
• Naive Bayes
• Logistic Regression.

You might also like