CONFUSION MATRIX FOR MULTI -CLASS CLASSIFICATION

Intelligence is the capacity for logic, understanding, self-awareness, learning, emotional knowledge, reasoning,
planning, creativity, critical thinking, and problem-solving. More generally, it can be described as the ability to
perceive or infer information, and to retain it as knowledge to be applied towards adaptive behaviors within an
environment or context.

Fakultas Ilmu Komputer

Computer Science
Confusion Matrix for Multi-Class Classification
 (Previous meeting…)

Binary Classification
BINARY CLASSIFICATION

1) A confusion matrix is a tabular way of visualizing the

performance of the prediction model.

2) Each entry in a confusion matrix denotes the number of

predictions made by the model where it classified the
classes correctly or incorrectly.

For a binary classification problem, we would have a

2 x 2 matrix as shown below (next slide) with 4 values.
BINARY CLASSIFICATION
1) True Positive (TP): It refers to the number of
predictions where the classifier correctly predicts the
positive class as positive.
2) True Negative (TN): It refers to the number of
predictions where the classifier correctly predicts the
negative class as negative.
3) False Positive (FP): It refers to the number of
predictions where the classifier incorrectly predicts the
negative class as positive.
4) False Negative (FN): It refers to the number of
predictions where the classifier incorrectly predicts the
positive class as negative.

A Confusion matrix is an N × N matrix used for evaluating the performance of a classification model, where N is the number
of target classes. The matrix compares the actual target values with those predicted by the machine learning model. This
gives the holistic view of how well the classification model is performing and what kinds of errors it is making.
BINARY CLASSIFICATION

A binary classification
problem has only two
classes to classify,
preferably a positive
and a negative class.
BINARY CLASSIFICATION
[1] Prediksi Pasien Menderita Cancer atau Tidak
Diketahui:
Total 20 pasien dengan 9 pasien positif cancer (label 1) dan 11 pasien negatif cancer (label 0).

Perbandingan Actual dan predicted Classes/Labels:

Actual (origin) Class: {1 , 1 , 1 , 1 , 1 , 1 , 1 , 1 , 1 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0}
Predicted Class : {1 , 1 , 0 , 1 , 0 , 1 , 0 , 1 , 1 , 0 , 0 , 1 , 0 , 0 , 0 , 0 , 0 , 1 , 0 , 0}

Penyelesaian:

6 6
TP = 6 𝑺𝒆𝒏𝒔𝒊𝒕𝒊𝒇𝒊𝒕𝒚 = = = 0.66667 ∗ 100% = 66.67%
6+3 9
TN = 9 9 9
𝑺𝒑𝒆𝒄𝒊𝒇𝒊𝒄𝒊𝒕𝒚 = = = 0.8182 ∗ 100% = 81.82%
FN = 3 9 + 2 11
FP = 2 6+9 15
𝑶𝒗𝒆𝒓𝒂𝒍𝒍 𝑨𝒄𝒄𝒖𝒓𝒂𝒄𝒚 = = = 0.75 ∗ 100% = 75.00%
6 + 2 + 9 + 3 20
6 × 9 − (2 × 3) 54 − (6) 48 48
𝑴𝑪𝑪 = = = = = 0.49
6 + 2 ∗ 6 + 3 ∗ 9 + 2 ∗ (9 + 3) 8 ∗ 9 ∗ 11 ∗ (12) 9504 97.49
 Multi-Class
Classification Problem
MULTI-CLASS CLASSIFICATION
1) Unlike binary classification, there are
no positive or negative classes in
multi-class classification.

2) Basically, it might be a little difficult

to find TP, TN, FP and FN since there
are no positive or negative classes.

3) First thing we need to do is to find

TP, TN, FP and FN for each
individual class.

4) For example, check in next slide. If

we take class Apple, then let’s see
what are the values of the metrics
from the confusion matrix.
MULTI-CLASS CLASSIFICATION

Target class Apple:

TP = 7
TN = (2+3+2+1) = 8
FP = (8+9) = 17
FN = (1+3) = 4

1) Typical metrics used in multiclass are the same as the metrics used in the binary classification case.
2) The metric is calculated for each class by treating it as a binary classification problem after grouping all the
other classes as belonging to the second class.
3) The binary metric is averaged over all the classes to get either a macro average (treat each class equally)
or weighted average (weighted by class frequency) metric.
MULTI-CLASS CLASSIFICATION
 Example: a confusion matrix for a multiclass problem where we have
to predict whether a person loves Facebook, Instagram or Snapchat.
The confusion matrix is shown in a 3 x 3 matrix:

 Setelah menemukan nilai TP, FP, TN dan FN dari masing-masing

class, selanjutnya akan menghitung Sensitivity (Sens), specificity
(Spec), overall accuracy (Acc) dan MCC tersebut juga dari
masing-masing class.

 Selanjutnya dijumlah berdasarkan Sensitivity (Sens), specificity

(Spec), overall accuracy (Acc) dan MCC untuk metrics yang sama
dan dibagi dengan jumlah class (semua hasil rata-rata).
 MULTI-CLASS CLASSIFICATION
[1] Klasifikasi pengguna social media Facebook, Instagram atau Snapchat

Diketahui:
Total 50 pengguna social media dengan 15 users Facebook (label 0), 15 users Instagram (label 1), dan 20 users Snapchat (label 2).

Perbandingan Actual dan predicted Classes/Labels:

Actual (origin) Class: {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2}
Predicted Class : {0, 1, 1, 0, 2, 0, 1, 1, 0, 0, 2, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 1, 2, 2, 2, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 0, 2, 2, 1, 1, 1, 1, 1, 2, 2, 2, 2}

Penyelesaian:
Actual Class

9 2 1
Predicted Class

4 10 5

2 3 14
MULTI-CLASS CLASSIFICATION
[2] Sweet Fruit (Apple, Orange and Mango) Classification

Diketahui:
Total 36 buah yang akan di klasifikasikan dengan 11 buah apple (label 0), 12 buah orange (label 1), dan 13 buah mango (label 2).

Perbandingan Actual dan predicted Classes/Labels:

Actual (origin) Class: {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2}
Predicted Class : {0, 1, 0, 2, 2, 2, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 1, 2, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0}
Penyelesaian:
Actual Class
Apple Orange Mango
Apple

? ? ?
Predicted Class
Orange

? ? ?
Mango

? ? ?
MULTI-CLASS CLASSIFICATION
[2] Sweet Fruit (Apple, Orange and Mango) Classification (Jawaban)

Diketahui:
Total 36 buah yang akan di klasifikasikan dengan 11 buah apple (label 0), 12 buah orange (label 1), dan 13 buah mango (label 2).

Perbandingan Actual dan predicted Classes/Labels:

Actual (origin) Class: {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2}
Predicted Class : {0, 1, 0, 2, 2, 2, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 1, 2, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0}
Penyelesaian:

Actual Class
Apple Orange Mango
Apple

7 8 9
Predicted Class
Orange

1 2 3
Mango

3 2 1
CONFUSION MATRIX ONLINE CALCULATOR
https://confusionmatrixonline.com/
exercises for students
 Python Library for
Binary & Multi-Class Classifications
MACHINE LEARNING LIBRARY & NOTEBOOKS

Scikit-learn: machine learning in Python Google Colab or Jupyter notebooks

Scikit-learn (formerly scikits. learn and also Colab notebooks are Jupyter notebooks that run in
known as sklearn) is a free software machine the cloud and are highly integrated with Google
learning library for the Python programming Drive, making them easy to set up, access, and
language. share.

Link: https://scikit-learn.org/stable/ Link: https://colab.research.google.com/

BINARY CLASSIFICATION
[1] Prediksi Pasien Menderita Cancer atau Tidak
# Import a confusion matrix for binary-class
from sklearn.metrics import confusion_matrix

# Actual and predicted classes

lst_actual_class = [1 , 1 , 1 , 1 , 1 , 1 , 1 , 1 , 1 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0]
lst_predicted_class = [1 , 1 , 0 , 1 , 0 , 1 , 0 , 1 , 1 , 0 , 0 , 1 , 0 , 0 , 0 , 0 , 0 , 1 , 0 , 0]

# Total 20 pasien dengan 9 pasien positif cancer (label 1) dan 11 pasien negatif cancer (label 0)
lst_classes = [0, 1]

# Compute binary-class confusion matrix

tn, fp, fn, tp = confusion_matrix(lst_actual_class, lst_predicted_class, labels=lst_classes).ravel()

sensitivity = round(tp/(tp+fn), 3);

specificity = round(tn/(tn+fp), 3);
accuracy = round((tp+tn)/(tp+fp+tn+fn), 3);
balanced_accuracy = round((sensitivity+specificity)/2, 3);

print("TP={0}, FP={1}, TN={2}, FN={3}".format(tp, fp, tn, fn)); Output

print("Sensitivity: {0}".format(sensitivity));
print("Specificity: {0}".format(specificity));
print("Accuracy: {0}".format(accuracy));
print("Balanced Accuracy: {0}\n".format(balanced_accuracy));
PLEASE NOTE !!!
If you found the same problem here. Don’t directly copy/paste your code from this slide to spyder or google colab. It might contain some invalid
characters. First, you can utilize MS. Office to paste your code to keep text only and then copy it to your text editor.

Paste to keep text only

MULTI-CLASS CLASSIFICATION
[2] Sweet Fruit (Apple, Orange & Mango) Classification
# Import a confusion matrix for multi-class
from sklearn.metrics import multilabel_confusion_matrix

# Actual and predicted classes

lst_actual_class = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]
lst_predicted_class = [0, 1, 0, 2, 2, 2, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 1, 2, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0]

# Class = apple (label 0), orange (label 1), dan mango (label 2)
lst_classes = [0, 1, 2]
Output
# Compute multi-class confusion matrix
arr_out_matrix = multilabel_confusion_matrix(lst_actual_class, lst_predicted_class, labels=lst_classes)

for no_class in range(len(lst_classes)):

arr_data = arr_out_matrix[no_class];
print("Print Class: {0}".format(no_class));

tp = arr_data[1][1]
fp = arr_data[0][1]
tn = arr_data[0][0]
fn = arr_data[1][0]
sensitivity = round(tp/(tp+fn), 3);
specificity = round(tn/(tn+fp), 3);
accuracy = round((tp+tn)/(tp+fp+tn+fn), 3);
balanced_accuracy = round((sensitivity+specificity)/2, 3);

print("TP={0}, FP={1}, TN={2}, FN={3}".format(tp, fp, tn, fn));

print("Sensitivity: {0}".format(sensitivity));
print("Specificity: {0}".format(specificity));
print("Accuracy: {0}".format(accuracy));
print("Balanced Accuracy: {0}\n".format(balanced_accuracy));
MULTI-CLASS CLASSIFICATION
[2] Sweet Fruit (Apple, Orange and Mango) Classification
Try also
this one
EVALUATION METRICS FOR CLASSIFICATION
Calculate common Evaluation Metrics using previous example:
1. Precision ?
2. F1-Score ?
3. Matthews Correlation Coefficient (MCC) ?
REFERENCES
1) Multi-Class Metrics Made Simple, Part I: Precision and Recall
https://towardsdatascience.com/multi-class-metrics-made-simple-part-i-precision-and-recall-9250280bddc2

2) Everything you Should Know about Confusion Matrix for Machine Learning
https://www.analyticsvidhya.com/blog/2020/04/confusion-matrix-machine-learning/

3) Cara Menghitung Confusion Matrix 4 Kelas

https://www.pengalaman-edukasi.com/2020/01/confusion-matrix-multi-class-menghitung.html

4) Perhitungan Confusion Matrix Pada Kasus 3 Kelas

https://www.pengalaman-edukasi.com/2020/11/menghitung-confusion-matrix-3-kelas.html

5) Confusion Matrix for Your Multi-Class Machine Learning Model

https://towardsdatascience.com/confusion-matrix-for-your-multi-class-machine-learning-model-ff9aa3bf7826

6) Multiclass Classification
https://docs.aws.amazon.com/machine-learning/latest/dg/multiclass-classification.html

7) Confusion Matrix three classes python

https://datascience.stackexchange.com/questions/40067/confusion-matrix-three-classes-python

8) Binary and multiclass classifiers

https://stats.stackexchange.com/questions/272934/binary-and-multiclass-classifiers

9) Tips and Tricks for Multi-Class Classification

https://medium.com/@b.terryjack/tips-and-tricks-for-multi-class-classification-c184ae1c8ffc

10) Multi-class Classification — One-vs-All & One-vs-One

https://towardsdatascience.com/multi-class-classification-one-vs-all-one-vs-one-94daed32a87b

11) sklearn.metrics.multilabel_confusion_matrix
https://scikit-learn.org/stable/modules/generated/sklearn.metrics.multilabel_confusion_matrix.html

12) sklearn.metrics.confusion_matrix
https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html
END PRESENTATION
Thank you for your attention
S–W–T