   

(../../index.html) (../../pages/about- (../../archive.html) (../../categories/index.html)

me/index.html)

Metrics - Confusion Matrix (.)

 3 years ago (.)  Harun  Source (index.ipynb) 
 Tags: confusion matrix (../../categories/confusion-matrix/)

In a typical data science project we try several models like (logistic regression, SVM, tree-classifiers
etc) on our data.
Then we measure the predicting performance of these models to find the best performing one.
Finally we decide to implement the best performing model.
In this notebook we talk about one of the classification model evaluation tools: Confusion matrix.
They can help us to see deeper how much reliable our models are.
We are going to look at the confusion matrices of a variety of Scikit-Learn models and compare them
using visual diagnostic tools from Yellowbrick in order to select the best model for our data.

In [4]: # Notebook setup

import pandas as pd

from sklearn.model_selection import train_test_split

from sklearn.pipeline import Pipeline

from sklearn.linear_model import LogisticRegression

from sklearn.svm import LinearSVC

from sklearn.ensemble import RandomForestClassifier

from sklearn.linear_model import SGDClassifier

import category_encoders as ce

from yellowbrick.classifier import ConfusionMatrix

import matplotlib.pyplot as plt

import warnings

warnings.filterwarnings("ignore")

%matplotlib inline

Confusion Matrix
Since we know the labels of the test set we can measure how succesfull are the predictions of the
model by comparing the the actual labels and predictions
We can see if our classifier identifies the samples succesfully, or it is "CONFUSED" with another label?
Confusion matrix shows the amount of confusion.
We use confusion matrices to understand which classes are most easily confused
There are two sets of labels in a confusion matrix of binary (2 class) classification:
 {POSITIVE, NEGATIVE}- first, the model makes the prediction. It returns the labels 1 (POSITIVE)
or 0 (NEGATIVE).
{TRUE, FALSE}- then the model's prediction is evaluated if the prediction is made correctly
(TRUE) or incorrectly (FALSE) based on the actual known labels.

Tip: If you have difficulty to remember these terms because of the similarity of the terms just insert
the word "PREDICTED" in the middle.
For instance if you are confused with the meaning of "false positive" read it like
"false(ly) PREDICTED positive"

We create a matrix of predictions and actual labels for a binary classification.

One diagonal show the succesful predictions the other diagonal unsuccessful predictions

In multi-class classification ie if the class labels are more than 2 (not 1 or 0, positive or negative), the
confusion matrix looks like something like this

We do not use the terms like "true positive" with the confusion matrix with classes more than 2
The size of the confusion matrix is nxn where n is the number of classes
Different references may use a different convention for axes ie actual and predicted classes can take
place on different axes

Examples of confusion matrices

Lets demonstrate some confusion matrices
We will utilize the tools from yellowbrick library (http://www.scikit-
yb.org/en/latest/api/classifier/confusion_matrix.html). This is a nice library for machine learning
visualization.

Loading and Exploring the Dataset

This tutorial uses a modified version of the mushroom dataset from the UCI Machine Learning
Repository.
Even though these toy datasets are not interesting anymore because of repetitive usage, here our
focus is the classification metrics not other steps of data processing. So we try to get the advantage of
fast implementation of this dataset
Our objective is to predict if a mushroom is poisonous or edible based on its characteristics.
 The data include descriptions of hypothetical samples corresponding to 23 species of mushrooms
Each species was identified as definitely edible or poisonous

In [5]: # Url of the dataset

url='https://raw.githubusercontent.com/rebeccabilbro/rebeccabilbro.github.io/master/data/a
garicus-lepiota.txt'

# Column names list

column_names=['class', 'cap-shape', 'cap-surface', 'cap-color']

# Load the data

mushrooms=pd.read_csv(url, header=None, names= column_names)

mushrooms.head(3)

Out[5]:
class cap-shape cap-surface cap-color

0 edible convex smooth yellow

1 edible bell smooth white

2 poisonous convex scaly white

In [6]: mushrooms.info()

<class 'pandas.core.frame.DataFrame'>

RangeIndex: 8123 entries, 0 to 8122

Data columns (total 4 columns):

class 8123 non-null object

cap-shape 8123 non-null object

cap-surface 8123 non-null object

cap-color 8123 non-null object

dtypes: object(4)

memory usage: 253.9+ KB

In [7]: ## Count the unique values in each column

mushrooms.nunique()

Out[7]: class 2

cap-shape 6

cap-surface 4

cap-color 10

dtype: int64

We see that target and feature columns contain different categorical values.
We need to encode them into numerical types in order to fit Sklearn models.
For this purpose, we will utilize Category Encoders (http://contrib.scikit-learn.org/categorical-
encoding/index.html)
library which provides scikit-learn-compatible categorical variable encoders.
All the transformers of Category Encoders can be used in Sklearn pipelines.
Later in a separate post we will analyse the encoders

Target and Features Datasets

In [8]: # Create the features dataset (X) and target dataset (y)

features = ['cap-shape', 'cap-surface', 'cap-color']

target = 'class'

X = mushrooms[features]

y = mushrooms[target]

Classifiers Dictionary
Now, tet's create a dictionary which contains the classifiers we want to use for our classification task
Here we create the dictionary with instantiates of Sklearn estimators without hyperparameter
tuning.
In reality we need to evaluate the performance of tuned classifiers.
In [10]: # Estimators dictionary

# We can add as more classifiers to our dictionary

# This is just a sample

estimators_dct={"Logistic Legression": LogisticRegression(),

"Linear SVC" : LinearSVC(),

"Random Forest": RandomForestClassifier(n_estimators=8),

"SGD Classifier": SGDClassifier(),

confusion_matrices function
Let's define a function to get the confusion matrices of a given dictionary of models (like in the upper
cell) easily without repetion.

Our function will

take X, y datasets and an estimator dictionary
return the confusion matrices produced by the predictions of each model in the dictionary

In [9]: # set up the figure size for the confusion matrices

plt.rcParams['figure.figsize'] = (6, 4)

plt.rcParams['font.size'] = 15

def confusion_matrices(X, y, estimator_dict):

"""

Takes X, y datasets and an estimator dictionary -> returns confusion matrices of the c
lassifiers

"""

# Split the data as train and test

X_train, X_test, y_train, y_test= train_test_split(X, y, test_size=0.2, random_state=1

1)

# Loop over the estimator_dict keys to get the each estimator

for estimator in estimator_dict.keys():

print(estimator)

# In the pipeline we use OneHotEncoder from Category Encoders

model = Pipeline([('encoder', ce.OneHotEncoder()),

('estimator', estimator_dict[estimator])])

# Instantiate the classification model and visualizer

model.fit(X_train, y_train)

# The ConfusionMatrix visualizer takes a model

cm = ConfusionMatrix(model, fontsize=13, cmap='YlOrBr')

# To create the ConfusionMatrix, we need some test data

# Score runs predict() on the data and then

# creates the confusion_matrix from scikit-learn

cm.score(X_test, y_test)

cm.poof()

In [11]: # Call the confusion_matrices function to get the confusion matrices

confusion_matrices(X, y, estimators_dct)

Logistic Legression

Linear SVC

Random Forest

 SGD Classifier

ConfusionMatrix from Yellowbrick

takes a fitted scikit-learn classifier and a set of X_test and y_test values and
returns a report showing how each of the test samples predicted classes compare to their actual
classes.
Confusion matrices provide similar information as what is available in a ClassificationReport (we
will talk about it soon), but rather than top-level scores, they provide deeper insight into the classifica-
tion of individual data points.
Creates a heatmap visualization of the sklearn.metrics.confusion_matrix().
We can choose between displaying values as the percent of true (cell value divided by sum of row) or
as direct counts.

Conclusion
Even though we can get deeper insight on predictions of the classifiers by confusion matrices, still it is
not very practical to compare several models performance with each other
Since confusion matrices provides tables of actual and prediction comparision we still need some
more metrics to interpret the result more directly to choose the best model
So we will continue to work on classification metrics like precision, recall, roc, auc etc in the next
posts

Sources:
http://www.scikit-yb.org/en/latest/api/classifier/confusion_matrix.html (http://www.scikit-
yb.org/en/latest/api/classifier/confusion_matrix.html)
http://contrib.scikit-learn.org/categorical-encoding/index.html (http://contrib.scikit-
learn.org/categorical-encoding/index.html)

Previous post (../metrics/) Next post (../gps-data-analysis/)

Comments
Contents © 2019 Harun (mailto:[email protected]) - Powered by Nikola (https://getnikola.com)