Evaluation Metrics For Classification Model in Python

Classification is a supervised machine-learning technique that predicts the class label based on the input data. There are different classification algorithms to build a classification model, such as Stochastic Gradient Classifier, Support Vector Machine Classifier, Random Forest Classifier, etc. To choose the right model, it is important to gauge the performance of each classification algorithm.

This tutorial will look at different evaluation metrics to check the model's performance and explore which metrics to choose based on the situation.

Understanding Classification Evaluation Metrics

Understanding classification evaluation metrics is crucial for assessing the performance of machine learning models, especially in tasks like binary or multiclass classification. Some common metrics are:

• Accuracy
• Confusion Matrix
• Precision, Recall and F1 Score
• AUC-ROC Curve

Let's consider the MNIST dataset and try to understand the metrics based on the classifier. MNIST has a set of 70,000 small, handwritten-digit images. Let's go through the dataset before we start. 

from keras.datasets import mnist
from keras.utils import to_categorical
from keras.models import Sequential
from keras.layers import Conv2D, MaxPooling2D, Flatten, Dense
from sklearn.metrics import accuracy_score, confusion_matrix, precision_score, recall_score, f1_score, roc_auc_score, roc_curve
import matplotlib.pyplot as plt
import seaborn as sns

(train_X, train_y), (test_X, test_y) = mnist.load_data()
train_X = train_X.reshape((train_X.shape[0], 28, 28, 1)).astype('float32') / 255
test_X = test_X.reshape((test_X.shape[0], 28, 28, 1)).astype('float32') / 255
train_y = to_categorical(train_y)
test_y = to_categorical(test_y)

model = Sequential([
    Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)),
    MaxPooling2D((2, 2)),
    Flatten(),
    Dense(100, activation='relu'),
    Dense(10, activation='softmax')
])

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
model.fit(train_X, train_y, epochs=3, batch_size=200, validation_split=0.2, verbose=2)
y_pred = model.predict(test_X)
y_pred_classes = y_pred.argmax(axis=1)
y_test_classes = test_y.argmax(axis=1)

Output:

Epoch 1/3
240/240 - 24s - loss: 0.3217 - accuracy: 0.9088 - val_loss: 0.1426 - val_accuracy: 0.9573 - 24s/epoch - 98ms/step
Epoch 2/3
240/240 - 18s - loss: 0.1022 - accuracy: 0.9707 - val_loss: 0.0805 - val_accuracy: 0.9770 - 18s/epoch - 76ms/step
Epoch 3/3
240/240 - 21s - loss: 0.0667 - accuracy: 0.9808 - val_loss: 0.0659 - val_accuracy: 0.9815 - 21s/epoch - 89ms/step
313/313 [==============================] - 2s 6ms/step

1. Accuracy

Classification accuracy is the simplest evaluation metric. It is defined as the number of correct predictions divided by the total number of predictions multiplied by 100. The accuracy metric works great if the target variable classes in the data are approximately balanced. For example, if 60% of the classes in an animal dataset are dogs and 40% are cats, then we can say that it is a balanced dataset. It calculates the ratio of correctly predicted instances to the total instances. It's calculated as:

Accuracy= \frac{\text Total Number of Predictions} {\text Number of Correct Predictions}

In the context of the MNIST dataset, accuracy measures how often the model correctly identifies the handwritten digits.

from sklearn.metrics import accuracy_score
accuracy = accuracy_score(y_test_classes, y_pred_classes)
print(f'Accuracy: {accuracy:.4f}')

Output:

Accuracy: 0.9822

2. Confusion Matrix

The confusion matrix is another way to evaluate the performance of a classifier. Here, it counts the number of times instances of class A are classified as class B. For example, the number of times the classifier confused images of 5s with non-5s.

This is a table that is often used to describe the performance of a classification model. It presents a summary of the predictions made by the model against the actual class labels. The confusion matrix is a matrix with four different combinations of predicted and actual classes: True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN).

Let's compute the confusion matrix to evaluate the performance of a classifier. We can make use of MNIST dataset to compute the confusion matrix. The stepsare as follows:

cm = confusion_matrix(y_test_classes, y_pred_classes)
plt.figure(figsize=(10, 4))
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues')
plt.xlabel('Predicted')
plt.ylabel('Actual')
plt.title('Confusion Matrix')
plt.show()

Output:

3. Precision, Recall and F1 Score

A confusion matrix is a great way to evaluate the performance of a classifier, but sometimes we may need a more concise metric. Here comes the importance of precision.

3.1 Precision: Precision provides the accuracy of the positive prediction made by the classifier. The equation is as follows:

Precision = True Positive / (True Positive + False Positive)

When to choose precision?

In some cases, we need high precision. For example, consider that we trained a classifier to detect videos that are safe for kids. Here, we prefer a classifier that keeps only the safe one (high precision), irrespective of whether the classifier rejects many good videos (low recall).

Precision is typically used with another metric called recall (sensitivity, or the true positive rate ━ TPR).

3.2 Recall: Recall is the ratio of number of true positive predictions (correctly detected by the classifer) to the total number of actual positive instances in the dataset. It measures the completeness of positive predictions. The equation is as follows:

Recall = True Positve / (True Positive + False Negative)

When to choose recall?

In some cases, high recall is given importance instead of high precision. Suppose you train a classifier for fire detection with high precision; certain actual cases were not considered. So it is important to maintain a high recall. Here, security guards will get a few false alarms, but they will be alarmed in almost every actual case.

3.3 F1 Score: The F1 score is the harmonic mean of precision and recall. It favors classifiers that have similar precision and recall. Here, the classifier will only get a high F1 score if both recall and precision are high. The equation is as follows:

F1 = 2 * (Precision * Recall) / (Precision + Recall)

When to choose F1 Score?

F1 Score is invaluable in binary classification tasks, especially with imbalanced datasets, where accuracy can be misleading. It strikes a balance between precision and recall, crucial in scenarios where both are equally important, like medical diagnosis. This metric effectively captures the trade-off between precision and recall, offering a comprehensive evaluation of model performance.

For implementation refer to code below:

precision = precision_score(y_test_classes, y_pred_classes, average='macro')
recall = recall_score(y_test_classes, y_pred_classes, average='macro')
f1 = f1_score(y_test_classes, y_pred_classes, average='macro')

print(f'Precision: {precision:.4f}')
print(f'Recall: {recall:.4f}')
print(f'F1 Score: {f1:.4f}')

Output:

Precision: 0.9823
Recall: 0.9821
F1 Score: 0.9822

In the above code, we make use of the f1_score() method from the sklearn metric to calculate the F1 score.

4. ROC Curve

The Receiver Operating Characteristic (ROC) curve is a graphical representation of the performance of a classification model at various thresholds. It plots the True Positive Rate (TPR) against the False Positive Rate (FPR). The Area Under the ROC Curve (AUC-ROC) is a metric to evaluate the performance of a binary classification model. AUC-ROC value lies between 0 and 1, where a higher value indicates better performance. AUC-ROC is insensitive to class distribution and gives an aggregate measure of performance across all possible classification thresholds.

The true positive rate is calculated as:

TPR = True Positives / (True Positives + False Negatives)

It defines how good the model is at predicting the positive class for a positive outcome. It is also known as sensitivity.

The false positive rate is calculated as:

FPR = False Positives / (False Positives + True Negatives)

It is also referred to as inverted specificity (1 - specificity), where specifity is calculated as:

Specificity = True Negative / (True Negaive + False Positive)

Let's get to the implementation part using Sklearn.The code is as follows:

# Assuming y_test and y_pred_prob are the true labels and predicted probabilities respectively
y_pred_prob = model.predict(test_X)
roc_auc = roc_auc_score(test_y, y_pred_prob, multi_class='ovr')

print(f'ROC AUC: {roc_auc:.4f}')

# Plotting ROC Curve for one class (e.g., class 0)
fpr, tpr, _ = roc_curve(y_test_classes == 0, y_pred_prob[:, 0])
plt.plot(fpr, tpr, label='Class 0 ROC curve')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve')
plt.legend(loc='best')
plt.show()

Output:

ROC AUC: 0.9998

Conclusion

It is important to gain insights on how well a machine learning algorithm performs on unseen data. Choosing the right evaluation metrics will help identify the right ML algorithm that performs well. Here, we have gone through different evaluation metrics and also discussed how to choose the right evaluation metrics for classification.