Scribd
Upload
65 views23 pages

Confusion Matrix & Evaluation Metrics in Machine Learning

The document discusses the importance of evaluation metrics for machine learning classification algorithms, focusing on the confusion matrix and various performance metrics such as accuracy, precision, recall, F1 score, and AUC-ROC. It explains the components of a confusion matrix, including true positives, true negatives, false positives, and false negatives, and highlights the significance of these metrics in assessing model performance. Additionally, it covers regression evaluation metrics like Mean Absolute Error and Root Mean Square Error, emphasizing their roles in predicting continuous values.

Uploaded by

Haleem
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
65 views23 pages

Confusion Matrix & Evaluation Metrics in Machine Learning

The document discusses the importance of evaluation metrics for machine learning classification algorithms, focusing on the confusion matrix and various performance metrics such as accuracy, precision, recall, F1 score, and AUC-ROC. It explains the components of a confusion matrix, including true positives, true negatives, false positives, and false negatives, and highlights the significance of these metrics in assessing model performance. Additionally, it covers regression evaluation metrics like Mean Absolute Error and Root Mean Square Error, emphasizing their roles in predicting continuous values.

Uploaded by

Haleem
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 23

Confusion Matrix &

Evaluation Metrics in
Machine Learning
Metrics to Evaluate Machine
Learning Classification Algorithms
• Now that we have an idea of the different types of
classification models, it is crucial to choose the right
evaluation metrics for those models.
• We will cover the most commonly used metrics: accuracy,
precision, recall, F1 score, and area under the ROC
(Receiver Operating Characteristic) curve and AUC
(Area Under the Curve).
Understanding the Confusion Matrix
in Machine Learning
• Machine learning models are increasingly used in
various applications to classify data into different
categories.
• However, evaluating the performance of these models
is crucial to ensure their accuracy and reliability.
• One essential tool in this evaluation process is the
confusion matrix
What is a Confusion Matrix?
• A confusion matrix is a simple table that shows how well a
classification model is performing by comparing its
predictions to the actual results.
• It breaks down the predictions into four categories:
• Correct predictions for both classes (true positives and true
negatives) and
• Incorrect predictions (false positives and false negatives).
• This helps you understand where the model is making
mistakes, so you can improve it
A 2X2 Confusion matrix
• The matrix displays the number of instances produced by the
model on the test data.
• True Positive (TP): The model correctly predicted a positive
outcome (the actual outcome was positive).
• True Negative (TN): The model correctly predicted a negative
outcome (the actual outcome was negative)
• False Positive (FP): The model incorrectly predicted a positive
outcome (the actual outcome was negative). Also known as a
Type I error.
• False Negative (FN): The model incorrectly predicted a
negative outcome (the actual outcome was positive). Also
known as a Type II error
Example - Confusion Matrix for Dog
Image Recognition with Numbers
• Actual Dog Counts = 6

• Actual Not Dog Counts = 4

• True Positive Counts = 5

• False Positive Counts = 1

• True Negative Counts = 3

• False Negative Counts = 1


Why do we need a Confusion
Matrix?
• A confusion matrix helps you see how well a model is
working by showing correct and incorrect predictions.
• It also helps calculate key measures like accuracy,
precision, and recall, which give a better idea of
performance, especially when the data is imbalanced.
Metrics based on Confusion Matrix Data
What is the AUC-ROC curve?

• The AUC-ROC curve, or Area Under the Receiver


Operating Characteristic curve, is a graphical
representation of the performance of a binary
classification model at various classification thresholds.
• It is commonly used in machine learning to assess the
ability of a model to distinguish between two classes,
typically the positive class (e.g., presence of a disease)
and the negative class (e.g., absence of a disease).
• Receiver Operating Characteristics (ROC)
Curve
• ROC stands for Receiver Operating
Characteristics, and the ROC curve is the
graphical representation of the
effectiveness of the binary classification
model.
• It plots the true positive rate (TPR) vs the
false positive rate (FPR) at different
classification thresholds
Area Under Curve (AUC) Curve

• The AUC curve represents the area under the ROC curve.
• It measures the overall performance of the binary classification
model.
• As both TPR and FPR range between 0 to 1, So, the area will always
lie between 0 and 1, and A greater value of AUC denotes better
model performance.
• Our main goal is to maximize this area in order to have the highest
TPR and lowest FPR at the given threshold.
• The AUC measures the probability that the model will assign a
randomly chosen positive instance a higher predicted probability
compared to a randomly chosen negative instance.
Type 1 and Type 2 error
Type 1 error
• A Type 1 Error occurs when the model incorrectly predicts a positive instance, but the
actual instance is negative.
• This is also known as a false positive. Type 1 Errors affect the precision of a model,
which measures the accuracy of positive predictions.
Example:

Scenario: A diagnostic test is used to detect a particular disease in patients.

• Type 1 Error (False Positive):

• This occurs when the test predicts a patient has the disease
(positive result), but the patient is actually healthy (negative
case).
Type 2 Error
• A Type 2 Error occurs when the model fails to predict a positive instance,
even though it is actually positive.
• This is also known as a false negative.
• Type 2 Errors impact the recall of a model, which measures how well the
model identifies all actual positive cases

Example:
Scenario: A diagnostic test is used to detect a particular disease in patients.

• Type 2 Error (False Negative):


• This occurs when the test predicts the patient is healthy
(negative result), but the patient actually has the disease
(positive case).
Strategies to choose the right metric
Regression Evaluation Metrics
• In the regression task, we are supposed to predict the
target variable which is in the form of continuous
values.
• To evaluate the performance of such a model below
mentioned evaluation metrics are used
• Mean Absolute Error
• Mean Squared Error
• Root Mean Square Error
• Root Mean Square Logarithmic Error
• R2 – Score
Mean Absolute Error(MAE)
• It is the average distance between Predicted and
original values.
• Basically, it gives how we have predicted from the
actual output.
• However, there is one limitation i.e. it doesn’t give any
idea about the direction of the error which is whether
we are under-predicting or over-predicting our data.
• It can be represented mathematically in this way:
Root Mean Square Error(RMSE)
• We can say that RMSE is a metric that can be obtained
by just taking the square root of the MSE value.
• As we know that the MSE metrics are not robust to
outliers and so are the RMSE values.
• This gives higher weightage to the large errors in
predictions.
Root Mean Squared Logarithmic Error(RMSLE)
• There are times when the target variable varies in a
wide range of values.
• And hence we do not want to penalize the
overestimation of the target values but penalize the
underestimation of the target values.
• For such cases, RMSLE is used as an evaluation metric
which helps us to achieve the above objective.
• Some changes in the original formula of the RMSE code
will give us the RMSLE formula that is as shown below:

You might also like