Tikrit University

College of computer science and

mathematics
Department of computer science
Machine learning

Lecture Four
Introduction to machine learning(cont.)

1
 Topics
Performance evaluation metrics for machine learning methods:

1. Classification Metrics

2. Regression Metrics

3. Clustering Metrics

2
 Some performance evaluation metrics for machine learning methods
• The topic of evaluating the performance of machine learning methods and
algorithms is highly important as it determines the effectiveness and
performance of a particular machine learning technique or algorithm.
• Here are key performance metrics for different types of machine learning
problems:
1. Classification Metrics

2. Regression Metrics

3. Clustering Metrics

3
 Classification Metrics

For tasks where the goal is to assign a label to each input (e.g., spam
detection, image classification):
- Accuracy
- Confusion Matrix
- Precision
- Recall
- F1-score

4
 Accuracy

Accuracy is a performance metric that is based on the correct results of

the learning process and divides them by the total results, as shown in
the following equation:

𝑁𝑜. 𝑜𝑓 𝐶𝑜𝑟𝑟𝑒𝑐𝑡𝑒𝑑 𝑅𝑒𝑠𝑢𝑙𝑡𝑠

𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 =
𝑇𝑜𝑡𝑎𝑙 𝑁𝑜. 𝑜𝑓 𝑅𝑒𝑠𝑢𝑙𝑡𝑠
It is best for balanced datasets where each class has a roughly equal
number of samples.
5
 Confusion matrix
• A confusion matrix is a n*n matrix that is used for evaluating the
performance of the classification model.

For Binary classification —The confusion Matrix is a 2*2 matrix.

• If the target class is 3 means the Confusion Matrix is 3*3 matrix and so on.
• The classifiers for the two-category confusion matrix are shown in the next
slide.

6
 Confusion matrix(cont.)

Terminologies used in Confusion Matrix:

• True Positive → Positive class which is predicted as positive.
• True Negative → Negative class which is predicted as negative.
• False Positive → Negative class which is predicted as positive.[Type I Error]
• False Negative →Positive class which is predicted as negative.[Type II Error]

7
 Precision

Measures how many of the predicted positive cases were

positive. High precision means few false positives. The
precision model is written as follows:

8
 Recall
Measures how many actual positive cases were correctly predicted.
High recall means few false negatives. The recalled model is calculated
as follows:

9
 F1-Score
• The F1 score is a metric used to evaluate the performance of a
classification model in machine learning. It is a weighted average of
precision and recall. It provides a balance between these two metrics. The
F1 score is particularly useful when there is an imbalance in the number of
instances between the positive and negative classes, as it considers both
false positive and false negative predictions.

𝟐 (𝒑𝒓𝒆𝒄𝒊𝒔𝒊𝒐𝒏 ∗ 𝒓𝒆𝒄𝒂𝒍𝒍)
𝐹 − 𝑠𝑐𝑜𝑟𝑒 =
𝒑𝒓𝒆𝒄𝒊𝒔𝒊𝒐𝒏 + 𝒓𝒆𝒄𝒂𝒍𝒍
10
 F1 score vs Accuracy

• Accuracy deals with True positive and True Negative. It doesn’t mention about
False-positive and False-negative. So we are not aware of the distribution of
False-positive and False-negative. If accuracy is 95% means, we don’t know how
the remaining 5% is distributed between False-positive and False-negative.

• F1 Score deals with False-positive and False-negative. For some models, we want
to know about the distribution of False-negative and False-positive. For those
models, the F1 Score metric is used for evaluating the performance.

11
• Example. The cancer data set has 100 records, out of which 94 are cancer records
and 6 are non-cancer records. But the model is predicting 90 out of 94 cancer
records correctly. Four cancer records are not predicted correctly [ 4 —FN].
Compute Recall.

12
Calculating metrics from the Confusion Matrix
• Let’s take the “Email Spam Filtering” example. Our task is to detect spam
emails. So spam emails are marked as 1 and not spam emails are marked as 0.
• I have taken 10 records. Let’s say our model prediction looks like this.

13
14
15
 Regression Metrics

For tasks where the goal is to predict continuous values (e.g., house
prices, stock prices):
- Mean Absolute Error (MAE)
- Mean Squared Error (MSE)
- Root Mean Squared Error (RMSE)

16
 Mean absolute error
• It is measured as the average absolute difference between the
predicted values and the actual values and is used to assess the
effectiveness of a regression model.
• T h e Mean Absolute Error is calculated as:

17
• Example (MAE): Here’s a set of actual prices of crafts and the prices
predicted by the algorithm:

Solution:
• Here, n = 5,
• MAE = 1/5 * (|25-28| + |15-14| + |20-22| + |30-29| + |40-38|) = 1/5
* (3+1+2+1+2) = 1/5 * (9) = 1.8

18
 Mean square error
• T h e Mean Squared Error measures how close a regression line is to a
set of data points.

• A smaller MSE is preferred because it indicates that your data points

are dispersed closely around its central moment (mean).

• T h e Mean Squared Error is calculated as:

Where n represents the number of samples.

19
• Calculate Mean Square Error for the sales data of a product of all the
months.
• Step 1: Calculate the squared error of each data :

20
• Step2: Calculate the Mean Squared Error:

The MSE for this model is 8.17.

21
 Root Mean Squared Error (RMSE)

It is the square root of MSE and gives an error in the same units as the
target variable. The RMSE is calculated as:

The RMSE for the model mentioned in the previous slide is 2.86.

22
 Clustering Metrics

- Clustering algorithms are unsupervised learning techniques used to

divide data into groups (clusters) such that the data points within
each group are more similar to each other than to those in other
groups.

- Evaluating the performance of clustering algorithms can be

challenging since there is no predefined classification to rely on.

23
 Clustering Metrics(cont.)
Several evaluation metrics are used to assess the quality of clustering. These
metrics can be divided into two main categories:

- Internal Evaluation Metrics: These metrics rely on the information available

from the data itself, such as the distances between points or their similarity. The
goal is to evaluate the clustering quality based on the internal structure of the
data.

- External Evaluation Metrics: These metrics rely on prior knowledge of the

correct labels or classifications of the data and are usually used when labeled or
pre-categorized data is available.

24