Scribd
Upload
11 views16 pages

Evaluation Metrics

The document discusses evaluation metrics in machine learning, specifically focusing on accuracy, precision, recall, and F1-score, which are essential for assessing classifier performance. It distinguishes between accuracy (closeness to a specific value) and precision (closeness to each other) and introduces the confusion matrix as a tool for visualizing classifier performance. The document emphasizes the importance of understanding these metrics, especially in the context of applications like spam filtering, where misclassification can have significant consequences.

Uploaded by

Sadbin Mohshin
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
11 views16 pages

Evaluation Metrics

The document discusses evaluation metrics in machine learning, specifically focusing on accuracy, precision, recall, and F1-score, which are essential for assessing classifier performance. It distinguishes between accuracy (closeness to a specific value) and precision (closeness to each other) and introduces the confusion matrix as a tool for visualizing classifier performance. The document emphasizes the importance of understanding these metrics, especially in the context of applications like spam filtering, where misclassification can have significant consequences.

Uploaded by

Sadbin Mohshin
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 16

Evaluation Metrics

• Not only in machine learning but also in general


life, especially business life, you will hear questions
"How accurate is your product?" or "How precise is
your machine?".

• When people get replies like "the most accurate


product in its field!" or "This machine has the
highest imaginable precision!", they comforted by
both answers. Shouldn't they? Indeed, the terms
accurate and precise are very often used
interchangeably.

• But in a nutshell, we can say: Accuracy measure for


the closeness of some measurements to a specific
value, while precision is the closeness
measurements to each other.
• We need them for evaluating ML algorithms or better their
results.
• Four important metrics are used to evaluate the results of
classifications. The metrics are:

• Accuracy
• Precision
• Recall
• F1-Score

We will introduce each of these metrics and we will discuss the pro
and cons of each of them. Each measures something different
about a classifiers performance. The metrics will be of outmost
importance for all machine learning.
Accuracy
• Accuracy is a measure for the closeness of the
measurements to a specific value, while precision is
closeness of the measurements to each other, i.e. not
necessarily to a specific value.
• To put it in other If we have a set of data points from
repeated measurements of the same quantity, the set is
said to be accurate if their average is close to the true
value of the quantity being measured. On the other
hand, the set to be precise, if the values are close to
each other.
• The two concepts are independent of each which means
that the set of data can be accurate, or precise, or both,
Confusion Matrix
• Before we continue with the term accuracy, we want to
make sure that you understand what a confusion matrix is
about.
• A confusion matrix, also called a contingency table or error
matrix, is used to visualize the performance classifier.
• The columns of the matrix represent the instances of the
predicted classes and the rows represent the instances of the
actual class. (Note: It can be the other way around as well.)
• In the case of binary classification the table has 2 rows and 2
columns.
Accuracy in Classification
• Accuracy is also used as a statistical measure.
• Accuracy statistical measure which is defined as the quotient of correct
predictions (both True positives (TP) negatives (TN)) made by a classifier
divided by the sum of all predictions made by the classifier, including False
positives (FP) and False negatives (FN). Therefore, the formula for quantifying
bin
Exercise: Before you go on with the text think about what the value precision
means. If you look at the precision measure of our spam filter example, what
does it tell you about the quality of the spam filter? What do the results of the
confusion matrix of an ideal spam filter look like? What is worse, high FP or FN
values?

You will find the answers indirectly in the following explanations.

Incidentally, the ideal spam filter would have 0 values for both FP and FN.

The previous result means that 11 mail pieces out of a hundred will be classified
as ham, even though they are spam. 89 are correctly classified as ham. This is a
point where we should talk about the costs of misclassification. It is troublesome
when a spam mail is not recognized as "spam" and is instead presented to us as
"ham". If the percentage is not too high, it is annoying but not a disaster. In
contrast, when a non-spam message is wrongly labeled as spam, the email will
not be shown in many cases or even automatically deleted. For example, this
carries a high risk of losing customers and friends. The measure precision makes
no statement about this last-mentioned problem class. What about other
measures?

You might also like