Statistical Machine Learning

TEXTBOOKS/LEARNING RESOURCES:
a) Masashi Sugiyama, Introduction to Statistical Machine Learning (1 st ed.), Morgan Kaufmann, 2017. ISBN 978-0128021217.
b) T. M. Mitchell, Machine Learning (1st ed.), McGraw Hill, 2017. ISBN 978-1259096952.

REFERENCE BOOKS/LEARNING RESOURCES:

a) Richard Golden, Statistical Machine Learning A Unified Framework (1 st ed.), unknown, 2020.
Dr. Tej Bahadur Chandra October 30, 2024 1
 Lecture - 7
 Bayes Algorithm

 Probability Theory

 Python Implementation

 Realtime Problems

Dr. Tej Bahadur Chandra October 30, 2024 2

Bayes Algorithm
Naïve Bayes Algorithm

Dr. Tej Bahadur Chandra October 30, 2024 3

 Naïve Bayes Algorithm

It is a supervised learning algorithm, which is based on Bayes theorem and used for
solving classification problems.

Powerful Supervised Classification Algorithm

Naïve Bayes
Algorithm Used for Predictive Analysis

Based on Bayes Theorem

Based on Posterior Probability

Dr. Tej Bahadur Chandra October 30, 2024 4

 Naïve Bayes Algorithm

Naive Bayes classifier is a probabilistic machine learning model.

Bayes theorem: we can find the probability of happening A, given that B has already
occurred.

Bayes The assumption made here is that the predictors/features are independent. That is presence of

Theorem one particular feature does not affect the other. Hence it is called Naive.

??
{ Thomas Bayes }

Where, A, B = events
P(A) =Probability of happening event A
P(B) =Probability of happening event B
P(A|B) =Probability of happening event A provided event B has occurred
P(B|A) =Probability of happening event B provided event A has occurred

Dr. Tej Bahadur Chandra October 30, 2024 5

 Naïve Bayes Algorithm

Example:
Suppose we have a dataset of weather conditions and
corresponding target variable "Play“.

Don’t
play
Given
Weather
Condition
Play

To solve this problem we will use Posterior Probability/

Conditional Probability approach.

Dr. Tej Bahadur Chandra October 30, 2024 6

 Naïve Bayes Algorithm

Example:
Steps to solve this problem:

• Convert the Step-2 • Use Bayes theorem

given dataset to calculate the
into frequency posterior
• Generate Likelihood
tables. probability.
table by finding the
Step-1 probabilities of given Step-3
features

Dr. Tej Bahadur Chandra October 30, 2024 7

 Naïve Bayes Algorithm

Problem -1: If the weather is sunny, then the Player should play or not?
Step -1: Frequency table for the Weather Conditions:
Yes No

Sunny 3 2
Overcast 4 0
Rainy 2 3
Total 9 5
P(Yes|Sunny) =
Step -3: Applying Bayes’ theorem: 0.60 > P(No|Sunny) = 0.40
P(Yes|Sunny)= P(Sunny|Yes)*P(Yes)/P(Sunny) P(No|Sunny)= P(Sunny|No)*P(No)/P(Sunny)
Hence, on a Sunny day, Player can play the game.
P(Sunny|Yes)= 3/9=0.33 P(Sunny|No)= 2/5= 0.5
P(Sunny)= 5/14=0.36 P(Sunny)= 5/14
P(Yes)=9/14=0.64 P(No)=5/14
So P(Yes|Sunny) = ((3/9)*(9/14))/(5/14)= 0.60 So P(No|Sunny)= ((2/5)*(5/14))/(5/14) =0.40

Dr. Tej Bahadur Chandra October 30, 2024 8

 Naïve Bayes Algorithm
Let’s say we have data on 1000 pieces of fruit. We know 3 features of each fruit.

Example
-2

Based on our training set we can also say the following:

• From 500 bananas 400 (0.8) are Long, 350 (0.7) are Sweet and 450 (0.9) are Yellow
• Out of 300 oranges, 0 are Long, 150 (0.5) are Sweet and 300 (1) are Yellow
• From the remaining 200 fruits, 100 (0.5) are Long, 150 (0.75) are Sweet and 50 (0.25) are Yellow
Dr. Tej Bahadur Chandra October 30, 2024 9
 Naïve Bayes Algorithm

Problem-2: Find probability of playing golf when temperature = (Hot)

Problem-4: Find probability of playing golf when windy = (No)

Problem-5: Find probability of playing golf when today = (Sunny, Hot, Normal, False)

Dr. Tej Bahadur Chandra October 30, 2024 10

 Naïve Bayes Algorithm

Naïve: It is called Naïve because it Bayes: It is called Bayes because

assumes that the occurrence of a certain it depends on the principle of
Bayes' Theorem
feature is independent of the occurrence of
other features.
Why is it
called Applications of Naïve Bayes Classifier:
Naïve
Bayes ?? • It is used in medical data classification.

• Used in real-time predictions because Naïve Bayes Classifier is an eager learner.

• It is used in Text classification such as Spam filtering and Sentiment analysis.

Dr. Tej Bahadur Chandra October 30, 2024 11

 Naïve Bayes Algorithm

Advantages of Naïve Bayes Classifier:

• Naïve Bayes is one of the fast and easy ML algorithms to predict a class of datasets.

• It can be used for Binary as well as Multi-class Classifications.

Why is it
• It performs well in Multi-class predictions as compared to the other Algorithms.
called
Naïve • It is the most popular choice for text classification problems.

Bayes ??
Disadvantages of Naïve Bayes Classifier:

• Naive Bayes assumes that all features are independent or unrelated, so it cannot learn the
relationship between features.

Dr. Tej Bahadur Chandra October 30, 2024 12

 Naïve Bayes Algorithm

 Gaussian: It assumes that features follow a normal distribution. This means if predictors
take continuous values instead of discrete, then the model assumes that these values are
sampled from the Gaussian distribution.

Types of  Multinomial: The Multinomial Naïve Bayes classifier is used when the data is
Naïve Bayes multinomial distributed. It is primarily used for document classification problems, it
Model: means a particular document belongs to which category such as Sports, Politics,
education, etc.

 Bernoulli: The Bernoulli classifier works similar to the Multinomial classifier, but the
predictor variables are the independent Booleans variables. Such as if a particular word is
present or not in a document. This model is also famous for document classification tasks.

Dr. Tej Bahadur Chandra October 30, 2024 13

 Naïve Bayes Algorithm

Find the probability of class

provided the given data

Bayesian
P(class|data) = (P(data|class) * P(class)) / P(data)
Classifier

What is the probability that the given

data belongs to the class

Dr. Tej Bahadur Chandra October 30, 2024 14

 Naïve Bayes Algorithm

 Shopping/Market Basket Analysis: To predict whether a person will purchase a product

on a specific combination of day, discount, and free delivery.

Where is
Naive Bayes
Used?

Dr. Tej Bahadur Chandra October 30, 2024 15

 Naïve Bayes Algorithm

 Weather Prediction: To predict if the weather will be good or bad.

 News Classification: With the help of a Naive Bayes classifier, Google News recognizes
whether the news is political, world news, and so on…
Where is
Naive Bayes  Real time Prediction: Naive Bayes is an eager learning classifier and it is sure fast. Thus,
Used? it could be used for making predictions in real time.

 Text classification/ Spam Filtering/ Sentiment Analysis:

 Recommendation System:

Dr. Tej Bahadur Chandra October 30, 2024 16

 Naïve Bayes Algorithm
Iris Dataset – Classification

Python
Code

Dr. Tej Bahadur Chandra October 30, 2024 17

 Confusion Matrix

Confusion
Matrix

Dr. Tej Bahadur Chandra October 30, 2024 18

 Confusion Matrix

Example of 100 patients (60 positive and 40 negative)

 True Positive (TP) refers to a sample belonging

to the positive class being classified correctly.

 True Negative (TN) refers to a sample belonging

Confusion to the negative class being classified correctly.

Matrix  False Positive (FP) refers to a sample belonging

to the negative class but being classified wrongly
as belonging to the positive class.

 False Negative (FN) refers to a sample belonging

to the positive class but being classified wrongly
as belonging to the negative class.

Dr. Tej Bahadur Chandra October 30, 2024 19

 Accuracy(ACC) =

Specificity = TN

Precision =

Performance
Measures FP Recall = TP FN

F1- Measure =

Area Under Curve (AUC) =

Where, P=TP+FN and N=TN+FP.

Dr. Tej Bahadur Chandra October 30, 2024 20

  Accuracy. The number of samples correctly classified out of all the samples present in the
test set.

 Precision. The number of samples actually belonging to the positive class out of all the
Performance samples that were predicted to be of the positive class by the model.
Measures

 Recall (Sensitivity). The number of samples predicted correctly to be belonging to the

positive class out of all the samples that actually belong to the positive class.

What is the test's sensitivity? That is, how many diseased

individuals does it correctly identify as diseased?

Dr. Tej Bahadur Chandra October 30, 2024 21

  Specificity. The number of samples predicted correctly to be belonging to the negative
class out of all the samples that actually belong to the negative class.

What is the test's specificity? That is, how many healthy

individuals does it correctly identify as healthy?

 F1-Score. The harmonic mean of the precision and recall scores obtained for the positive
Performance class.
Measures

 AUC or AUROC. is a performance measurement for the classification problems at various

threshold settings. Its value ranges between 0 and 1.

Dr. Tej Bahadur Chandra October 30, 2024 22

 Sensitivity and specificity of a test are inversely related. Selecting the optimal balance of
sensitivity and specificity depends on the purpose for which the test is used.

Generally, a screening test should be highly sensitive, whereas a follow-up confirmatory test
should be highly specific.

How should I
balance  Sensitivity? How many diseased
sensitivity individuals does the model correctly
with identify as diseased?
specificity?
 Specificity? How many healthy
individuals does the model correctly
identify as healthy?

Dr. Tej Bahadur Chandra October 30, 2024 23

 Confusion Matrix

Confusion
Matrix for
Multiclass
Problems

3 Class 4 Class

Dr. Tej Bahadur Chandra October 30, 2024 24

 Confusion Matrix

Confusion
Matrix for
Multiclass
Problems

3 Class 4 Class

Dr. Tej Bahadur Chandra October 30, 2024 25

Dr. Tej Bahadur Chandra October 30, 2024 26
Why Accuracy is not a good measure ???

Dr. Tej Bahadur Chandra October 30, 2024 27

Further Readings:

• https://en.wikipedia.org/wiki/Naive_Bayes_classifier

• http://gerardnico.com/wiki/data_mining/naive_bayes

• http://scikit-learn.org/stable/modules/naive_bayes.html

• Confusion Matrix: How To Use It & Interpret Results, https://www.v7labs.com/blog/confusion-matrix-guide

• Micro, Macro & Weighted Averages of F1 Score, Clearly Explained,

https://towardsdatascience.com/micro-macro-weighted-averages-of-f1-score-clearly-explained-b603420b292f

Dr. Tej Bahadur Chandra October 30, 2024 28