Bayesian Theory & Naïve Bayes Classifiers

Course 4232: Machine Learning

Dept. of Computer Science

Faculty of Science and Technology

Lecturer No: Week No: Semester: Summer 22-23

Instructor: Prof. Dr. Md. Asraf Ali ([email protected])
 Bayesian Classifier
 A statistical classifier: performs probabilistic prediction, i.e.,
predicts class membership probabilities
 Foundation: Based on Bayes’ Theorem.
 Performance: A basic Bayesian classifier, naïve Bayesian
classifier, has comparable performance with decision tree and
selected neural network classifiers
 Standard: Even when Bayesian methods are computationally
intractable, they can provide a standard of optimal decision
making against which other methods can be measured
 Bayes’ Theorem: Basics

 Bayes’ Theorem:

 Let X be a data sample (“evidence”): class label is unknown

 Let H be a hypothesis that X belongs to class C
 Classification is to determine P(H|X), (i.e., posteriori probability): the
probability that the hypothesis holds given the observed data sample X
 P(H) (prior probability): the initial probability
 E.g., X will buy computer, regardless of age, income, …
 P(X): probability that sample data is observed
 P(X|H) (likelihood): the probability of observing the sample X, given that
the hypothesis holds
 E.g., Given that X will buy computer, the prob. that X is 31..40, medium
income
Prediction Based on Bayes’ Theorem

 Given training data X, posteriori probability of a hypothesis H,

P(H|X), follows the Bayes’ theorem

 Informally, this can be viewed as

posteriori = likelihood x prior/evidence

 Predicts X belongs to Ci iff the probability P(Ci|X) is the highest
among all the P(Ck|X) for all the k classes
 Practical difficulty: It requires initial knowledge of many
probabilities, involving significant computational cost
Classification Is to Derive the Maximum Posteriori
 Let D be a training set of tuples and their associated class labels, and
each tuple is represented by an n-D attribute vector X = (x1, x2, …, xn)
 Suppose there are m classes C1, C2, …, Cm.
 Classification is to derive the maximum posteriori, i.e., the maximal
P(Ci|X)
 This can be derived from Bayes’ theorem

 Since P(X) is constant for all classes, only

needs to be maximized
Does patient have cancer or not?

 A patient takes a lab test and the result comes back positive. It is
known that the test returns a correct positive result in only 99% of
the cases and a correct negative result in only 95% of the cases.
Furthermore, only 0.03 of the entire population has this disease.

1. What is the probability that this patient has cancer?

2. What is the probability that he does not have cancer?
3. What is the diagnosis?
Applying Bayes’ rule:
A basic Example

The conditional probability P(effect | cause) quantifies the

relationship in the causal direction,
whereas P(cause | effect) describes the diagnostic
direction

In a task such as medical diagnosis, we often have

conditional probabilities on causal relationships (that is,
the doctor knows P(symptoms | disease)) and want to
derive a diagnosis, P(disease | symptoms).
Applying Bayes’ rule:
A basic Example

A doctor knows that

the disease meningitis causes the patient to have a stiff neck, say, 70% of the time.
The doctor also knows some unconditional facts: the prior probability
that a patient has meningitis is 1/50,000,
and the prior probability that any patient has a stiff neck is 1%.

Letting
s be the proposition that the patient has a stiff neck and
m be the proposition that the patient has meningitis, we have
Bayesian Methods
Learning and classification methods based on
probability theory.
Bayes theorem plays a critical role in probabilistic
learning and classification.
Uses prior probability of each category given no
information about an item.
Categorization produces a posterior probability
distribution over the possible categories given a
description of an item.
 Bayes Classifiers
Assumption: training set consists of instances of different classes
described cj as conjunctions of attributes values
Task: Classify a new instance d based on a tuple of attribute values
into one of the classes cj C
Key idea: assign the most probable class using Bayes
Theorem.
Naïve Bayes Classifier

 A simplified assumption: attributes are conditionally

independent (i.e., no dependence relation between attributes):

 This greatly reduces the computation cost: Only counts the class
distribution
 If A is categorical, P(x |C ) is the # of tuples in C having value
k k i i
xk for Ak divided by |Ci, D| (# of tuples of Ci in D)
 If A is continous-valued, P(x |C ) is usually computed based on
k k i
Gaussian distribution with a mean μ and standard deviation σ

and P(xk|Ci) is
Parameters estimation
P(cj)
 Can be estimated from the frequency of classes in the
training examples.
P(x1,x2,…,xn|cj)
 O(|X|n•|C|) parameters
 Could only be estimated if a very, very large number of
training examples was available.
 Independence Assumption: attribute values are conditionally
independent given the target value: naïve Bayes.
Naïve Bayes Classifier: Training Dataset

Class:
C1:buys_computer = ‘yes’
C2:buys_computer = ‘no’

Data to be classified:
X = (age <=30,
Income = medium,
Student = yes
Credit_rating = Fair)
Naïve Bayes Classifier: An Example
 P(C ): P(buys_computer = “yes”) = 9/14 = 0.643
i

P(buys_computer = “no”) = 5/14= 0.357

 Compute P(X|C ) for each class
i
P(age = “<=30” | buys_computer = “yes”) = 2/9 = 0.222
P(age = “<= 30” | buys_computer = “no”) = 3/5 = 0.6
P(income = “medium” | buys_computer = “yes”) = 4/9 = 0.444
P(income = “medium” | buys_computer = “no”) = 2/5 = 0.4
P(student = “yes” | buys_computer = “yes) = 6/9 = 0.667
P(student = “yes” | buys_computer = “no”) = 1/5 = 0.2
P(credit_rating = “fair” | buys_computer = “yes”) = 6/9 = 0.667
P(credit_rating = “fair” | buys_computer = “no”) = 2/5 = 0.4
 X = (age <= 30 , income = medium, student = yes, credit_rating = fair)
P(X|Ci) : P(X|buys_computer = “yes”) = 0.222 x 0.444 x 0.667 x 0.667 = 0.044
P(X|buys_computer = “no”) = 0.6 x 0.4 x 0.2 x 0.4 = 0.019
P(X|Ci)*P(Ci) : P(X|buys_computer = “yes”) * P(buys_computer = “yes”) = 0.028
P(X|buys_computer = “no”) * P(buys_computer = “no”) = 0.007
Therefore, X belongs to class (“buys_computer = yes”)
What is discriminant Function
For classification problem, for each class, define a
function such that we choose Ci if
K=2 Classes
Dichotomizer (K=2) vs Polychotomizer (K>2)
g(x) = g1(x) – g2(x)

Log odds:
Discriminant Functions

K decision regions R1,...,RK

Properties
Estimating instead of greatly
reduces the number of parameters (and the data
sparseness).
The learning step in Naïve Bayes consists of
estimating and based on the
frequencies in the training data
An unseen instance is classified by computing the
class that maximizes the posterior
When conditioned independence is satisfied, Naïve
Bayes corresponds to MAP classification.
 Maximum A Posterior
Based on Bayes Theorem, we can compute the Maximum A
Posterior (MAP) hypothesis for the data
We are interested in the best hypothesis for some space H
given observed training data D.

H: set of all hypothesis.

Note that we can drop P(D) as the probability of the data is constant
(and independent of the hypothesis).
Desirable Properties of Bayes Classifier
Incrementality: with each training example, the prior
and the likelihood can be updated dynamically:
flexible and robust to errors.
Combines prior knowledge and observed data: prior
probability of a hypothesis multiplied with probability
of the hypothesis given the training data
Probabilistic hypothesis: outputs not only a
classification, but a probability distribution over all
classes
Naïve Bayes Classifier: Comments

 Advantages
Easy to implement
Good results obtained in most of the cases
 Disadvantages
Assumption: class conditional independence, therefore loss of
accuracy
Practically, dependencies exist among variables
 E.g., hospitals: patients: Profile: age, family history, etc.
Symptoms: fever, cough etc., Disease: lung cancer, diabetes,
etc.
 Dependencies among these cannot be modeled by Naïve Bayes Classifier
Project Proposal Submission Instructions
Submit hard copy of project proposal on xx-xx-2023
in the class time. One copy for one group.
A short presentation (5-7 minutes) on the proposal xx-
xx-2023 Class Time to explain what has to be done.
One person is fine to present, but all members must
attend in the Q&N session.
Report has to submit with apocopate cover page with
all members information. Team also need to propose a
Group Name, e.g., Group ML Learners
Textbook/ Reference Materials

Introduction to Machine Learning by Ethem Alpaydin

Machine Learning: An Algorithmic Perspective by
Stephen Marsland
Pattern Recognition and Machine Learning by
Christopher M. Bishop