UNIT II PROBABILISTIC REASONING

Acting under uncertainty, Bayesian inference, naïve bayes models,

Probabilistic reasoning, Bayesian Networks, exact inference in BN,
approximate inference in BN – causal networks.
Bayes' theorem in Artificial intelligence

• Bayes' theorem:
• Bayes' theorem is also known as Bayes' rule, Bayes' law, or Bayesian
reasoning, which determines the probability of an event with uncertain
knowledge.
• In probability theory, it relates the conditional probability and marginal
probabilities of two random events.
• Bayes' theorem was named after the British mathematician Thomas
Bayes. The Bayesian inference is an application of Bayes' theorem,
which is fundamental to Bayesian statistics.
• It is a way to calculate the value of P(B|A) with the knowledge of P(A|B).
 Bayesian inference
• Bayes' theorem allows updating the probability prediction of an event
by observing new information of the real world.
• Example: If cancer corresponds to one's age then by using Bayes'
theorem, we can determine the probability of cancer more accurately
with the help of age.
• Bayes' theorem can be derived using product rule and conditional
probability of event A with known event B:
• As from product rule we can write:P(A ⋀ B)= P(A|B) P(B) or
• Similarly, the probability of event B with known event A:
• P(A ⋀ B)= P(B|A) P(A)
• Equating right hand side of both the equations, we will get:
The above equation (a) is called as Bayes' rule or Bayes' theorem. This equation is basic of
most modern AI systems for probabilistic inference.
• It shows the simple relationship between joint and conditional probabilities. Here,
• P(A|B) is known as posterior, which we need to calculate, and it will be read as Probability of
hypothesis A when we have occurred an evidence B.
• P(B|A) is called the likelihood, in which we consider that hypothesis is true, then we calculate
the probability of evidence.
• P(A) is called the prior probability, probability of hypothesis before considering the evidence
• P(B) is called marginal probability, pure probability of an evidence.
• In the equation (a), in general, we can write P (B) = P(A)*P(B|Ai), hence the Bayes' rule can
be written as:
nce.
 Applying Bayes' rule:
• Where A1, A2, A3,........, An is a set of mutually exclusive and exhaustive
events.
• Applying Bayes' rule:
• Bayes' rule allows us to compute the single term P(B|A) in terms of
P(A|B), P(B), and P(A). This is very useful in cases where we have a
good probability of these three terms and want to determine the fourth
one. Suppose we want to perceive the effect of some unknown cause,
and want to compute that cause, then the Bayes' rule becomes:
 Applying Bayes' rule: .

Examples
• Example-1:
• Question: what is the probability that a patient has diseases meningitis with a stiff neck?
• Given Data:
•A doctor is aware that disease meningitis causes a patient to have a stiff neck, and it occurs 80% of
the time. He is also aware of some more facts, which are given as follows:
•The Known probability that a patient has meningitis disease is 1/30,000.
•The Known probability that a patient has a stiff neck is 2%.
Let a be the proposition that patient has stiff neck and b be the proposition that patient has meningitis. ,
so we can calculate the following as:
P(a|b) = 0.8
P(b) = 1/30000
P(a)= .02
Hence, we can assume that 1 patient out of 750 patients has meningitis disease with a stiff
neck
• Example-2:
• Question: From a standard deck of playing cards, a single card is
drawn. The probability that the card is king is 4/52, then calculate
posterior probability P(King|Face), which means the drawn face card
is a king card.
• Solution:
• P(king): probability that the card is King= 4/52= 1/13
• P(face): probability that a card is a face card= 3/13
• P(Face|King): probability of face card when we assume it is a king = 1
• Putting all values in equation (i) we will get:
Application of Bayes' theorem in Artificial intelligence:

• Following are some applications of Bayes' theorem:

• It is used to calculate the next step of the robot when the already
executed step is given.
• Bayes' theorem is helpful in weather forecasting.
• It can solve the Monty Hall problem.
Naïve Bayes Classifier Algorithm

• Naïve Bayes algorithm is a supervised learning algorithm, which is based

on Bayes theorem and used for solving classification problems.
• It is mainly used in text classification that includes a high-dimensional
training dataset.
• Naïve Bayes Classifier is one of the simple and most effective Classification
algorithms which helps in building the fast machine learning models that
can make quick predictions.
• It is a probabilistic classifier, which means it predicts on the basis of the
probability of an object.
• Some popular examples of Naïve Bayes Algorithm are spam filtration,
Sentimental analysis, and classifying articles.
 Naïve Bayes

• Why is it called Naïve Bayes?

• The Naïve Bayes algorithm is comprised of two words Naïve and Bayes,
Which can be described as:
• Naïve: It is called Naïve because it assumes that the occurrence of a
certain feature is independent of the occurrence of other features. Such
as if the fruit is identified on the bases of color, shape, and taste, then
red, spherical, and sweet fruit is recognized as an apple. Hence each
feature individually contributes to identify that it is an apple without
depending on each other.
• Bayes: It is called Bayes because it depends on the principle of
Bayes' Theorem.
 Bayes' Theorem
• Bayes' theorem is also known as Bayes' Rule or Bayes' law, which is used to
determine the probability of a hypothesis with prior knowledge. It depends on the
conditional probability.
• The formula for Bayes' theorem is given as:

• Where, P(A|B) is Posterior probability: Probability of hypothesis A on the observed

event B.
• P(B|A) is Likelihood probability: Probability of the evidence given that the
probability of a hypothesis is true.
• P(A) is Prior Probability: Probability of hypothesis before observing the evidence.
• P(B) is Marginal Probability: Probability of Evidence.
Working of Naïve Bayes' Classifier:
• Working of Naïve Bayes' Classifier can be understood with the help of
the below example:
• Suppose we have a dataset of weather conditions and corresponding
target variable "Play". So using this dataset we need to decide that
whether we should play or not on a particular day according to the
weather conditions. So to solve this problem, we need to follow the
below steps:
• Convert the given dataset into frequency tables.
• Generate Likelihood table by finding the probabilities of given features.
• Now, use Bayes theorem to calculate the posterior probability.
• Problem: If the weather is sunny, then the Player should play or not?
• Solution: To solve this, first consider the below dataset:
Play
Outlook

0 Rainy Yes

1 Sunny Yes

2 Overcast Yes

3 Overcast Yes

4 Sunny No

5 Rainy Yes

6 Sunny Yes

7 Overcast Yes

8 Rainy No

9 Sunny No

10 Sunny Yes

11 Rainy No

12 Overcast Yes

13 Overcast Yes
• Frequency table for the Weather Conditions:

Weather Yes No
Overcast 5 0
Rainy 2 2
Sunny 3 2
Total 10 5

Likelihood table weather condition:

Weather No Yes
Overcast 0 5 5/14= 0.35
Rainy 2 2 4/14=0.29
Sunny 2 3 5/14=0.35
All 4/14=0.29 10/14=0.71
• Applying Bayes'theorem:
• P(Yes|Sunny)= P(Sunny|Yes)*P(Yes)/P(Sunny)
• P(Sunny|Yes)= 3/10= 0.3
• P(Sunny)= 0.35
• P(Yes)=0.71
• So P(Yes|Sunny) = 0.3*0.71/0.35= 0.60
• P(No|Sunny)= P(Sunny|No)*P(No)/P(Sunny)
• P(Sunny|NO)= 2/4=0.5
• So P(No|Sunny)= 0.5*0.29/0.35 = 0.41
• So as we can see from the above calculation that P(Yes|Sunny)>P(No|Sunny)
• Hence on a Sunny day, Player can play the game.
• Advantages of Naïve Bayes Classifier:
• Naïve Bayes is one of the fast and easy ML algorithms to predict a
class of datasets.
• It can be used for Binary as well as Multi-class Classifications.
• It performs well in Multi-class predictions as compared to the other
Algorithms.
• It is the most popular choice for text classification problems.
• Disadvantages of Naïve Bayes Classifier:
• Naive Bayes assumes that all features are independent or unrelated,
so it cannot learn the relationship between features.
• Applications of Naïve Bayes Classifier:
• It is used for Credit Scoring.
• It is used in medical data classification.
• It can be used in real-time predictions because Naïve Bayes Classifier is an eager learner.
• It is used in Text classification such as Spam filtering and Sentiment analysis.
• Types of Naïve Bayes Model:
• There are three types of Naive Bayes Model, which are given below:
• Gaussian: The Gaussian model assumes that features follow a normal distribution. This
means if predictors take continuous values instead of discrete, then the model assumes
that these values are sampled from the Gaussian distribution.
• Multinomial: The Multinomial Naïve Bayes classifier is used when the data is multinomial
distributed. It is primarily used for document classification problems, it means a particular
document belongs to which category such as Sports, Politics, education, etc.
The classifier uses the frequency of words for the predictors.
• Bernoulli: The Bernoulli classifier works similar to the Multinomial classifier, but the
predictor variables are the independent Booleans variables. Such as if a particular word is
present or not in a document. This model is also famous for document classification tasks.
• Python Implementation of the Naïve Bayes algorithm:
• Now we will implement a Naive Bayes Algorithm using Python. So for
this, we will use the "user_data" dataset, which we have used in our
other classification model. Therefore we can easily compare the Naive
Bayes model with the other models.
• Steps to implement:
• Data Pre-processing step
• Fitting Naive Bayes to the Training set
• Predicting the test result
• Test accuracy of the result(Creation of Confusion matrix)
• Visualizing the test set result.
• Data Pre-processing step:
• In this step, we will pre-process/prepare the data so that we can use it efficiently in our code. It is similar as
we did in data-pre-processing. The code for this is given below:
• Importing the libraries
• import numpy as nm
• import matplotlib.pyplot as mtp
• import pandas as pd

• # Importing the dataset

• dataset = pd.read_csv('user_data.csv')
• x = dataset.iloc[:, [2, 3]].values
• y = dataset.iloc[:, 4].values
•
• # Splitting the dataset into the Training set and Test set
• from sklearn.model_selection import train_test_split
• x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.25, random_state = 0)
•
• # Feature Scaling
• from sklearn.preprocessing import StandardScaler
• sc = StandardScaler()
• x_train = sc.fit_transform(x_train)
• x_test = sc.transform(x_test)
• In the above code, we have loaded the dataset into our program using
"dataset = pd.read_csv('user_data.csv'). The loaded dataset is
divided into training and test set, and then we have scaled the feature
variable.
• The output for the dataset is given as:
• Fitting Naive Bayes to the Training Set:
• After the pre-processing step, now we will fit the Naive Bayes model to
the Training set. Below is the code for it:
• # Fitting Naive Bayes to the Training set
• from sklearn.naive_bayes import GaussianNB
• classifier = GaussianNB()
• classifier.fit(x_train, y_train)
• In the above code, we have used the GaussianNB classifier to fit it to the
training dataset. We can also use other classifiers as per our
requirement.

• Output:

• Out[6]: GaussianNB(priors=None, var_smoothing=1e-09)

• Prediction of the test set result:
• Now we will predict the test set result. For this, we will create a new
predictor variable y_pred, and will use the predict function to make
the predictions.
• # Predicting the Test set results
• y_pred = classifier.predict(x_test)
• Output:
• The above output shows the result for prediction vector y_pred and
real vector y_test. We can see that some predications are different
from the real values, which are the incorrect predictions.
• Creating Confusion Matrix:
• Now we will check the accuracy of the Naive Bayes classifier using the
Confusion matrix. Below is the code for it:
• # Making the Confusion Matrix
• from sklearn.metrics import confusion_matrix
• cm = confusion_matrix(y_test, y_pred)
Output:

Naïve Bayes Classifier Algorithm 3

As we can see in the above confusion matrix output, there are 7+3= 10 incorrect predictions, and 65+25=90 correct predictions.

5) Visualizing the training set result:

Next we will visualize the training set result using Naïve Bayes Classifier. Below is the code for it:

# Visualising the Training set results

from matplotlib.colors import ListedColormap
x_set, y_set = x_train, y_train
X1, X2 = nm.meshgrid(nm.arange(start = x_set[:, 0].min() - 1, stop = x_set[:, 0].max() + 1, step = 0.01),
nm.arange(start = x_set[:, 1].min() - 1, stop = x_set[:, 1].max() + 1, step = 0.01))
mtp.contourf(X1, X2, classifier.predict(nm.array([X1.ravel(), X2.ravel()]).T).reshape(X1.shape),
alpha = 0.75, cmap = ListedColormap(('purple', 'green')))
mtp.xlim(X1.min(), X1.max())
mtp.ylim(X2.min(), X2.max())
for i, j in enumerate(nm.unique(y_set)):
mtp.scatter(x_set[y_set == j, 0], x_set[y_set == j, 1],
c = ListedColormap(('purple', 'green'))(i), label = j)
mtp.title('Naive Bayes (Training set)')
mtp.xlabel('Age')
mtp.ylabel('Estimated Salary')
mtp.legend()
Visualizing the Test set result:

• # Visualising the Test set results

• from matplotlib.colors import ListedColormap
• x_set, y_set = x_test, y_test
• X1, X2 = nm.meshgrid(nm.arange(start = x_set[:, 0].min() - 1, stop = x_set[:, 0].max() + 1, step = 0.01),
• nm.arange(start = x_set[:, 1].min() - 1, stop = x_set[:, 1].max() + 1, step = 0.01))
• mtp.contourf(X1, X2, classifier.predict(nm.array([X1.ravel(), X2.ravel()]).T).reshape(X1.shape),
• alpha = 0.75, cmap = ListedColormap(('purple', 'green')))
• mtp.xlim(X1.min(), X1.max())
• mtp.ylim(X2.min(), X2.max())
• for i, j in enumerate(nm.unique(y_set)):
• mtp.scatter(x_set[y_set == j, 0], x_set[y_set == j, 1],
• c = ListedColormap(('purple', 'green'))(i), label = j)
• mtp.title('Naive Bayes (test set)')
• mtp.xlabel('Age')
• mtp.ylabel('Estimated Salary')
• mtp.legend()
• mtp.show()
• Output:The above output is final output for test set data. As we can see the classifier has created a Gaussian curve to divide the
"purchased" and "not purchased" variables. There are some wrong predictions which we have calculated in Confusion matrix.
But still it is pretty good classifier.
 Exact inference in BN

• Naïve Bayes Classifier Algorithm

• Naïve Bayes algorithm is a supervised learning algorithm, which is based on Bayes
theorem and used for solving classification problems.
• It is mainly used in text classification that includes a high-dimensional training
dataset.
• Naïve Bayes Classifier is one of the simple and most effective Classification
algorithms which helps in building the fast machine learning models that can make
quick predictions.
• It is a probabilistic classifier, which means it predicts on the basis of the
probability of an object.
• Some popular examples of Naïve Bayes Algorithm are spam filtration, Sentimental
analysis, and classifying articles.