Artificial Intelligence

Probability and Bayesian Reasoning

Instructor: Dr. Sohail Iqbal

 Quote of the Day

Life is Uncertain, Still

Probability Theory tells
Your Which Life
Insurance to take!
 Example of Probability
• Problem: A spinner has 4 equal sectors
coloured yellow, blue, green and red.
• What are the chances of landing on blue after
spinning the spinner? What are the chances of
NOT landing on red?
• Solution: The chances of landing on blue are
1 in 4, or one fourth. The chances of not
landing on red are 3 in 4, or three fourth.
 Probability Of An Event

• P(A) = The Number Of Ways Event A Can Occur

The total number Of Possible Outcomes

• Example 1:

A coin has two outcomes and

One way of events happening

P(A) = 1/2
Example 2
A single 6-sided die is rolled. What is the
probability of each outcome? What is the
probability of rolling an even number? of
rolling an odd number?

P(1..6) = 1/6
P(even number) = 3/6
P(odd number) = 3/6
Example 3
• A glass jar contains 6 red, 5 green, 8 blue and 3
yellow marbles. If a single marble is chosen at
random from the jar, what is the probability of
choosing a red marble? a green marble? a blue
marble? a yellow marble?

P(red) = # of ways to choose red = 6

total # of marbles 22
P(green) = # of ways to choose green = 5
total # of marbles 22
Possible or Impossible?
• Impossible event A; P(A)=0;

Example: picking the Ace of swords out of a standard pack of

cards.

• Certain event B; P(B)=1;

Example: A teacher chooses a student at random from a class of

girls. What is the probability that the student will be a girl?

• P(X) must be between 0 and 1, both inclusive;

Sample Spaces
• A sample space is the set of all possible outcomes. The sum of all
the probabilities in the sample space is 1.

Example: What would be the sample space for the rolling of a

standard die?

{1,2,3,4,5,6} - all the possible outcomes.

Example: What about flipping two coins?

{HH, HT, TH, TT} - all outcomes denoted by (H)eads or (T)ails.

Are Sample Spaces Unique?
• Reconsidering the previous coin flipping example:
Instead of denoting the sample space using (H)eads and
(T)ails we could for example count the number of
heads in which case the sample space would be:
{0,1,2} - For example HH would be equivalent to 2 in
this sample space

So an experiment can have multiple sample spaces all of

which are technically correct depending on the
modelling choices we make.
Are all Sample Spaces as Useful?
• For the coin flipping example we have Sample Spaces
of:
{HH, HT, TH, TT} - all outcomes denoted by (H)eads or
(T)ails.

OR
{0,1,2} - For example HH would be equivalent to 2 in
this sample space.
The Compliment of an Event
• Definition: The complement of an event A is
the set of all outcomes in the sample space that
are not included in the outcomes of event A.
• The complement of event A is represented
by (read as A bar) Ā.
• Rule: Given the probability of an event, the
probability of its complement can be found by
subtracting the given probability from 1.
• P(Ā) = 1 - P(A)
 Example
• A spinner has 4 equal sectors colored yellow,
blue, green and red. What is the probability of
landing on a sector that is not green after
spinning this spinner?
Sample Space: {yellow, blue, green, red}

Probability:
P(not green) = 1 - P(green) = 1 - 1 = 3
4 4
The Compliment of an Event
• A single card is chosen at random from a standard
deck of 52 playing cards. What is the probability of
choosing a card that is not a king?

There are four kings in the sample space therefore

the probability of choosing one is 4/52.

Using the compliment rule:

P(Not King) = 1 – 4/52 = 48/52 = 12/13

Mutually Exclusive Events
• Two events are mutually exclusive if they cannot both occur at the
same time. Another word that means mutually exclusive is
disjoint.
• If two events are disjoint, then the probability of them both
occurring at the same time is 0.
• Disjoint: P(A and B) = 0
• If two events are mutually exclusive, then the probability of either
occurring is the sum of the probabilities of each occurring.

Specific Addition Rule

• Only valid when the events are mutually exclusive.

• P(A or B) = P(A) + P(B)
Mutually Exclusive Events
• Example: What is the probability of throwing a 1 or 2 using
a fair 6-sided die?
P(X=1) = 1/6
P(X=2) = 1/6
P(X=1 OR X=2) = P(X=1) + P(X=2) = 1/6 + 1/6 = 2/6
• Note that the two events are mutually exclusive as the die
can’t be in two states at the same time.
• Example: A single 6-sided die is rolled. What is the
probability of rolling a 5 or an odd number?
The number rolled can be a 5 and odd. These events are
not mutually exclusive since they can occur at the same
time.
Mutually Exclusive Events
Non-Mutually Exclusive Events
Non-Mutually Exclusive Events
• In events which aren't mutually exclusive, there is
some overlap.
• When P(A) and P(B) are added, the probability of
the intersection (and) is added twice.
• To compensate for that double addition, the
intersection needs to be subtracted.

• General Addition Rule

• Always valid.
• P(A or B) = P(A) + P(B) - P(A and B)
 Practice

• Construction of unique examples:

• Please make two sample spaces each
explaining one of the following concepts:
1. Mutually Exclusive Events
2. Mutually non-Exclusive Events
 Reading
• Set of 52 poker playing cards, must know:
– Colors
– Suits
– Face cards etc.
http://en.wikipedia.org/wiki/Playing_card
 Recap: Additive Rule
• Mutually Exclusive Events: Two events A and
B of a single experiment are said to be
mutually exclusive or disjoint if and only if they
cannot both occur at the same time.
• Example: Event of getting a Head is mutually
exclusive with the event of getting a Tail.
• Counter Example: Event of getting a Jack and
event of getting a spade in a deck of
card are not mutually exclusive.
21
 Recap: Additive Rule
• When two events A and B are mutually
exclusive, there probabilities are added.
ie. P(A or B)= P(A U B)= P(A)+P(B)

Example: We draw a card randomly from a deck.

What is the probability that it is a card of black
color or belong to diamond (suit).

22
 Observation
• To define an event, first define the sample
space.
• In obvious case of tossing coin, or rolling a die
we do not explicitly mention their sample
spaces.

23
Four suits of playing cards

Exercise: Elaborate cards’ probabilities using 3 Venn diagrams

24
 Multiplicative Rule
• Two events A and B are independent if the
occurrence of one does not change the
probability of the other occurring.
In this particular case:
P(A and B)= P(A)*P(B)

25
 Multiplicative Rule
• Example: The probability that a man will be alive
in 2050 is 3/5, and the probability that his wife
will be alive in 2050 is 2/3 (these are
independent events). Find the probability that:
1. both will be alive
2. only man will be alive
3. at least one of them will be alive
4. neither will be alive in 2050 ?

Robots Will Win Soccer's World Cup in 2050 !!! 26

 Solve the example yourself
• Answers are:

1. P(both will be alive)= 6/15

2. P(only man will be alive)= 3/15
3. P(at least one of them will be alive)= 13/15
4. P(neither will be alive in 2050)= 2/15

27
 Multiplicative Rule
• If the occurrence of one event A affects the
probability of the event B, then these events
are dependent.
Then
P(A and B)= P(A)*P(B|A)

Where P(B|A) is called the Conditional

probability of B when A is already occurred.
28
 Conditional Probability
• The probability of an event A occurring given
that another event B is already occurred is
called conditional probability (of A given B). It is
denoted by P(A|B).
P (A Ç B )
P (A B ) = , P (B ) ¹ 0
P (B )

Example: What is the probability of a card drawn

to be diamond, given that it is red?
29
 General Multiplication Rule
of Probability
• Theorem: If A and B are any events in S, then
P (A Ç B ) = P (A) ×P (B A) if P (A) ¹ 0
= P (B ) ×P (A B ) if P (B ) ¹ 0

Proof follows directly from the definition of

conditional probability.
Example: What is the probability of a card
drawn to be red as well as face card?
30
 Conditional Probability
Example: Four coins are tossed. Given that there is at
least one head. What is the (conditional) probability
of:
i. (exactly) 3 heads?
ii. At most 2 heads?
iii. No tails

Practice 2: Solve the above example by using sample

space and then by using the conditional probability.

31
Any Questions till here?
 Multiplicative Rule
• If the occurrence of one event A affects the
probability of the event B, then these events
are dependent.
Then
P(A and B)= P(A)*P(B|A)

Where P(B|A) is called the Conditional

probability of B when A is already occurred.
33
 Conditional Probability
• The probability of an event A occurring given
that another event B is already occurred is
called conditional probability (of A given B). It is
denoted by P(A|B).
P (A Ç B )
P (A B ) = , P (B ) ¹ 0
P (B )

Example: What is the probability of a card drawn

to be diamond, given that it is red?
34
 General Multiplication Rule
of Probability
• Theorem: If A and B are any events in S, then
P (A Ç B ) = P (A) ×P (B A) if P (A) ¹ 0
= P (B ) ×P (A B ) if P (B ) ¹ 0

Proof follows directly from the definition of

conditional probability.
Example: What is the probability of a card
drawn to be red as well as face card?
35
 Partition of a Set
• A collection of disjoint subsets of a given set.
The union of the subsets must equal the entire
original set.
• For example, one possible partition of {1, 2, 3,
4, 5, 6} is {1, 3}, {2}, {4, 5, 6}.
• If events A1, A2,…., Ak are mutually exclusive
and their union is entire space S, that they
form partition of S.

36
Four different partitions

37
Initial form of Bayes’ Theorem
Since

P (A Ç B ) = P (A) ×P (B A) if P (A) ¹ 0
= P (B ) ×P (A B ) if P (B ) ¹ 0

Therefore:

P (B ) ×P (A B ) = P (A) ×P (B A)

P (A) ×P (B A)
Þ P (A B ) =
P (B )
 Bayes’ Theorem (Full Form)

• Events A1 , A2 ,… Ak form a partition of a

sample space S,
• B is any other event of S such that it can occur
only if one of Ai occurs, then for any i,

P (Ai )P (B | Ai )
P (Ai | B ) = for i = 1,2,..., k
k

å P (Ai )P (B | Ai )
i =1

39
 Example (Bayes’ theorem)
• Four technicians regularly make repairs when
breakdowns occur on an automated production line.
Janet, who services 20% of the breakdowns, makes a
bad repair 1 time in 20; Tom who services 60% of the
breakdowns, makes a bad repair 1 time in 10;
Georgia, who services 16% of the breakdowns, makes
a bad repair 1 time in 10; and Victor, who services 4%
of the breakdowns, makes bad repair 1 time in 20.
For a problem diagnosed as being due to a bad initial
repair, find the probability that the initial repair was
made by Janet.
 Example (Bayes’ theorem)
• Let B be the event that the initial repair was
bad, J that the initial repair was made by
Janet, T it was made by Tom, G it was made by
Georgia, and V that it was done by Victor.

• Solution: Substitute value in Bayes’ theorm

(extended form)
 Bayes formula after substitutions

P (J )P (B | J )
P (J | B ) =
P (J )P (B | J ) + P (T )P (B | T ) + P (G )P (B | G ) + P (V )P (B | V )

For deeper understanding of the formula, lets introduce

the concept of “Contribution in the Bad Repair” (CBR)

42
 Contributions
• CBR by Janet = P(J)*P(B|J)=10/1000
• CBR by Tom = P(T)*P(B|T)=60/1000
• CBR by Georgia= P(G)*P(B|G)=16/1000
• CBR by Victor = P(V)*P(B|V)=2/1000
• CBR by all= 88/1000
Thus
P(J|B)=CBR by Janet/CBR by all= 10/88

43
 Practice
A cab was involved in a hit and run accident at night.
Two cab companies, the Green and the Blue, operate
in the city. The following facts are known; 85% of the
cabs in the city are Green and 15% are Blue.
A Witness identified the cab as Blue. The court
tested the reliability of the witness under the same
circumstances that existed on the night of the
accident and concluded that the witness correctly
identified each one of the two colours 80% of the
time and failed 20% of the time. What is the
probability that the cab involved in the accident was
actually Blue? 44
 Conclusion
• Bayes’ theorem/ Bayesian reason provides us
the posterior probabilities, based on the prior
probabilities, unveiling the most probable
culprit behind a cab accident/ bad repair.

• Bayes theorem will provide us reasoning to

understand Hidden Markov Models.
Questions

Thank You!