Conditional Probability and Bayes Theorem

Supervisor: Dr.Shaista Rais

DCS-UOK

Copyright © 2010 Pearson Education, Inc.

The General Addition Rule
 When two events A and B are disjoint, we can
use the addition rule for disjoint events from
Chapter 14:
P(A  B) = P(A) + P(B)
 However, when our events are not disjoint, this
earlier addition rule will double count the
probability of both A and B occurring. Thus, we
need the General Addition Rule.
 Let’s look at a picture…

Copyright © 2010 Pearson Education, Inc.

The General Addition Rule (cont.)

 General Addition Rule:

 For any two events A and B,

P(A  B) = P(A) + P(B) – P(A  B)

 The following Venn diagram shows a situation in
which we would use the general addition rule:

Copyright © 2010 Pearson Education, Inc.

 When we want the probability of an event from a
conditional distribution, we write P(B|A) and
pronounce it “the probability of B given A.”
 A probability that takes into account a given
condition is called a conditional probability.

Copyright © 2010 Pearson Education, Inc.

 To find the probability of the event B given the
event A, we restrict our attention to the outcomes
in A. We then find the fraction of those outcomes
B that also occurred.

P(B|A) P(A  B)
P(A)
 Note: P(A) cannot equal 0, since we know that A
has occurred.

Copyright © 2010 Pearson Education, Inc.

The General Multiplication Rule
 When two events A and B are independent, we
can use the multiplication rule for independent
events from Chapter 14:
P(A  B) = P(A) x P(B)
 However, when our events are not independent,
this earlier multiplication rule does not work.
Thus, we need the General Multiplication Rule.

Copyright © 2010 Pearson Education, Inc.

The General Multiplication Rule (cont.)
 We encountered the general multiplication rule in
the form of conditional probability.
 Rearranging the equation in the definition for
conditional probability, we get the General
Multiplication Rule:
 For any two events A and B,

P(A  B) = P(A)  P(B|A)

or
P(A  B) = P(B)  P(A|B)

Copyright © 2010 Pearson Education, Inc.

Independence
 Independence of two events means that the
outcome of one event does not influence the
probability of the other.
 With our new notation for conditional
probabilities, we can now formalize this definition:
 Events A and B are independent whenever

P(B|A) = P(B). (Equivalently, events A and B

are independent whenever P(A|B) = P(A).)

Copyright © 2010 Pearson Education, Inc.

Independent ≠ Disjoint
 Disjoint events cannot be independent! Well, why not?
 Since we know that disjoint events have no outcomes

in common, knowing that one occurred means the

other didn’t.
 Thus, the probability of the second occurring changed

based on our knowledge that the first occurred.

 It follows, then, that the two events are not

independent.
 A common error is to treat disjoint events as if they were
independent, and apply the Multiplication Rule for
independent events—don’t make that mistake.

Copyright © 2010 Pearson Education, Inc.

Depending on Independence
 It’s much easier to think about independent
events than to deal with conditional probabilities.
 It seems that most people’s natural intuition for
probabilities breaks down when it comes to
conditional probabilities.
 Don’t fall into this trap: whenever you see
probabilities multiplied together, stop and ask
whether you think they are really independent.

Copyright © 2010 Pearson Education, Inc.

Drawing Without Replacement
 Sampling without replacement means that once one
individual is drawn it doesn’t go back into the pool.
 We often sample without replacement, which doesn’t

matter too much when we are dealing with a large

population.
 However, when drawing from a small population, we

need to take note and adjust probabilities accordingly.

 Drawing without replacement is just another instance of
working with conditional probabilities.

Copyright © 2010 Pearson Education, Inc.

Tree Diagrams
 A tree diagram helps us think through conditional
probabilities by showing sequences of events as
paths that look like branches of a tree.
 Making a tree diagram for situations with
conditional probabilities is consistent with our
“make a picture” mantra.

Copyright © 2010 Pearson Education, Inc.

Tree Diagrams (cont.)
 Figure 15.5 is a nice
example of a tree
diagram and shows how
we multiply the
probabilities of the
branches together.
 All the final outcomes
are disjoint and must
add up to one.
 We can add the final
probabilities to find
probabilities of
compound events.
Copyright © 2010 Pearson Education, Inc.
Reversing the Conditioning
 Reversing the conditioning of two events is rarely intuitive.
 Suppose we want to know P(A|B), and we know only
P(A), P(B), and P(B|A).
 We also know P(A  B), since
P(A  B) = P(A) x P(B|A)
 From this information, we can find P(A|B):

P(A|B) P(A  B)
P(B)

Copyright © 2010 Pearson Education, Inc.

Bayes’s Rule

 When we reverse the probability from the

conditional probability that you’re originally given,
you are actually using Bayes’s Rule.

P A | B P B 
P B | A  
P A | B P B   P A | B P B
C
C
 

Copyright © 2010 Pearson Education, Inc.

 A problem

P(Test result|Event) Outcome 1

+
The data are + Test
- Outcome 2
organized like Event
this: + Outcome 3
- Test
Outcome 4
-
P(Event|Test result) Outcome 1
+
But to make a + Event
- Outcome 2
decision you Test
need: + Outcome 3
- Event
Outcome 4
-
Copyright © 2010 Pearson Education, Inc. 16
 Bayes Theorem flips probabilities
“A” is the event we are
Conditional probability interested in, e.g., a
disease. “X” is the
 P(A|X) means probability of A given X evidence, e.g., a test
result.
PA and X   P( X | A )P( A )  P( A | X)P( X)

By symmetry: P(X|A)P(A) = P(A|X)P(X)

P( X | A )P( A )
P( A | X) 
See next two slides
for expansion of P(X)
P( X)
in denominator.
Bayes theorem A means
P( X | A )P( A ) “not A”
P A | X  
P( X | A )P( A )  P( X | A )P( A )
Copyright © 2010 Pearson Education, Inc.
Expansion of P(A)

A
X

Occurrence of X includes events “X and A” and

“X and not A”
Copyright © 2010 Pearson Education, Inc. 18
 Expansion of P(A) for the denominator of
Bayes Theorem

P X   P( X and A )  P( X and A )
P X and A   P( X | A )P( A )
PX and A   P( X | A )P( A )
Therefore…
P( X)  P( X | A )P( A )  P( X | A )P( A )
Bayes theorem:
P( X | A )P( A )
P A | X  
P( X | A )P( A )  P( X | A )P( A )

19
Copyright © 2010 Pearson Education, Inc.
Likelihood ratio form

P( A | X)  P( A )   P( X | A ) 
   
P( A | X)  P( A )   P( X | A ) 

Copyright © 2010 Pearson Education, Inc. 20

 Bayesian belief updating

p0 is the prior probability of the event A

Odds0 = p0/(1-p0) are the prior odds in favor of A

lrX = P(X|A)/P(X|not A) is the likelihood ratio for new data X

Odds1 = [p0/(1-p0)]*lrX are the posterior odds in favor of A

p1 = Odds1/(1 + Odds1) = posterior probability of event A

= p(A|X)

Copyright © 2010 Pearson Education, Inc. 21

Dealing with causality
 Case 1
 X represents some data or evidence. It

provides a clue to whether event A will occur

(prediction) or has or is occurring (diagnosis). It
is not causal.
 Examples
 Doctor observes redness in the ear

 Social worker observes unclean house.

Copyright © 2010 Pearson Education, Inc. 22

Dealing with causality
 Case 2
 X represents some event that influences the

likelihood that event A will occur. It has a causal

influence
 Examples
 Federal Reserve chairman says tax cut not a

good idea.
 Asian stock markets fall sharply.

Copyright © 2010 Pearson Education, Inc. 23

What is easier to estimate?
 Both require prior probability of A
 Case 1
 Probability of X given A

 Probability of X given not A

 Case 2
 Probability of A given X

 Impact multiplier (Odds multiplier)

Copyright © 2010 Pearson Education, Inc. 24

What Can Go Wrong?
 Don’t use a simple probability rule where a
general rule is appropriate:
 Don’t assume that two events are independent
or disjoint without checking that they are.
 Don’t find probabilities for samples drawn without
replacement as if they had been drawn with
replacement.
 Don’t reverse conditioning naively.
 Don’t confuse “disjoint” with “independent.”

Copyright © 2010 Pearson Education, Inc.