Probability and Statistics

Probability
Conditional Probability
Lecture 8

Tauseef Iftikhar

Government College University

1/38
Today’s Agenda

Conditional Probability
Multiplication Theorem of Probability
Law of Total Probability
Bayes’ Theorem

Independence

2/38
Conditional Probability

Table: Absolute frequencies of test results and infection status

Infection Total(rows)
Present Absent
Test Positive(+) 30 10 40
Negative(−) 15 45 60
Total(column) 45 55 100

I There is infection and the test diagnoses it, i.e. the test is
correctly diagnosing the infection.
I There is no infection and the test does not diagnose it, i.e.
the test is correctly diagnosing that there is no infection.
I There is an infection and the test does not diagnose it, i.e.
the test is incorrect in stating that there is no infection.
I There is no infection but the test diagnoses it, i.e. the test is
incorrect in stating that there is an infection.
3/38
Conditional Probability
I Relative frequencies as probabilities – assumption n is larger
than 100

Table: Relative frequencies of test results and infection status

Infection Total(rows)
Present Absent
Test Positive(+) 0.30 0.10 0.40
Negative(−) 0.15 0.45 0.60
Total(column) 0.45 0.55 1.0

I What is the probability that a test is positive P(T +)?

I What is the probability that infection is presentP(IP)
I If one already know that the test is positive then what is the
probability that infection is present P(IP|T +)?
4/38
Conditional Probability
I Laplace probability – assumption that all simple events in
Ω = {ω1 , ω2 , . . . , ωk } are equally probable,
i.e.P(ωj ) = k1 , j = 1, 2, , . . . , k.
I Let A and B be two events containing nA and nB numbers of
simple events.
I Let further A ∩ B contain nAB numbers of simple events.
I Laplace Probability

P(A) =?, P(B) =? and P(A ∩ B) =?

5/38
Conditional Probability
I Laplace probability – assumption that all simple events in
Ω = {ω1 , ω2 , . . . , ωk } are equally probable,
i.e.P(ωj ) = k1 , j = 1, 2, , . . . , k.
I Let A and B be two events containing nA and nB numbers of
simple events.
I Let further A ∩ B contain nAB numbers of simple events.
I Laplace Probability
nA nB nAB
P(A) = , P(B) = and P(A ∩ B) = .
k k k

6/38
Conditional Probability
I Laplace probability – assumption that all simple events in
Ω = {ω1 , ω2 , . . . , ωk } are equally probable,
i.e.P(ωj ) = k1 , j = 1, 2, , . . . , k.
I Let A and B be two events containing nA and nB numbers of
simple events.
I Let further A ∩ B contain nAB numbers of simple events.
I Laplace Probability
nA nB nAB
P(A) = , P(B) = and P(A ∩ B) = .
k k k
I Assume we have prior knowledge that A has occurred. Then
conditional probability of event B i.e., P(B|A) =?

P(A ∩ B)
P(B|A) = (1)
P(A)

where P(A) > 0. The role of A and B can be interchanged.

7/38
Conditional Probability

8/38
Conditional Probability

9/38
Conditional Probability

P(A|B) = Probability of A given B has occurred.

P(A|B) = P(A∩B)
P(B)
10/38
Conditional Probability

I Rolling of two dice.

6
5
4
I Each outcome is equally
3
likely, 1/36
2
1
1 2 3 4 5 6
Figure: Finite sample space.

11/38
Conditional Probability

I Rolling of two dice.

6
5
I Each outcome is equally
4
likely, 1/36
3
2
I Let B be the event,
min(X,Y) = 3 1
1 2 3 4 5 6
Figure: Finite sample space.

12/38
Conditional Probability

I Rolling of two dice.

6
I Each outcome is equally 5
likely, 1/36
4
I Let B be the event, 3
min(X,Y) = 3
2
I Let M = max(X , Y ) 1

I P(M = 1|B) = 1 2 3 4 5 6
Figure: Finite sample space.

13/38
Conditional Probability

I Rolling of two dice.

6
I Each outcome is equally 5
likely, 1/36
4
I Let B be the event, 3
min(X,Y) = 3
2
I Let M = max(X , Y ) 1

I P(M = 1|B) = 0 1 2 3 4 5 6
Figure: Finite sample space.

14/38
Conditional Probability

I Rolling of two dice.

6
I Each outcome is equally
5
likely, 1/36
4
I Let B be the event,
min(X,Y) = 3 3

I Let M = max(X , Y ) 2
1
I P(M = 1|B) = 0
1 2 3 4 5 6
I P(M = 3|B) =
Figure: Finite sample space.

15/38
Conditional Probability

I Rolling of two dice.

6
I Each outcome is equally
5
likely, 1/36
4
I Let B be the event,
min(X,Y) = 3 3

I Let M = max(X , Y ) 2

I P(M = 1|B) = 0 1

P(M=3 ∩ B)
1 2 3 4 5 6
I P(M = 3|B) = P(B)
Figure: Finite sample space.

16/38
Conditional Probability

I Rolling of two dice.

6
I Each outcome is equally
5
likely, 1/36
4
I Let B be the event,
min(X,Y) = 3 3

I Let M = max(X , Y ) 2

I P(M = 1|B) = 0 1

1/36
1 2 3 4 5 6
I P(M = 3|B) = 1
7/36 = 7
Figure: Finite sample space.

17/38
Conditional Probabilities

I Event A: Airplane is flying

above

I Event B: Something
registers on radar screen

I P(A ∩ B) =

18/38
Conditional Probabilities

I Event A: Airplane is flying

above

I Event B: Something
registers on radar screen

I P(A ∩ B) = P(A)P(B|A)

19/38
Conditional Probabilities

I Event A: Airplane is flying

above

I Event B: Something
registers on radar screen

I P(A ∩ B) =
P(A)P(B|A) = 0.0495

20/38
Conditional Probabilities

I Event A: Airplane is flying

above

I Event B: Something
registers on radar screen

I P(A ∩ B) =
P(A)P(B|A) = 0.0495

I P(B) = P(A)P(B|A) +
P(A0 )P(B|A0 )

21/38
Conditional Probabilities

I Event A: Airplane is flying

above
I Event B: Something
registers on radar screen
I P(A ∩ B) =
P(A)P(B|A) = 0.0495
I P(B) = P(A)P(B|A) +
P(A0 )P(B|A0 ) =
0.05 × 0.99 + 0.95 × 0.1

22/38
Conditional Probabilities

I Event A: Airplane is flying

above
I Event B: Something
registers on radar screen
I P(A ∩ B) =
P(A)P(B|A) = 0.0495
I P(B) = P(A)P(B|A) +
P(A0 )P(B|A0 ) =
0.05 × 0.99 + 0.95 × 0.1
I P(A|B) =

23/38
Conditional Probabilities

I Event A: Airplane is flying

above
I Event B: Something
registers on radar screen
I P(A ∩ B) =
P(A)P(B|A) = 0.0495
I P(B) = P(A)P(B|A) +
P(A0 )P(B|A0 ) =
0.05 × 0.99 + 0.95 × 0.1
I P(A|B) = P(A∩B)
P(B) = 0.34

24/38
Multiplication Theorem of Probability
Theorem 1 (Multiplication Theorem of Probability)
For two arbitrary events A and B, the following holds:

P(A ∩ B) = P(B)P(A|B) = P(A)P(B|A) (2)

More generally

P(A ∩ B ∩ C ) = P(A)P(B|A)P(C |A ∩ B) (3)

26/38
Multiplication Theorem of Probability
∩ B) A∩B ∩C
P (C |A

|A) A∩B
P (B
P (C 0
|A ∩
B) A ∩ B ∩ C0
A
0
P (B 0 P (C |A
∩B ) A ∩ B0 ∩ C
|A)
P(A) A∩ B0
P (C 0
|A ∩
B 0) A ∩ B0 ∩ C 0

P (C |A
0 ∩ B) A0 ∩ B ∩ C
P(A0 ) 0) A0 ∩B
|A
P (B
P (C 0 0
|A ∩
B) A0 ∩ B ∩ C 0
A0
0
P (B 0 0 ∩B ) A0 ∩ B 0 ∩ C
P (C |A
|A 0)
A0 ∩ B0
P (C 0 0
|A ∩
B 0) A0 ∩ B 0 ∩ C 0

27/38
Law of Total Probability
Theorem 2 (Law of Total Probability)
Assume that A1 , A2 , . . . , Am are events such that m
S
i=1 Ai = Ω and
Ai ∪ Aj = ∅, ∀i 6= j, P(Ai ) > 0, S ∀i, i.e., A1 , A2 , . . . , Am form a
complete decomposition of Ω = m i=1 Ai in pairwise disjoint events,
then the probability of an event B can be calculated as
m
X
P(B) = P(B|Ai )P(Ai ). (4)
i=1

29/38
Law of Total Probability
Theorem 3 (Law of Total Probability)
Assume that A1 , A2 , . . . , Am are events such that m
S
i=1 Ai = Ω and
Ai ∪ Aj = ∅, ∀i 6= j, P(Ai ) > 0, S ∀i, i.e., A1 , A2 , . . . , Am form a
complete decomposition of Ω = m i=1 Ai in pairwise disjoint events,
then the probability of an event B can be calculated as
m
X
P(B) = P(B|Ai )P(Ai ). (5)
i=1

30/38
Bayes’ Theorem
Bayes Theorem gives a connection between P(A|B) and P(B|A).
For events A and B with P(A) > 0 and P(B) > 0, using (1) and
(3), we get

P(A ∩ B) P(A ∩ B)P(A)

P(A|B) = =
P(B) P(A)P(B)
P(B|A)P(A)
= (6)
P(B)

Let A1 , A2 , . . . , Am are events such that m

S
i=1 Ai = Ω and
Ai ∪ Aj = ∅, ∀i 6= j, P(Ai ) > 0, ∀i, and B is another event than
A, then using (5) and (6), we get

P(B|Aj )P(Aj )
P(Aj |B) = P (7)
i P(B|Ai )P(Ai )

Prior probability P(Aj ), Model Probability P(B|Aj ), Posterior

Probability P(Aj |B)
32/38
Bayes’ Theorem
Example 1
Suppose someone rents movies from two different DVD stores.
Sometimes it happens that the DVD does not work because of
scratches. We consider the following events: Ai (i = 1, 2): ”the
DVD is rented from store i”. Further let B denote the event that
the DVD is working without any problems. Assume we know that
P(A1 ) = 0.6 and P(A2 ) = 0.4 (note that A2 = Ā1 ) and
P(B|A1 ) = 0.95, P(B|A2 ) = 0.75.

1. What is the probability that a rented DVD works fine?

P(B) = P(B|A1 )P(A1 ) + P(B|A2 )P(A2 )

2. What is the probability that movie is rented from store 1 and

working
P(B ∩ A1 ) = P(B|A1 )P(A1 )

33/38
Bayes’ Theorem
3. If we have a properly working DVD what is the probability
that it is from store 1.
P(A1 ∩ B)
P(A1 |B) =
P(B)

4. Now assume we have a DVD which does not work, i.e. B̄

occurs. The probability that a DVD is not working given that
it is from store 1 is P(B̄|A) = 0.05. Similarly P(B̄|A2 ) = 0.25
for store 2. We can now calculate the conditional probability
that a DVD is from store 1 given that it is not working:

P(B̄|A1 )P(A1 )
P(A1 |B̄) =
P(B̄|A1 )P(A1 )P(B̄|A2 )P(A2 )

34/38
Bayes’ Theorem

I Cause/effect model: Given cause Ai how likely effect B is

observed?
Ai −−−−−→ B
P(B|Ai )

I Inference model: Give the observation B how likely is it due to

the state Ai ?
inference
Ai ←−−−−− B
P(Ai |B)

35/38
Independence
Two events A and B are independent if the probability of
occurrence of B has no effect on the probability of occurrence of A
P(A|B) = P(A) and P(A|B̄) = P(A).
from (1) we know that
P(A ∩ B)
P(A|B) =
P(B)
and
P(A ∩ B̄)
P(A|B̄) =
P(B̄)
This yields:
P(A ∩ B)P(B̄) = P(A ∩ B̄)P(B)
P(A ∩ B)(1 − P(B)) = P(A ∩ B̄)P(B)
P(A ∩ B) = P(A ∩ B̄)P(B) + P(A ∩ B)P(B)
P(A ∩ B) = (P(A ∩ B̄) + P(A ∩ B))P(B)
P(A ∩ B) = P(A)P(B) 36/38
Independence
Definition 1
Two random events A and B are called (stochastically)
independent if
P(A ∩ B) = P(A)P(B)
i.e. if the probability of simultaneous occurrence of both events A
and B is the product of the individual probabilities of occurrence of
A and B.

Definition 2
The n events A1 , A2 , . . . , An are stochastically mutually
independent, if for any subset of m events Ai1 , Ai2 , ..., Aim (m ≤ n)

P(Ai1 ∩ Ai2 · · · ∩ Aim ) = P(Ai1 )P(Ai2 ) . . . P(Aim ) (8)

holds.
If condition (8) is fulfilled only for two arbitrary events, i.e. m = 2,
then the events are called pairwise independent. 37/38
Independence
Example 2
Consider an urn with four balls. The following combinations of
zeroes and ones are printed on the balls: 110, 101, 011, 000. One
ball is drawn from the urn. Define the following events:

A1 : The first digit on the ball is 1.

A2 : The second digit on the ball is 1.
A3 : The third digit on the ball is 1.

I P(A1 ) =?, P(A2 ) =?, P(A3 ) =?

I P(A1 ∩ A2 ∩ A3 ) =?
I P(A2 )P(A2 )P(A3 ) =?
I P(A1 ∩ A2 ) =? , P(A1 )P(A2 ) =?
I P(A1 ∩ A3 ) =? , P(A1 )P(A3 ) =?
I P(A2 ∩ A3 ) =? , P(A2 )P(A3 ) =?
38/38