Probability and Random

Variables
Dr. Sadiq Ali
 Motivation

“Probability theory enables AI systems to draw

conclusions and make predictions from data.”

Read the article I shared, and we will discuss it in

the coming class.
 Lecture Outline
• Theorem on Total Probability
 Theorem on Total Probability

• The law of total probability has numerous applications in

various fields, including medical diagnosis, machine
learning, and decision analysis
Mutually Exclusive and Exhaustive
If A ∩ B = Φ (events A and B are disjoint) and A ∪
B = S , then events A and B are called mutually
exclusive and exhaustive events
Mutually Exclusive and Exhaustive
• Partitioning of the entire sample space.
• B1, B2 and B3 are mutually exclusive & mutually
exhaustive sets (events).
Set A in a Partitioned Sample Space

A = A ∩ B1 ∪ A ∩ B2 ∪ A ∩ B3
Set A in a Partitioned Sample Space

A = A ∩ B1 ∪ A ∩ B2 ∪ A ∩ B3
• Note that the three sets A∩B1, A∩B2, and A∩B3 are
mutually exclusive (i.e. disjoint). Then, according to
Axiom 4,
P A = P A ∩ B1 + P A ∩ B2 + P A ∩ B3
 Theorem on Total Probability
• P A = P A ∩ B1 + P A ∩ B2 + P A ∩ B3

• After substituting the multiplication rule in the above,

we get,

• P A = P B1 P A|B1 + P B2 P A|B2 + P B3 P A|B3

• Which is the theorem of total probability

 Theorem on Total Probability
• If our sample space is partitioned into mutually exclusive
and exhaustive sets B1, B2, … , Bn, the probability of event A
is given by:
• P A = P A|B1 P B1 + P A|B2 P B2 + ⋯ + P A|Bn P Bn

B3 … Bn-1
B1

A
B2
Bn
…
 Theorem on Total Probability
• If our sample space is partitioned into mutually exclusive and
exhaustive sets A1, A2, … , An, the probability of event B is
given by:

• P B = P B|A1 P A1 + P B|A2 P A2 + ⋯ + P B|An P An

A3 … An-1
A1

B
A2
An
…
 Theorem on Total Probability
• Conditional probabilities often confuse students, so it's
best to focus on the previously explained intersection
concept.

• P B = P B ∩ A1 + P B ∩ A2 + ⋯ + P B ∩ An

• This equation is the same as the previous one, derived

by applying the multiplication rule.

• It is easier to remember and understand using a Venn

diagram.
 Example
• The probability that Pakistan wins a match against Australia if it
wins the toss is 0.4. The probability that Pakistan wins a match
against Australia if it looses the toss is 0.23.

• Find the probability that Pakistan will win the match.

 Example
• A – Pakistan winning the toss
• B – Pakistan winning the match
• We have to find P[B]
P B = P A ∩ B + P AC ∩ B

• P[A∩B] = Probability that Pakistan wins the toss and wins the
match.
• P[AC∩B] = Probability that Pakistan looses the toss and wins
the match.
P[B] = (0.5×0.4) + (0.5×0.23) = 0.315
 Example – Alternate Method
• A – Pakistan winning the toss
• B – Pakistan winning the match
 Example – Alternate Method
• A – Pakistan winning the toss
• B – Pakistan winning the match
 Example – Alternate Method

𝑃[𝐵] = (0.5 × 0.4) + (0.5 × 0.23) = 0.315

 Example
• The probability that Pakistan wins a match against
Australia if it wins the toss is 0.4. The probability that
Pakistan wins a match against Australia if it looses the
toss is 0.23.
• Find the probability that Pakistan does not win the
match.
 Example
• A – Pakistan winning the toss
• B – Pakistan winning the match
• We have to find P[BC]
P B C = P A ∩ B C + P AC ∩ B C
• P[A∩BC] = Probability that Pakistan wins the toss and
does not win the match.
• P[AC∩ BC] = Probability that Pakistan looses the toss
and does not win the match.
P[BC] = (0.5×0.6) + (0.5×0.77) = 0.685
Example – Alternate Method

P[BC ] = (0.5 × 0.6) + (0.5 × 0.77) = 0.685

 Theorem on Total Probability:
Independent Events
• P B = P B ∩ A1 + P B ∩ A2 + ⋯ + P B ∩ An

• After substituting the multiplication rule in the above, we get,

• P B = P B|A1 P A1 + P B|A2 P A2 + ⋯ + P B|An P An

• But if Ai is independent of B then

• P B = P B P A1 + P B P A2 + ⋯ + P B P An
 Total Probability and Bayes’ Rule
• Using the law of total probability and applying it to
the definition of the conditional probability, yields

P Ai ∩ B P Ai ∩ B
P A i |B = = n
P B σi=1 P Ai ∩ B

P Ai P(B|Ai )
= n
σi=1 P(B|Ai )P Ai

22
Example 1: Binary (Symmetric) Channel
• Many communication systems can be modeled in the
following way.

• First, the user inputs a 0 or a 1 into the system, and a

corresponding signal is transmitted.

• Second, the receiver makes a decision about what

was the input to the system, based on the signal it
received.
Example 1: Binary (Symmetric) Channel
• Suppose that the user sends 0s with probability 1 − p
and 1s with probability p.

• Suppose that the receiver makes random decision

errors with probability 𝜀.

• For i = 0,1, let Ai be the event “input was i,” and let
Bi be the event “receiver decision was i.”

• Find the probabilities P Ai ∩ Bj for i = 0,1 and j =

0,1
Example 1: Binary (Symmetric) Channel

Input Output

1
0

1
0

0
Example 1: Binary (Symmetric) Channel

Input Output

1
1−𝜀

1 𝜀
𝑝 0

1−𝑝
1
0 𝜀

1−𝜀
0
Example 1: Binary (Symmetric) Channel

Input Output
P A1 ∩ B1 = p 1 − ε
1
1−𝜀

1 𝜀
𝑝 0
P A1 ∩ B0 = pε
1−𝑝 P A0 ∩ B1 = 1 − p ε
1
0 𝜀

1−𝜀
0
P A 0 ∩ B0 = 1 − p 1 − ε
 Example 2 : Binary (Symmetric) Channel
• Given the binary symmetric channel depicted in
figure and given that P A0 = 0.4, P A1 = 0.6.
find P A𝑖 |Bj ; i, j = 0,1.
P B0 |A0 = 2/3
0 0
P B1 |A0 = 1/3

Input Output

P B0 |A1 = 1/4

1 1
P B1 |A1 = 3/4

28
 Example : Binary (Symmetric) Channel
• Using Bayes Rule:
P A i ∩ Bj P A i ∩ Bj
P Ai ቚBj = =
P Bj σ1i=0 P Ai ∩ Bj

P Ai P(Bj หAi )
=
σ1i=0 P(Bj หAi )P Ai

P Aj P(Bi |Aj )
=
P Bj หA0 P A0 + P(Bj หA1 )P A1

29
Example : Binary (Symmetric) Channel
P A0 P(B0 |A0 )
P A0 หB0 =
P B0 |A0 P A0 + P(B0 |A1 )P A1

P A1 P(B0 |A1 )
P A1 หB0 =
P B0 |A0 P A0 + P(B0 |A1 )P A1

P A0 P(B1 |A0 )
P A0 หB1 =
P B1 |A0 P A0 + P(B1 |A1 )P A1

P A1 P(B1 |A1 )
P A1 หB1 =
P B1 |A0 P A0 + P(B1 |A1 )P A1

30
Example : Binary (Symmetric) Channel
0.4 (2/3)
P A0 หB0 = = 0.64
2/3 0.4 + (1/4) 0.6

0.6 (1/4)
P A1 หB0 = = 0.36
2/3 0.4 + (1/4) 0.6

0.4 (1/3)
P A0 หB1 = = 0.23
1/3 0.4 + (3/4) 0.6

0.6 (3/4)
P A1 หB1 = = 0.77
1/3 0.4 + (3/4) 0.6

31
 Home Practice: Example 1
• Ahmed tosses a biased coin three times with probability
of heads 1/3.
• Find the probability that he gets one head in the
three tosses.
 Solution: Using Total Probability
• A – Heads in the first coin toss

• B – Heads in the second coin toss

• C – Heads in the third coin toss

• We have to find,
P A ∩ B C ∩ C c + P AC ∩ B ∩ C C + P AC ∩ B C ∩ C

=(1/3×2/3×2/3) + (2/3×1/3×2/3) + (2/3×2/3×1/3)

=4/27 + 4/27 + 4/27 = 12/27

Solution: Using Tree Diagram

4/27 + 4/27 + 4/27 = 12/27

 Home Practice: Example 2
• Both Ibrahim and Ismail flip a fair coin twice.

a) Write the sample space of this experiment.

b) Find the probability of each outcome.

c) Draw the probability tree diagram for this situation.

 Solution
a) S = {(HH, HH), (HH, HT), (HH, TH), (HH, TT), (HT,
HH), (HT, HT), (HT, TH), (HT, TT), (TH, HH), (TH,
HT), (TH, TH), (TH, TT), (TT, HH), (TT, HT), (TT,
TH), (TT, TT)}
 Solution
b) Since there are 16 total possible outcomes and each
outcome is equally likely, the probability of each
outcome is 1/16.
 Solution
c) Tree Diagram
 Solution
c) Tree Diagram
Scientific knowledge and methods are essentially an advanced
form of everyday reasoning, emphasizing the importance of
systematic observation and critical thinking

See you next week