NPTEL- Probability and Distributions

MODULE 1
PROBABILITY
LECTURE 5
Topics
1.3.2 Bayes’ Theorem

1.3.2 Bayes’ Theorem

The following theorem provides a method for finding the probability of occurrence of an
event in a past trial based on information on occurrences in future trials.

1.3.2 Theorem 3.4 (Bayes’ Theorem)

Let 𝛺, ℱ, 𝑃 be a probability space and let 𝐸𝑖 : 𝑖 ∈ 𝛬 be a countable collection of

mutually exclusive and exhaustive events with 𝑃 𝐸𝑖 > 0, 𝑖 ∈ 𝛬 . Then, for any event
𝐸 ∈ ℱ with 𝑃 𝐸 > 0, we have

𝑃 𝐸|𝐸𝑗 𝑃 𝐸𝑗
𝑃 𝐸𝑗 |𝐸 = , 𝑗∈𝛬∙
𝑖∈𝛬 𝑃 𝐸|𝐸𝑖 𝑃 𝐸𝑖

Proof. We have, for 𝑗 ∈ 𝛬,

𝑃 𝐸𝑗 ∩ 𝐸
𝑃 𝐸𝑗 |𝐸 =
𝑃(𝐸)

𝑃 𝐸|𝐸𝑗 𝑃 𝐸𝑗
=
𝑃(𝐸)

𝑃 𝐸|𝐸𝑗 𝑃 𝐸𝑗
= using Theorem of Total Probability . ▄
𝑖∈𝛬 𝑃 𝐸|𝐸𝑖 𝑃 𝐸𝑖

Remark 3.2

(i) Suppose that the occurrence of any one of the mutually exclusive and
exhaustive events 𝐸𝑖 , 𝑖 ∈ 𝛬, causes the occurrence of an event 𝐸. Given that the
event 𝐸 has occurred, Bayes’ theorem provides the conditional probability that
the event 𝐸 is caused by occurrence of event 𝐸𝑗 , 𝑗 ∈ 𝛬.

Dept. of Mathematics and Statistics Indian Institute of Technology, Kanpur 1

NPTEL- Probability and Distributions

(ii) In Bayes’ theorem the probabilities 𝑃 𝐸𝑗 , 𝑗 ∈ 𝛬 , are referred to as prior

probabilities and the probabilities 𝑃 𝐸𝑗 |𝐸 , 𝑗 ∈ 𝛬, are referred to as posterior
probabilities. ▄

To see an application of Bayes’ theorem let us revisit Example 3.4.

Example 3.5

Urn 𝑈1 contains 4 white and 6 black balls and urn 𝑈2 contains 6 white and 4 black balls.
A fair die is cast and urn 𝑈1 is selected if the upper face of die shows five or six dots.
Otherwise urn 𝑈2 is selected. A ball is drawn at random from the selected urn.

(i) Given that the drawn ball is white, find the conditional probability that it came
from urn 𝑈1 ;
(ii) Given that the drawn ball is white, find the conditional probability that it came
from urn 𝑈2 .

Solution. Define the events:

𝑊 ∶ drawn ball is white;

𝐸1 ∶ urn 𝑈1 is selected
mutually exclusive & exhaustive events
𝐸2 ∶ urn 𝑈2 is selected

(i) We have
𝑃 𝑊|𝐸1 𝑃 𝐸1
𝑃 𝐸1 |𝑊 =
𝑃 𝑊|𝐸1 𝑃 𝐸1 + 𝑃 𝑊|𝐸2 𝑃 𝐸2
4 2
×6
10
= 4 2 6 4
×6 + ×6
10 10

1
= ∙
4
(ii) Since 𝐸1 and 𝐸2 are mutually exclusive and 𝑃 𝐸1 ∪ 𝐸2 |𝑊 = 𝑃 𝛺|𝑊 = 1,we
have

𝑃 𝐸2 |𝑊 = 1 − 𝑃 𝐸1 |𝑊

3
= ∙▄
4
In the above example

1 1
𝑃 𝐸1 |𝑊 = < = 𝑃 𝐸1 ,
4 3

Dept. of Mathematics and Statistics Indian Institute of Technology, Kanpur 2

NPTEL- Probability and Distributions

3 2
and 𝑃 𝐸2 |𝑊 = > = 𝑃 𝐸2 ,
4 3
i.e.,

(i) the probability of occurrence of event 𝐸1 decreases in the presence of the

information that the outcome will be an element of 𝑊;
(ii) the probability of occurrence of event 𝐸2 increases in the presence of information
that the outcome will be an element of 𝑊.

These phenomena are related to the concept of association defined in the sequel.

Note that

𝑃 𝐸1 |𝑊 < 𝑃 𝐸1 ⇔ 𝑃 𝐸1 ∩ 𝑊 < 𝑃 𝐸1 𝑃(𝑊),

and

𝑃 𝐸2 |𝑊 > 𝑃 𝐸2 ⇔ 𝑃 𝐸2 ∩ 𝑊 > 𝑃 𝐸2 𝑃(𝑊).

Definition 3.2
Let 𝛺, ℱ, 𝑃 be a probability space and let 𝐴 and 𝐵 be two events. Events 𝐴 and 𝐵 are
said to be

(i) negatively associated if 𝑃 𝐴 ∩ 𝐵 < 𝑃 𝐴 𝑃 𝐵 ;

(ii) positively associated if 𝑃 𝐴 ∩ 𝐵 > 𝑃 𝐴 𝑃 𝐵 ;
(iii) independent if 𝑃 𝐴 ∩ 𝐵 = 𝑃 𝐴 𝑃 𝐵 . ▄

Remark 3.3

(i) If 𝑃 𝐵 = 0 then 𝑃 𝐴 ∩ 𝐵 = 0 = 𝑃 𝐴 𝑃 𝐵 , ∀𝐴 ∈ ℱ, i.e., if 𝑃 𝐵 = 0 then

any event 𝐴 ∈ ℱ and 𝐵 are independent;
(ii) If 𝑃 𝐵 > 0 then 𝐴 and 𝐵 are independent if, and only if, 𝑃 𝐴|𝐵 = 𝑃(𝐴),
i.e., if 𝑃 𝐵 > 0, then events 𝐴 and 𝐵 are independent if, and only if, the
availability of the information that event 𝐵 has occurred does not alter the
probability of occurrence of event 𝐴. ▄

Now we define the concept of independence for arbitrary collection of events.

Dept. of Mathematics and Statistics Indian Institute of Technology, Kanpur 3

NPTEL- Probability and Distributions

Definition 3.3
Let 𝛺, ℱ, 𝑃 be a probability space. Let 𝛬 ⊆ ℝ be an index set and let 𝐸𝛼 : 𝛼 ∈ 𝛬 be a
collection of events in ℱ.

(i) Events 𝐸𝛼 : 𝛼 ∈ 𝛬 are said to be pairwise independent if any pair of events 𝐸𝛼

and 𝐸𝛽 , 𝛼 ≠ 𝛽 in the collection 𝐸𝑗 : 𝑗 ∈ 𝛬 are independent. i.e., if 𝑃 𝐸𝛼 ∩
𝐸𝛽 = 𝑃 𝐸𝛼 𝑃 𝐸𝛽 , whenever 𝛼, 𝛽 ∈ 𝛬 and 𝛼 ≠ 𝛽;

(ii) Let 𝛬 = 1, 2, … , n , for some 𝑛 ∈ ℕ, so that 𝐸𝛼 : 𝛼 ∈ 𝛬 = 𝐸1 , … , 𝐸𝑛 is a

finite collection of events in ℱ. Events 𝐸1 , … , 𝐸𝑛 are said to be independent if,
for any sub collection 𝐸𝛼 1 , … , 𝐸𝛼 𝑘 of 𝐸1 , … , 𝐸𝑛 𝑘 = 2,3, … , 𝑛
𝑘 𝑘

𝑃 𝐸𝛼 𝑗 = 𝑃 𝐸𝛼 𝑗 . (3.6)
𝑗 =1 𝑗 =1

(iii) Let 𝛬 ⊆ ℝ be an arbitrary index set. Events 𝐸𝛼 : 𝛼 ∈ 𝛬 are said to be

independent if any finite sub collection of events in 𝐸𝛼 : 𝛼 ∈ 𝛬 forms a
collection of independent events. ▄

Remark 3.4

(i) To verify that 𝑛 events 𝐸1 , … , 𝐸𝑛 ∈ ℱ are independent one must verify

𝑛
2𝑛 − 𝑛 − 1 = 𝑛𝑗=2 𝑗 conditions in (3.6). For example, to conclude that
three events 𝐸1 , 𝐸2 and 𝐸3 are independent, the following 4 = 23 − 3 − 1
conditions must be verified:
𝑃 𝐸1 ∩ 𝐸2 = 𝑃 𝐸1 𝑃 𝐸2 ;
𝑃 𝐸1 ∩ 𝐸3 = 𝑃 𝐸1 𝑃 𝐸3 ;
𝑃 𝐸2 ∩ 𝐸3 = 𝑃 𝐸2 𝑃 𝐸3 ;
𝑃 𝐸1 ∩ 𝐸2 ∩ 𝐸3 = 𝑃 𝐸1 𝑃 𝐸2 𝑃 𝐸3 .

(ii) If events 𝐸1 , … , 𝐸𝑛 are independent then, for any permutation 𝛼1 , … , 𝛼𝑛 of

1, … , 𝑛 , the events 𝐸𝛼 1 , … , 𝐸𝛼 𝑛 are also independent. Thus the notion of
independence is symmetric in the events involved.

(iv) Events in any subcollection of independent events are independent. In

particular independence of a collection of events implies their pairwise
independence. ▄

Dept. of Mathematics and Statistics Indian Institute of Technology, Kanpur 4

NPTEL- Probability and Distributions

The following example illustrates that, in general, pairwise independence of a collection

of events may not imply their independence.

Example 3.6
Let 𝛺 = 1, 2, 3, 4 and let ℱ = 𝒫 𝛺 , the power set of 𝛺. Consider the probability space
1
𝛺, ℱ, P , where 𝑃 𝑖 = 4 , 𝑖 = 1, 2, 3, 4 . Let 𝐴 = 1, 4 , 𝐵 = 2, 4 and 𝐶 = 3, 4 .
Then,
1
𝑃 𝐴 = 𝑃 𝐵 = 𝑃 𝐶 = 2,

1
𝑃 𝐴 ∩ 𝐵 = 𝑃 𝐴 ∩ 𝐶 = 𝑃 𝐵 ∩ 𝐶 = 𝑃 {4} = 4,

1
and 𝑃 𝐴∩𝐵∩𝐶 =𝑃 4 = ∙
4

Clearly,

𝑃 𝐴 ∩ 𝐵 = 𝑃 𝐴 𝑃 𝐵 ; 𝑃 𝐴 ∩ 𝐶 = 𝑃 𝐴 𝑃(𝐶), and 𝑃 𝐵 ∩ 𝐶 = 𝑃 𝐵 𝑃(𝐶),

i.e., 𝐴, 𝐵 and 𝐶 are pairwise independent.

However,
1
𝑃 𝐴 ∩ 𝐵 ∩ 𝐶 = 4 ≠ 𝑃 𝐴 𝑃 𝐵 𝑃(𝐶).

Thus 𝐴, 𝐵 and 𝐶 are not independent. ▄

Dept. of Mathematics and Statistics Indian Institute of Technology, Kanpur 5