EE5110w: Probability Foundations for Electrical Engineers

January - April 2022, Axioms of Probability Theory

1 References
• See Chapter 1: Foundations, Bruce Hajek for a review of the axioms
of the probability theory and for practice problems.

• See Chapter 2: Probability, from Gray and Davisson, for an elementary

measure theoretic introduction to probability theory.

• See Chapter 1: Sample Space and Probability, Bertsekas and Tsitsik-

lis, for a discussion on Bertrand’s paradox, Cantor’s diagonalization
argument and inclusion-exclusion principle.

2 Solved Problems
1. Let F be a sigma-algebra of subsets of Ω. If A1 , A2 , · · · are in F, then
∞
T
Ai ∈ F.
i=1
Let A1 , A2 , · · · be events in F. Then, Ac1 , Ac2 , · · · are in F (by e2).
∞
If Ac1 , Ac2 , · · · are events in F, then Aci ∈ F (by e3).
S
i=1
∞
∞ c
Aci ∈ F, then Aci
S S
If ∈ F (by e2).
i=1 i=1
∞ c ∞ ∞
Aci
S T T
But = Ai (by DeMorgan’s law). Hence, Ai ∈ F. 
i=1 i=1 i=1

2. If A1 , A2 , · · · , An are disjoint events in F, then

n n
!
[ X
P Ai = P(Ai )
i=1 i=1

Let A1 , A2 , · · · , An be disjoint events in F, i.e., Ai ∩ Aj = φ, for

1 ≤ i, j ≤ n and i 6= j.
Define An+1 = φ = An+2 = An+3 = · · · . Then, the collection of
events, A1 , · · · , An , An+1 , An+2 , · · · are disjoint, i.e., Ai ∩ Aj = φ for
i 6= j. Further,
∞
[ [n
Ai = Ai
i=1 i=1

1
 We can now use countable additivity axiom (p3) on the disjoint count-
able collection {Ai }.
n ∞ ∞ n ∞
! !
[ [ X X X
P Ai = P Ai = P(Ai ) = P(Ai ) + P(Ai )
i=1 i=1 i=1 i=1 i=n+1

For i = n + 1, · · · , we note that P(Ai ) = P(φ) = 0. Further, 0 + 0 +

0 + · · · = 0. Hence,
n n
!
[ X
P Ai = P(Ai )
i=1 i=1

3. If A1 , A2 ∈ F and A1 ⊂ A2 , then P(A1 ) ≤ P(A2 ).

Let A1 , A2 ∈ F such that A1 ⊂ A2 . Now,

A2 = A2 ∩ Ω = A2 ∩ (A1 ∪ Ac1 ) = (A2 ∩ A1 ) ∪ (A2 ∩ Ac1 ) = A1 ∪ (A2 ∩ Ac1 )

We note that A1 and A2 Ac1 are disjoint as well, i.e.,

A1 ∩ (A2 ∩ Ac1 ) = A1 ∩ (Ac1 ∩ A2 ) = (A1 ∩ Ac1 ) ∩ A2 = φ ∩ A2 = φ

Thus, A2 = A1 ∪ (A2 ∩ Ac1 ) and A1 , (A2 Ac1 ) are disjoint. Hence, from
finite additivity property of the probability measure, we have,

P(A2 ) = P(A1 ) + P(A2 Ac1 )

We know that P(A2 Ac1 ) ≥ 0 (p1). Hence,

P(A2 ) = P(A1 ) + P(A2 Ac1 ) ≥ P(A1 )

or, P(A2 ) ≥ P(A1 ) when A2 ⊃ A1 . 

4. If A1 and A2 are events in F, then

P(A1 ∪ A2 ) = P(A1 ) + P(A2 ) − P(A1 ∩ A2 )

Let A1 and A2 be events in F.

Note that A1 Ac2 , Ac1 A2 and A1 A2 are disjoint events in F (F is a sigma-
algebra). Further, A1 = A1 Ac2 ∪ A1 A2 , A2 = A1 A2 ∪ Ac1 A2 and A1 ∪
A2 = A1 Ac2 ∪ A1 A2 ∪ Ac1 A2 .
Using the finite additivity property of the probability measure, we
have
P(A1 ) = P(A1 A2 ) + P(A1 Ac2 )

2
 P(A2 ) = P(A1 A2 ) + P(Ac1 A2 )

P(A1 ∪ A2 ) = P(A1 Ac2 ∪ A1 A2 ∪ Ac1 A2 )

= P(A1 Ac2 ) + P(A1 A2 ) + P(Ac1 A2 )
= P(A1 Ac2 ) + P(A1 A2 ) + P(Ac1 A2 ) + P(A1 A2 ) − P(A1 A2 )
= P(A1 ) + P(A2 ) − P(A1 A2 )

5. If A1 , A2 , · · · are events in F, then

∞ ∞
!
[ X
P Ai ≤ P(Ai )
i=1 i=1

Let A1 , A2 , · · · be events in F. Define

B1 = A1 , B2 = A2 − A1 , B3 = A3 − (A1 ∪ A2 )

and in general,
n−1
!
[
Bn = An − Ai
i=1

We note that that B1 , B2 , · · · are disjoint sets in F and

n
[ n
[
Ai = Bi
i=1 i=1

and
∞
[ ∞
[
Ai = Bi
i=1 i=1

From the countable additivity axiom of the probability measure (p3),

we have,
∞ ∞
! !
[ [
P Ai = P Bi
i=1 i=1
∞
X
= P(Bi )
i=1
n
X
= lim P(Bi )
n→∞
i=1

3
 Now, Bi ⊂ Ai , and from the monotonicity property of the probability
measure, P(Bi ) ≤ P(Ai ). Upper bounding the partial sum in the
previous expression, we get,
∞ n
!
[ X
P Ai = lim P(Bi )
n→∞
i=1 i=1
n
X
≤ lim P(Ai )
n→∞
i=1
∞
X
= P(Ai )
i=1

(Let {sn } and {rn } be monotone increasing sequences in R such that

sn ≤ bn . Then, limn→∞ sn ≤ limn→∞ rn .) 

6. Consider a probability space (Ω, F, P). Show that the countable addi-
tivity axiom implies the following statements.

(a) Let B1 , B2 , · · · be events in F. Then,

∞ n
! !
[ [
P Bi = lim P Bi
n→∞
i=1 i=1

(b) Let C1 ⊃ C2 ⊃ · · · be events in F. Then,

∞
!
\
P Ci = lim P (Cn )
n→∞
i=1

(a) Let B1 , B2 , · · · be events in F. Define A1 = B1 , A2 = B2 − B1 , · · · ,

An = Bn − (B1 ∪ · · · ∪ Bn−1 ), · · · . We note that A1 , A2 , · · · are a
disjoint collection of events in F. Further, we note that
n
[ n
[
Bi = Ai
i=1 i=1

and
∞
[ ∞
[
Bi = Ai
i=1 i=1

From the countable/finite additivity axiom of the probability measure,

4
we have,
∞ ∞ ∞
! !
[ [ X
P Bi =P Ai = P(Ai )
i=1 i=1 i=1
n
X
= lim P(Ai )
n→∞
i=1
n
!
[
= lim P Ai
n→∞
i=1
n
!
[
= lim P Bi
n→∞
i=1

which is our desired result.

(b) Let C1 ⊃ C2 ⊃ · · · be events in F. Define Bi = Cic for all
i = 1, 2, · · · . Then, B1 ⊂ B2 · · · are events in F and
n
[
Bi = Bn
i=1

From the continuity property of the probability measure (or, part (a)
of the problem), we have
∞ n
! !
[ [
P Bi = lim P Bi
n→∞
i=1 i=1
= lim P (Bn )
n→∞

The probabilities of the complements can be computed as follows.

∞
!
[
1−P Bi = 1 − lim P (Bn )
n→∞
i=1
= lim 1 − P (Bn )
n→∞
= lim P (Bnc )
n→∞
= lim P (Cn )
n→∞

Also,
∞
! ∞
!c ! ∞
! ∞
!
[ [ \ \
1−P Bi =P Bi =P Bic =P Ci
i=1 i=1 i=1 i=1

5
 Thus,
∞
!
\
P Ci = lim P (Cn )
n→∞
i=1

which is our desired result. 

7. Consider the following valid probability space

• Ω=R

• F = B(R) (Borel sigma-algebra on R)

• for [a, b] ∈ F, define P([a, b]) as

Z
P([a, b]) = 1 dx
[a,b]∩[0,1]

Compute P(Ω), P({0.5}) and P(Q).

Ω = R = (−∞, ∞). We note that Ω ∈ B(R). Hence, P(Ω) is well
defined.
Z Z 1
P(Ω) = P(R) = 1 dx = 1 dx = 1
(−∞,∞)∩[0,1] 0

1 1
 
Define An = 0.5 − 2n , 0.5 + 2n for n = 1, 2, · · · . Note that A1 ⊃
A2 ⊃ · · · are events in B(R), and,
∞ ∞  
\ \ 1 1
Ai = 0.5 − , 0.5 + = {0.5}
2n 2n
i=1 i=1

(A = B if A ⊂ B and B ⊂ A)
The probability of the event An can be computed as follows.
  Z
1 1
P(An ) = P 0.5 − , 0.5 + = 1 dx
2n 2n 1
[0.5− 2n 1
,0.5+ 2n ]∩[0,1]
Z 0.5+ 1
2n
= 1 dx
1
0.5− 2n
1
=
n
From the monotonicity property of the probability measure, we have,
∞
!
\ 1
P({0.5}) = P Ai = lim P(An ) = lim =0
n→∞ n→∞ n
i=1

6
 which is our desired result. (In fact, we can show that P({x}) = 0 for
all x ∈ R.)
In the previous exercise, we proved that the probability of any single-
ton set in R is zero. Now, Q is a countable set in B(R) which can be
written as a countable union of disjoint singleton sets in B(R) (i.e.,
Sn
Q = {qi } for an appropriate countable, disjoint collection {qi }).
i=1
Using the countable additivity axiom, we have,
∞ ∞
!
[ X
P (Q) = P {qi } = P({qi }) = 0
i=1 i=1

which is our desired result. The result implies that there are infinite
(countable) events in F with zero probability. We can also show that
P(P) = 1 − P(Q) = 1.

2.1 Examples of Important Probability Distributions

• Bernoulli (p : 0 ≤ p ≤ 1): ({0, 1}, 2Ω , P) where

P({0}) = 1 − p, and P({1}) = p

• Uniform (N : N ∈ N): ({1, 2, · · · , N }, 2Ω , P) where

1
P({i}) = for all i = 1, 2, · · · , N
N

• Geometric (p : 0 < p ≤ 1): (N, 2Ω , P) where

P({i}) = (1 − p)i−1 p for all i ∈ N

• Uniform (a, b : a < b ∈ R) : (R, B(R), P) where

Z
1
P([x, y]) = du
[x,y]∩[a,b] b − a

• Exponential (λ : λ > 0): (R, B(R), P) where

Z
P([x, y]) = λe−λu du
[x,y]∩[0,∞)

• Gaussian (µ, σ 2 : µ, σ ∈ R): (R, B(R), P) where

Z y
1 (u−µ)2
P([x, y]) = √ e− 2σ2 du
x 2πσ 2

7
3 Practice Problems
1. (Gray-Davisson) Given that the discrete sample space Ω has n ele-
ments, show that the power set of Ω consists of 2n elements.

2. Consider a sample space Ω (Ω may be finite, countable or uncount-

able). Show that the power set of Ω (set of all subsets of Ω) is an event
space (i.e., the power set is closed under complement and countable
union).

3. Consider a discrete sample space Ω (Ω may be finite or countable).

Show that the sigma-algebra generated by the singleton sets is the
power set of Ω (i.e., the smallest sigma-algebra containing all the sin-
gleton sets is the power set of Ω).

4. Consider the Borel sigma-algebra on R, B(R), i.e., a sigma-algebra

generated by intervals [a, b] where −∞ < a < b < ∞. Show that

(a) {a} ∈ B(R)

(b) [a, b) ∈ B(R) for −∞ < a < b ≤ ∞
(c) (a, b] ∈ B(R) for −∞ ≤ a < b < ∞
(d) (a, b) ∈ B(R) for −∞ ≤ a < b ≤ ∞
∞
S
(e) [ai , bi ] ∈ B(R) for −∞ < ai < bi < ∞ for all i = 1, 2, · · ·
i=1
∞
T
(f) [ai , bi ] ∈ B(R) for −∞ < ai < bi < ∞ for all i = 1, 2, · · ·
i=1
(g) Q, P, N, Z, R ∈ B(R)

5. (Stark & Woods) What is the smallest sigma-algebra containing the

events A and B?

6. (Sheldon Ross) Let E,F and G be three events. Find expressions for
the events so that, of E,F and G,

(a) only E occurs

(b) both E and G, but not F, occurs
(c) at least one of the event occurs
(d) at most two of the event occurs
(e) none of the event occurs

7. (Gray-Davisson) If G ⊂ F , prove that P(F − G) = P(F ) − P(G).

8
 8. (Gray-Davisson) Let {Fi } be a countable partition in F (i.e., {Fi } are
∞
S
disjoint and Fi = Ω). Show that for any A ∈ F,
i=1

∞
[
P(A) = P(A ∩ Fi )
i=1

9. (Gray-Davisson) Let F, G be events in F. Show that if P(F ) ≥ 1 − δ

and P(G) ≥ 1 − δ, then P(F ∩ G) ≥ 1 − 2δ. (In other words, if
two events have probability nearly one, then their intersection has
probability nearly one.)

10. (Bertsekas-Tsitsiklis) Out of the students in a class, 60% are geniuses,

70% love chocolate, and 40% fall into both categories. Determine the
probability that a randomly selected student is neither a genius nor a
chocolate lover.

11. (Hajek) What is P(ABC) if P(A) = P(B) = P(C) = 0.5, P(A ∪ B) =

0.55, P(A ∪ C) = 0.7, P(BC) = 0.3 and P(ABC) = 2P(ABC c )?

12. (Grimmett-Stirzaker) Let A and B be events with probabilities P(A) =

3 1 1 1
4 and P(B) = 3 . Show that 12 ≤ P(A ∩ B) ≤ 3 . Find corresponding
bounds for P(A ∪ B).

13. (Inclusion-Exclusion Principle) If A1 , A2 , · · · , An are events in F, then

n n
!
[ X X
P Ai = P(Ai )− P(Ai Aj )+· · ·+(−1)n−1 P(A1 · · · An )
i=1 i=1 1≤i<j≤n

14. Show that the standardized Gaussian probability density function,

f (x), integrates to one.
1 x2
f (x) = √ e− 2 , x ∈ R
2π

3.1 Associated Problems

1. (Gray-Davisson) Describe the sigma-field of subsets of R generated by
the points or singleton sets. Does this sigma-field contain intervals of
the form (a, b) for −∞ < a < b < ∞?

2. (Bertsekas-Tsitsiklis) Show that the unit interval [0, 1] is uncountable.

3. (Gray-Davisson) Consider the uniform probability measure on [0, 1].

Show that there exists an uncountable set in [0, 1] with probability
zero. (Hint: see Cantor’s set)

9
4. An algebra is a collection of subsets closed under complement and
finite unions. Show that the algebra made of singleton sets of N does
not contain the set Even = {2, 4, 6, · · · }. (The result implies that
countable unions are necessary to define events of interest.)

10