Chapter 2

Probability Theory
Ref: Alberto Leon-Garcia, Probability, Statistics, and
Random Processes for Electrical Engineering, Third
Edition, Pearson Education, Inc., 2009.

1-1
yhpang
 §2.1, 2.8, 2.9 Sample Space and Event Class
 A random experiment E  one and only one
outcome occurs
 The sample space of E: S = {all possible outcomes}
 An event class of S: ℱ ={subsets of S}, satisfying
(i) ∅ ∈ ℱ
(ii) if A1, A2, … ∈ ℱ, then ⋃ ∈ℱ
(iii) if A ∈ ℱ, then Ac ∈ ℱ
 ℱ is called a σ-field

 It implies that if A1, A2, … ∈ ℱ then ⋂ ∈ℱ

 An event: an element of ℱ
1-2
*{all possible outcomes} means the set of all possible outcomes yhpang
 §2.2 Axioms of Probability Law
 A probability law P: ℱ ⟶ [0, 1]
∀ ∈ ℱ, P[A] = the probability of A, satisfying
I. ≥0
II. =1
III. Any events A and B III′. Any events A1, A2, ... with
with A ∩ B = ∅ Ai ∩ Aj = ∅ for all i ≠ j

∪ = + =

(I, II, III for finite set S) (I, II, III′ for infinite set S)
(S, ℱ, P) is called probability space
1-3
yhpang
 Properties of Probability Law
II, III
 S = A ∪ Ac, A ∩ Ac = ∅  =1−
 = =1− =0
 Axion I: ≥0 =1− ≤1
 If Ai ∩ Aj = ∅ for all i, j = 1, ..,3, i ≠ j, then

1⋂ 2⋃ 3 = 1⋂ 2 ⋃ 1⋂ 3 = ∅
Axion III 1⋃ 2 ⋃ 3 = 1 + 2⋃ 3
= 1 + 2 + 3
Generalization: If Ai ∩ Aj = ∅ for all i ≠ j, then
⋃ =∑ , for $ ≥ 2
1-4
(Exercise: Prove it using mathematical induction.) yhpang
 Properties of Probability Law (Cont.)
Axiom III +
 S
Partition of A: = \ + ∩ A B

Partition of B: = \A + ∩
Partition of A ∪ B :
∪ = \ + \A + ∩
∪ = + − ∩
∪ ∪+ = + ∪+ − ∩ ∪+
= + + + − ∩ − ∩+ − ∩+
+ ∩ ∩+ 1-5
yhpang
 Properties of Probability Law (Cont.)
In general, for $ ≥ 1

= − ,⋂
-, ,…
-/

+ 0 ⋂ ,⋂ − ⋯ − −1 3
1,-, ,…
1/-/
Inclusion-Exclusion Principle

1-6
(Exercise: Prove it using mathematical induction.) yhpang
Example
 E : Toss a coin and notice if head or tail shows up.
 Sample space S = {H, T}
 An event class ℱ = 2S , the power set of S
ℱ = ∅, H , T , H, T ℱ
 An event A1 = {H} ∅ 0
H ⅓
T ⅔
H, T 1
 ℱ4 = ∅, H, T is also a valid event class
 ℱ7 = ∅, H , H, T is not a valid event class
1-7
yhpang
§2.2, 2.3 Computing Probability by
Counting – (I) Finite Sample Space
 For a finite sample space S = {s1, s2, ..., sn} with
event class ℱ = 2S
8 + 84 + ⋯ + 8 =1
 8 : elementary event, k = 1, 2, ..., n

 Special case: S = {s1, s2, ..., sn} with equal likely

outcomes (S is called a uniform sample space)
8 = 84 = ⋯ = 8 =
For any event A ∈ ℱ
#( )  # of elementary events of A
=
#( )  # of elementary events of S
1-8
yhpang
 Example – Permutation & Combination
 Choose k objects at random from n distinct objects
Permutation Combination
(w/i ordering) (w/o ordering)
w/o replacement (k ≤ n) (1) (3)
w/i replacement (2) (4)
$!
1 = $ · $ − 1 · ⋯· $ − +1 = ≡
$− !
2 =$
$! $
3 = = ≡ + HI
! ! $− !
$−1+
4 = ≡K 1-9
yhpang
Example – Permutation & Combination
without Replacement
 E.g.: Choose 3 objects at random from 4 distinct
objects w/o replacement
1, 2, 3 1, 3, 2 2, 1, 3 2, 3, 1 3, 1, 2 3, 2, 1
1, 2, 4 1, 4, 2 2, 1, 4 2, 4, 1 4, 1, 2 4, 2, 1
1, 3, 4 1, 4, 3 3, 1, 4 3, 4, 1 4, 1, 3 4, 3, 1
2, 3, 4 2, 4, 3 3, 2, 4 3, 4, 2 4, 2, 3 4, 3, 2

w/i ordering: 4 · 3 · 2 = 24
4L
w/o ordering: =4
7!

1-10
yhpang
Example – Combination with Replacement
 8 people are voting for one of 3 candidates.
Suppose that all votes are valid.
Choose k = 8 objects at random from n = 3 distinct objects
without ordering and with replacement
x1 + x2 + x3 = 8
x1 x2 x3
2 3 3
3 0 5

7M NO !
=45
O! 7M !

1-11
yhpang
 Example – Matching Problem (The Secret
Santa Problem)
 N female-male couples are at a party. Females and
males are randomly paired for a dance. A match
occurs if a couple is paired together. Find the
probability that no couple is matched.
e.g.: 1, 2, 3, 4, 5
Female: 1, 2, ..., n
$ couples Male: m1, m2, ..., mn
2, 3, 5, 1, 4
Permute 2, 3, 5, 4, 1

 Sample space
S = {(m1, m2, ..., mn): 1≤ mk ≤ n, mi ≠ mj, mk ∈ ℤN }
permutation of 1, 2, …, n
1-12
yhpang
Example – Matching Problem (The Secret
Santa Problem) (Cont.)
 Ak = k-th couple is matched
 A = no couple is matched = 1 ∪ ⋯∪ $
M ! M4 !
=
!
, ,⋂ =
!
,< ,…

1 ∪ ⋯∪ $

$ M ! $ M4 ! $ M7 ! $ V!
= − + …− −1
1 ! 2 ! 3 ! $ !

= − + − ⋯ − −1
! 4! 7! !

A =1− + − − ⋯ + −1 X M ≈ 0.368
! 4! 7! ! →

Note: X ] = 1 + ^+ ^4 + ^7 + ⋯ + ^ +⋯ 1-13
! 4! 7! ! yhpang
 Example – Birthday Probability
 A = {No two people in a group of n persons have common
birthday} (n > 1)
Stirling’s formula: $! ~ 2l$ ,$ ≫ 1
m
h
_àbc 7de! M 7deM N
XM
i
= = ~ 1−
7de` 7deM !×7de` 7de
or
7de· 7deM ·⋯· 7deM N M
= = 1− ⋯ 1−
7de` 7de 7de

if ^ ≪ 1 M7de
M
M 7de
M
M 4×7de
≈X …X = X if $ ≪ 365
X M] ≈ 1 − ^
< 0.5 → $ ≥ 23
n 23 56 69 79
If 0.4927 0.0117 0.0010 0.0001

1-14
yhpang
Math Toolbox – Cardinality of a Set
 Two sets A and B are said to have the same
cardinality if ∃ a bijection f : A → B
 For convenience, we denote the cardinality of A
as |A| even if it may not be finite.
 Given n ∈ ℕ, |A| ≡ n if |A| = |{1, 2, ..., n}|.
 Meaning: ∃ a bijection f : A → {1, 2, ..., n}

 Simply speaking, set A has n elements.

 A set A is finite if |A| = n for some n ≥ 0.

Otherwise, A is said to be infinite.
bijection: one-to-one and onto function
1-15
Ref: K. Ferland, Discrete Mathematics, Cengage Learning, 2009.
yhpang
Math Toolbox – Countable Infinity and
Uncountable Infinity
 Given A: any set
(i) A is said to be countably infinite if |A| = |ℤ+|
 |ℤ+| is denoted by ℵ0 (aleph zero), the smallest
infinite cardinality
(ii) A is said to be countable if A is finite or
countably infinite.
(iii) A is said to be uncountable if A is not
countable.

1-16
Ref: K. Ferland, Discrete Mathematics, Cengage Learning, 2009.
yhpang
 Computing Probability by Counting –
(II) Countably Infinite Sample Space
 For a countable sample space S = {s1, s2, ...} with
event class ℱ = 2S
8 + 84 + ⋯ = 1
 8 : elementary event, k = 1, 2, ...

 Ex: Toss a fair coin until the first heads shows up.
S = {# of tosses} = {1, 2, 3, ..}
 1 + 2 + ⋯+ $ +⋯

= + + ⋯+ +⋯=1
4 L 4`

1-17
yhpang
 Computing Probability by Counting –
(III) Uncountable Sample Space
q( )
=
q( )
q : measure of A, such as length, area, volume, ...
Example: Pick a number at random from [0, 1].
 Sample space: S ={x : 0 ≤ x ≤ 1} = [0, 1]
Assume uniform sample space:
rstuvw xy [z, {]
a≤x≤b = = } − ~, 0 ≤ ~ ≤ } ≤ 1
rstuvw xy |
^V = 0, ∀^V ∈
S A = 0.4
x x
0 1 0.5 0.9 1-18
yhpang
Borel Field over ℝ (Notation: ℬ )
 A σ-field generated by countable union, countable
intersection, and complements of the events of the
form (–∞, b], where –∞ < b < ∞)
 It can be shown that ℬ also contains events of the
following form:
(a, b), [a, b], (a, b], [a, b), [a, ∞), (a, ∞), (–∞,b), {b}
 Let A1, A2, ... be a sequence of events from ℬ, s.t.
A1 ⊂ A2 ⊂ ... ⊂ An ... or A1 ⊃ A2 ⊃ ... ⊃ An ...

lim = lim
→ →
1-19
yhpang
 Example – Bertrand Paradox
 Consider an equilateral triangle inscribed in a
circle. Choose a chord of the circle at random.
 P[the chord is longer than a side of the triangle] = ?
[M1] Select a point A on the D
circumference of the circle, and choose
C B
another point D on the circumference
randomly.
arc length of BC 1
= =
circumference of the circle 3 A

1-20
yhpang
Question – 請問 8 後面那張牌點數多少?

1-21
(圖片取自電影【賭俠】) yhpang
§2.4 Conditional Probability
 | = conditional probability of event A given
that event B has occurred (P[B] > 0)

∩
| ≡ A B

 i.e. take event B as a new sample space

 A useful formula
∩ = |

1-22
yhpang
 Example
 A urn contains m blue balls and n red balls. Select a
ball from the urn at random without returning. P[get
red ball at the k-th selection] = ? (k = 1, ..., m+n)
Sol: Let Ak = get red ball at the k-th selection
4
=
†N

4 = 4⋂ + 4⋂ 4
= 4‡ + 4‡ 4
M †
= + =
†N †N M †N †N M †N
... 4
= , = 1, 2, … , q + $
†N
1-23
[Q] Any intuitive methodology?
yhpang
 B1
B4
Bayes’ Rule A

,…, is a partition of S B2 B3
 A⊂S ⋂ ,…, is a partition of A
∑ ⋂ ∑ ‡
 Suppose that event A occurs
_ ˆ∩‰Š _ ‰Š _ ˆ‡‰Š Bayes’
1| _ˆ ∑`
ŠŒh _ ‰‹ _ ˆ‡‰‹ Rule
 Application:
Given , 1, … , $  A priori probabilities
If A occurs,
find 1|  A posteriori probability1-24
yhpang
 Example
non-carrier carrier
positive false positive (FP) true positive (TP)
negative true negative (TN) false negative (FN)
 Null hypothesis H0: non-carrier
••
Sensitivity (靈敏度) ≡ postive|KV •~Ž8X = (power of test)
••N‘’
•’
Specificity (特異性) ≡ negtive|KV “I”X = (confidence level)
‘•N•’
‘•
Type I error: • ≡ positive|KV “I”X =
‘•N•’
‘’
Type II error: – ≡ negative|KV •~Ž8X =
••N‘’
1-25
False positive/negative: 偽陽性/陰性 yhpang
 Example (Cont.)
 Total # = 1000, # of carrier = 10, α = 5%, β = 20%
KV “I”X ∩ positive
KV “I”X|positive =
positive
KV “I”X ∩ positive = KV “I”X positive|KV “I”X
——V e
= × ≜
VVV VV

KV •~Ž8X ∩ positive = KV •~Ž8X positive|KV •~Ž8X

V OV
= × ≜™
VVV VV
|
KV “I”X|positive = = 0.861
|Nš

Note: KV •~Ž8X = 0.01 v. s. KV •~Ž8X|positive = 0.139 1-26

yhpang
§2.5 Stochastically Independent
 Two events A and B are stochastically independent
(si.) if
∩ = ⋅
 Meaning: = | if ≠0
= | if ≠0
 Events A1, A2, ..., An are mutually stochastically
independent (msi.) if

1h ∩ ⋯∩ 1‹ = 1h ⋅ …⋅ 1‹

for any = 2, 3, … , $, 1 ≤ i1 < i2 < ... < ik ≤ n

1-27
yhpang
Stochastically Independent (Cont.)
 Example – For three events A1, A2, and A3
∩ 4 = ⋅ 4
∩ 7 = ⋅ 7
mutually
stochastically
4∩ 7 = 4 ⋅ 7 independent
∩ 4 ∩ 7 = ⋅ 4 ⋅ 7

∩ 4 = ⋅ 4 pairwisely
∩ 7 = ⋅ 7 stochastically

4∩ = ⋅
independent
7 4 7
∩ 4 ∩ 7 ≠ ⋅ 4 ⋅ 7 1-28
yhpang
Exercises
 Read articles about Monty Hall (Car-Goat)
Problem.

1-29
yhpang