Chapter 2
Basic Concepts of Probability
Theory
ENCS6161 - Probability and Stochastic
Processes
Concordia University
Specifying Random Experiments
Examples of random experiments: tossing a coin,
rolling a dice, the lifetime of a harddisk.
Sample space: the set of all possible outcomes of a
random experiment.
Sample point: an element of the sample space S
Examples: S = {H, T }
S = {1, 2, 3, 4, 5, 6}
S = {t|1 < t < 10}
Event: a subset of a sample space A ⊆ S
A = {H}
A = {2, 4, 6}
A=∅
ENCS6161 – p.1/16
The Axioms of Probability
A probability measure is a set function P (·) that
satisfies the following axioms.
1. P (A) ≥ 0
2. P (S) = 1
3. If A ∩ B = ∅, then P (A ∪ B) = P (A) + P (B)
4. If A1 , A2 , · · · are events s.t Ai ∩ Aj = ∅ for all i 6= j
∞ P∞
then P ∪k=1 Ak = k=1 P (Ak )
Corollary 1: P (Ac ) = 1 − P (A)
Corollary 2: P (A) ≤ 1 ∀A
Corollary 3: P (∅) = 0
ENCS6161 – p.2/16
The Axioms of Probability
Corollary 4: If A1 · · · An are mutually exclusive, i.e.
Ai ∩ Aj = ∅, ∀i 6= j then P (∪nk=1 Ak ) = nk=1 P (Ak )
P
Corollary 5: P (A ∪ B) = P (A) + P (B) − P (A ∩ B)
A B
A∩B
Union bound: P (A ∪ B) ≤ P (A) + P (B)
ENCS6161 – p.3/16
Counting Sample Points
Permutation: n distinct objects, how many ways can
we arrange them?
n · (n − 1) · (n − 2) · · · 2 · 1 = n!
Selection with order: select k objects from n ≥ k
objects
n!
n · (n − 1) · · · (n − k + 1) = = n Pk
(n − k)!
Combination (selection without order): select k
objects from n ≥ k objects
n n! n
Ck = =
(n − k)!k! k
ENCS6161 – p.4/16
Counting Sample Points
Example: choose a president and a treasurer from 50
students
1. no restrictions
2. A will serve only if he is president
3. B & C will serve together or not at all
4. D & E will not serve together
ENCS6161 – p.5/16
Conditional Probability
The conditional prob. of A given B has occurred is
defined by :
P (A ∩ B)
P (A|B) = , P (B) > 0
P (B)
Similarly,
P (A ∩ B)
P (B|A) = , P (A) > 0
P (A)
So
P (A ∩ B) = P (B)P (A|B) = P (A)P (B|A)
ENCS6161 – p.6/16
Bayes’ Rule
If B1 · · · Bn are mutually exclusive and ∪Bi = S, we
call these sets a partition of S.
Theorem of total probability
For any event A, if B1 , · · · , Bn is a partition of S,
P (A) = P (A ∩ B1 ) + P (A ∩ B2 ) + · · · + P (A ∩ Bn )
= P (A|B1 )P (B1 ) + P (A|B2 )P (B2 ) + · · · + P (A|Bn )P (Bn )
Bayes’ Rule
Let B1 , · · · , Bn be a partition of S , and P (A) ≥ 0, then
P (A ∩ Bj ) P (A|Bj )P (Bj )
P (Bj |A) = = n
P
P (A) k=1 P (A|Bk )P (Bk )
ENCS6161 – p.7/16
Bayes’ Rule
Example: binary communication system
1−ε
0 0
ε
P (X = 0) = P
X Y
P (X = 1) = 1 − P
ε
1 1
1−ε
1. Find P (Y = 0)
2. Find P (X = 0|Y = 0) and P (X = 1|Y = 0)
ENCS6161 – p.8/16
Bayes’ Rule
Solution: let A be event Y = 0, B0 be X = 0, B1 be
X=1
P (Y = 0) = P (A) = P (A|B0 )P (B0 ) + P (A|B1 )P (B1 )
= (1 − ε)P + ε(1 − P )
P (A ∩ B0 )
P (X = 0|Y = 0) = P (B0 |A) =
P (A)
P (A|B1 )P (B1 ) (1 − ε)P
= =
P (A) (1 − ε)P + ε(1 − P )
Similarly:
ε(1 − P )
P (X = 1|Y = 0) =
(1 − ε)P + ε(1 − P )
ENCS6161 – p.9/16
Bayes’ Rule
If P (X = 0|Y = 0) > P (X = 1|Y = 0), we decide
X = 0, i.e., if
(1 − ε)P > ε(1 − P ) ⇒ P > ε,
we decide X = 0.
Similarly, if P < ε, we decide X = 1.
A special case: if P = 0.5 and ε < 0.5 then
Y = 0 ⇒ we decide X = 0
Y = 1 ⇒ we decide X = 1
ENCS6161 – p.10/16
Bayes’ Rule
Example: 2% of people have on type of blood
disease. If a person has the disease and take a
blood test, with 96% probability, the result is positive
and with 4% probability, the result is negtive. If a
person without the disease takes a blood test, then
94% negative and 6% positive. Find
P (have disease|positive)
ENCS6161 – p.11/16
Bayes’ Rule
Solution: Let
D: a person has the disease D0 : no disease
B : blood test positive B 0 : blood test negative
Then P (D) = 2% and
P (B|D) = 0.96 P (B 0 |D) = 0.04
P (B|D0 ) = 0.06 P (B 0 |D0 ) = 0.94
Apply Bayes Rule:
P (D|B) = 0.246 = 24.6%
ENCS6161 – p.12/16
Bayes’ Rule
We can also use tree diagram.
0.96 B 0.02 ∗ 0.96 = 0.0192 (D ∩ B)
D
0.02 0.078 B
0.04 B’ 0.02 ∗ 0.04 = 0.0008 (D ∩ B0)
0.06 B 0.98 ∗ 0.06 = 0.0588 (D 0 ∩ B)
0.98 0.922 B’
D’ 0 0
0.94 B’ 0.98 ∗ 0.94 = 0.9212 (D ∩ B )
P (D ∩ B) 0.0192
P (D|B) = = = 0.246
P (B) 0.078
ENCS6161 – p.13/16
Bayes’ Rule
How to find P (D|BB)?
P (D ∩ BB)
P (D|BB) =
P (BB)
0.02 ∗ 0.96 ∗ 0.96
=
0.02 ∗ 0.96 ∗ 0.96 + 0.98 ∗ 0.06 ∗ 0.06
= 83.9%
ENCS6161 – p.14/16
Independence of Events
A and B are independent iff
P (A ∩ B) = P (A)P (B)
If A and B are independent
P (A ∩ B)
P (A|B) =
P (B)
P (A)P (B)
=
P (B)
= P (A)
Similarly
P (B|A) = P (B)
ENCS6161 – p.15/16
Independence of Events
Events A1 , A2 , · · · , An are independent if
P (Ai1 ∩ Ai2 · · · ∩ Aik ) = P (Ai1 )P (Ai2 ) · · · P (Aik )
where1 ≤ i1 < i2 · · · < ik ≤ n
Qn
A special case P (A1 ∩ · · · ∩ An ) = i=1 P (An )
In general if events are not necessarily independent
P (A1 ∩ A2 · · · ∩ An ) = P (A1 )P (A2 |A1 )P (A3 |A1 ∩ A2 ) · · ·
· · · P (An |A1 ∩ · · · ∩ An−1 )
ENCS6161 – p.16/16
