Probability & Random Variables

2023 Spring

Prof. Bang Chul Jung

1
 Lecture 01
Probabilistic Model

Target:
• To learn the definition of probability, which implies
a mathematical description of uncertain situations

2
 Roadmap Roadmap

(1) Probabilistic Model

Mathematical description of uncertain situations

(2) Sample Space, Event, Probability Law

Elements of probability theory

(3) Probability Axioms

3 axioms for the completeness of a theory

Probability & Random Variables 3 Mar. 6, 2023

 Roadmap Roadmap

(1) Probabilistic Model

Mathematical description of uncertain situations

(2) Sample Space, Event, Probability Law

Elements of probability theory

(3) Probability Axioms

3 axioms for the completeness of a theory

Probability & Random Variables 4 Mar. 6, 2023

What
What
What
What
What DoWe
Do
Do
Do DoWe
We
WeWe Want? What Do We Want?
Want?
Want?
Want?
Want?
Modeling:
Modeling:
Modeling:
Modeling:
Modeling: Understand
Understand
Understand
Understand
Understand reality
reality
reality
reality
reality with
with with
with
with
a
a aa simple
simple
simple
simplea (mathematical)
simple (mathematical)
(mathematical)
(mathematical)
(mathematical) model model
model
model
model
Modeling: Understand reality with a simple (mathematical) model
Experiment
Experiment
Experiment FlipFlip
Flip twotwo
two coins
coins
coins
Experiment
Experiment FlipFlip
two two
coinscoins
Observation:
Observation:
Observation: a random
aa random
random outcome
outcome
outcome for example,
for example,
for example, (H,H)
(H, (H,
H) H)
Observation:
Observation: aarandomrandomoutcome
outcome for for example,
example, (H, H)(H, H)
All All outcomes
Alloutcomes
outcomes {(H,{(H,
{(H,
H),
H),(H,
H),
(H,
(H,T T(T
T),),(T),,(T , H),
H),
, H),
(T
(T, ,T
(T T)},)}T )}
All outcomes
outcomes {(H, H), H),
{(H, (H, T ), (T
(H, , H),
T ), (T ,(T , T(T
H), )} , T

Ourgoal:
•• Our goal: Build
Build up
up aa probabilistic model for
for an
an experiment
experiment with
withrandom
randomoutcomes
outcomes
•
• Our
Our goal:
goal:Build
Buildupupa a probabilistic model
probabilistic modelfor for
an experiment with with
an experiment random outcom
random o
• Probabilistic model?
• Probabilistic model?
- Assign a number to each outcome or a set of outcomes
- Assign a number to each outcome or a set of outcomes
- Mathematical description of an uncertain situation
- Mathematical description of an uncertain situation
L1(1)
L1(1) September 23,
September
September 23, 2021
23, 2021
2021 4 //
44 /
• Which model is good or bad?
L1(1)
L1(1) September 23, 2021
September 23, 2021 4 / 20
4 / 20

L1(1) September 23, 20

L1(1) Septemb
Probability & Random Variables 5 Mar. 6, 2023
 Probabilistic Probabilistic Model
ProbabilisticModel
Model
Probabilistic
Probabilistic Model
Model
Probabilistic
Goal: Build
Goal: Model
upup
Build a probabilistic model.
a probabilistic Hmm...
model. Hmm...How?
How?
Goal:
Goal: Build
Build up
up aa probabilistic
probabilistic model. Hmm... How?
How?
Goal:
The
The Build
first upthing:
a probabilistic
thing:
first What are model.
the Hmm... How?
elements of a probabilistic model?
The
The first
first thing:
thing: What
What are
are the elements of a probabilistic
probabilistic model?
model?
The first thing: What are the elements of a probabilistic model?
Elements
Elements of
of Probabilistic
Probabilistic Model
Elements of Probabilistic Model
interest: Sample Space ⌦
1. All
1. All outcomes
outcomes of
of my
my interest:
1. All outcomes of my interest: Sample Space ⌦
2. Assigned
2. Assigned numbers to each outcome of ⌦: Probability Law P(·)
numbers to
2. Assigned numbers to each outcome of ⌦: Probability Law P(·)

Question:
Question:
Question:
Question:
Question: What are the conditions of ⌦ and P(·) under which their induced probability
model becomes ”legitimate”?
L1(1)
L1(1) September
September 23, 2021 55 // 20
L1(1) September23,
23,2021
2021 5 /20 20

L1(1) September 23, 2021 5 / 20

L1(1)
L1(1) Septemb2
September

Probability & Random Variables 6 Mar. 6, 2023

 Roadmap Roadmap

(1) Probabilistic Model

Mathematical description of uncertain situations

(2) Sample Space, Event, Probability Law

Elements of probability theory

(3) Probability Axioms

3 axioms for the completeness of a theory

Probability & Random Variables 7 Mar. 6, 2023

le Space ⌦ Space ⌦
1. Sample 1. Sample Space Ω
1. 1. Sample
1. Sample
Sample Space
Space Space
⌦
⌦ ⌦
1. Sample
et of allThe
outcomes
Space ⌦
ofoutcomes
my interest
setset
The of of
allall outcomesofof my
my interest
interest
1.The
The set
Sample
set of
of all outcomes
Space
all ⌦
outcomes
1. Sample Space ⌦
of
of my interest
my interest
The set of all outcomes of my interest
(1) (1)setMutually exclusive 1. Toss 1. Toss
Toss aa coin.
coin. Whata coin.
aboutWhat
this? about this?
(1) Mutually
The
Mutually exclusive
of all outcomes
exclusive of my interest
The set of all outcomes of my interest 1. What about this?
(1) Mutually exclusive ⌦=
⌦ {H,1.
= {H, TT,⌦Toss
HT
,HT a coin.
= }}{H, What
T , HT } about this?
(2)(1) Mutually
Collectively exclusive
exhaustive ⌦ = {H,
1. Toss a coin. What about HT }
T ,this?
1. Toss a coin. What about this?
(1) Mutually exclusive
(2) Collectively exhaustive 2. Toss a coin.
⌦ = {H, } about this? ⌦ = {H}
What
T , HT
(2) Collectively exhaustive ⌦ =2.{H,
Toss, HT
T }
a coin. What about this? ⌦
(2) Collectively exhaustive 2. Toss a coin. What about this? ⌦ = {H}
2. Toss a coin. What about this? ⌦ = {H}
(3) At(3)
theAt right
the granularity
right granularity 3. (a)3.Just figuring out prob. of H or T.
(a) Just figuring out prob. of H or T.
(not too
(notconcrete, not too
too concrete, notabstract)
too abstract) =) ⌦=) = {H, } T}
⌦ =T{H,
(b) The impact of the weather (rain or no
rain) on the coin’s behavior.

=) ⌦ = {(H, R), (T , R),

(H, NR), (T , NR)},

R(Rain), NR(No Rain).

L1(2)
L1(2) September23,
September 23,2021
2021 77//20
20
L1(2) September 23,
L1(2) September 23, 2021 7 / 20
L1(2)
L1(2) September
September 23, 2021 7/2
L1(2) September 2

Probability & Random Variables 8 Mar. 6, 2023

 my interest Sample Space Ω
et of all outcomes of Examples:
Examples: Sample Space ⌦
Examples: Sample Space ⌦

Discrete
Discretecase:
case: Two
Two rolls of aa tetrahedral
tetrahedral Continuouscase:
Continuous case: Dropping
Dropping a needle
a needle in ain a
die
die plain
plain

{(1,1),
- -⌦⌦=={(1, 1),(1,
(1,2),
2), . . . , (4, 4)}
4)} - -⌦⌦=={(x,
{(x, R2R|20|0 x,
y )y2) 2  yx,y 1}
 1}

L1(2) September 23, 2021 8 / 20

L1(2) September 23, 2021 8 / 20

L1(2) September 23
Probability & Random Variables 9 Mar. 6, 2023
 2. Probability Law Probability Law
2.Probability
2.2. Probability
Probability LawLaw
Law
2.
2. Probability Law
ProbabilityLaw
Probability
2. Probability
• Assign numbersLaw
to what? Each outcome?
• Assign numbers to what? Each outcome?
• ••Assign numbers to what?
•• Assign
Assign
Assign
Assign numbers
numbers toto
numbers to Each
what?
what?
what? Each
what? outcome?
outcome?
Each
Each Each outcome?
outcome?
outcome?
• What is the probability of dropping a needle at (0.5, 0.5) over the 1 ⇥ 1 plane?
• ••What
• What isisis
Assign
What the
theprobability
probability
numbers
the to
probabilityof
of dropping
dropping
what?
of a
Each
droppinga needle
needle at(0.5,
at
outcome?
a needle (0.5,
at 0.5)
0.5)
(0.5, over
over
0.5) the
the 1
over ⇥11⇥plane?
the 1⇥
1 plane?
•• What
What is the probability of dropping a needleaatneedle
of dropping (0.5, 0.5)
at over
(0.5,the0.5) 11plane?
plane?
1 ⇥ over the 1
• •Assign numbers
is the toprobability
to each
each subset of ⌦⌦
••• Assign
What
Assign
Assign
numbers
numbers of
of dropping
subset
to each subset ⌦ a needle at (0.5, 0.5) over the 1 ⇥ 1 p
of ⌦
of
• Assign numbers to each subset of ⌦
••• aaAssign numbers
subset of ⌦: an to each subset of ⌦
event
• a subset of ⌦: an event
• a subset of ⌦: an event
• P(A): Probability of an event A.
• P(A): Probability of an event A.
This is where probability meets set theory.
This is where probability meets set theory.

• Roll L1(2)
a dice. What is the probability of odd numbers? September 23, 2021
L1(2) September 23, 20219 / 20
P({1,
L1(2)
3, 5}), where {1, 3, 5} ⇢ ⌦ is an event.
L1(2) September 23, 2021
September 23, 2021
L1(2)
L1(2) September
September23,23,
2022

Probability & Random Variables 10 Mar. 6, 2023

 Roadmap Roadmap

(1) Probabilistic Model

Mathematical description of uncertain situations

(2) Sample Space, Event, Probability Law

Elements of probability theory

(3) Probability Axioms

3 axioms for the completeness of a theory

Probability & Random Variables 11 Mar. 6, 2023

 Howshould
How should
we we P(·)?P(·)?
construct
How Should
construct We Construct ℙ( ⋅ )?
How should we construct P(·)?
• •Need
Need to construct
to construct P(·) naturally
P(·) that that naturally
satisfiessatisfies the intention
the intention of a theory
of a probability probability t
designer
designer justjust
like like
you. you.
WhatWhat
about about the•followings
the followings Need startingP(·)
to construct
aspoints?
as starting that natural
points?
designer just like you. What about
P(A) 0 for any event A ⇢ ⌦
P(A) 0 for any event A ⇢ ⌦
P(A [ B) = P(A) + P(B) P(A \ B)
P(A [ B) = P(A) + P(B) P(A \ B
P(A [ B)  P(A) + P(B)
P(A [ B)  P(A) + P(B)
For two disjoint1 events A and B, P(A [ B) = P(A) + P(B)
For two disjoint1 events A and B, P
P(⌦) = 1 (Why not P(⌦) = 10?)
P(⌦) = 1 (Why not P(⌦) = 10?)
P(;) = 0
P(;) = 0
If A ⇢ B, P(A)  P(B)
If A ⇢ B, P(A)  P(B)
1 many others
Their intersection is empty.
L1(3) many others September 23, 2021 11 / 20
1
Their intersection is empty.
L1(3) 1 September
Their intersection is empty.
L1(3)
Probability & Random Variables 12 Mar. 6, 2023
Probability
Probability Axioms
Axioms
Probability Axioms Probability Axioms

• Surprisingly,
Surprisingly, we
wewe
we
• Surprisingly, need
need
need just
just
just
need the
the
justthe
the following
following
following
following three
three
three
three rules
rules
rules
rules (called
(called
(called
(called axioms):
axioms):
axioms):
axioms):

Probability
Probability Axioms:
Axioms: Version11
Version
Probability Axioms: Version 1
A1. Nonnegativity: P(A) 0 for any event A ⇢ ⌦
A1. Nonnegativity:P(A)
A1. Nonnegativity: P(A) 0 0forfor
any event
any A ⇢A⌦⇢ ⌦
event
A2. Normalization: P(⌦) = 1
A2. Normalization: P(⌦) = 1
A2. Normalization: P(⌦) = 1
A3. (Finite) additivity: For two disjoint events A and B, P(A [ B) = P(A) + P(B)
A3. (Finite) additivity: For two disjoint events A and B, P(A [ B) = P(A) + P(B)
A3. (Finite) additivity: For two disjoint events A and B, P(A [ B) = P(A) + P(B)
• No other things are necessary, and we can prove all other things from the above
• No other things are necessary, and we can prove all other things from the above
axioms.
axioms.
• Note that coming up with the above axioms is far from trivial.
L1(3) September 23, 2021 12 / 20

L1(3) September 23, 2021 12

L1(3) September 23, 2021
L1(3) September 23, 202

Probability & Random Variables 13 Mar. 6, 2023

Examples Examples

A1: Nonnegativity, A2: Normalization, A3: Finite additivity

Prove the following properties using the axioms:

1. For any event A, P(A)  1

A2 A3 A1
c c c
1 = P(⌦) = P(A [ A ) = P(A) + P(A ) =) P(A) = 1 P(A )  1
2. P(;) = 0
A3 A2 from 1.
P(⌦ [ ;) = P(⌦) + P(;) = 1 + P(;) =) P(;) = 0
3. If A ⇢ B, P(A)  P(B)
A3 A1
P(B) = P(A) + P(B \ A) P(A)

L1(3) Septembe

Probability & Random Variables 14 Mar. 6, 2023

 Probability
Probability Calculation Steps Calculation Steps

Toss a (biased) coin

1. Specify the sample space 1. ⌦ = {H, T }
2. Specify a probability law 2. P({H}) = 1/4, P({T }) = 3/4,
- from my earlier belief, from data, from
expert’s opinion 3. probability of head or tail

3. Identify an event of interest 4. 1/4, 3/4

4. Calculate

L1(3) September 23, 2021 14 /

Probability & Random Variables 15 Mar. 6, 2023

Discrete but infiniteBut
Discrete sample space
Infinite Sample Space

1
• ⌦ = {1, 2, 3, . . .}, P({n}) = 2n , n = 1, 2, . . .
• Is the above probability law legitimate? seems OK
1 1 1/2
P(⌦) = + 2 + · · · = =1
2 2 1 1/2
• P(even numbers)?
P(even) = P({2, 4, 6, . . .})
1 1 1
= 2 + 4 + 6 + · · · = 1/3
2 2 2

• Is the above right? If not, why?

Wrong: Finite additivity axiom does not allow this.

Probability & Random

L1(3)
Variables 16 Mar. 6, 2023
Probability Axioms Version 1
Probability Axioms Version 1

Probability Axioms: Version 1

A1. Nonnegativity: P(A) 0 for any event A ⇢ ⌦
A2. Normalization: P(⌦) = 1
A3. (Finite) additivity: For two disjoint events A and B, P(A [ B) = P(A) + P(B)

L1(3) September 23, 2021 16 / 20

Probability & Random Variables 17 Mar. 6, 2023

Probability AxiomsProbability
Version 2 Axioms Version 2

Probability Axioms: Version 2

A1. Nonnegativity: P(A) 0 for any event A ⇢ ⌦
A2. Normalization: P(⌦) = 1

A3. Countable additivity: If A1 , A2 , A3 , . . . is an infite sequence of disjoint events,

then P(A1 [ A2 [ · · ·) = P(A1 ) + P(A2 ) + · · · .

L1(3) September 23, 2021 16 / 20

Probability & Random Variables 18 Mar. 6, 2023

Interpretation ofInterpretation of Probability Theory
Probability Theory

• A narrow view: A branch of math

axioms ! theorems
Mathematicians work very hard to find the smallest set of necessary axioms (just like
atoms in physics)
• Frequencies: P(H) = 1/2
Understanding an uncertain situation: fractions of successes out of many experiments
• Beliefs: P(He is reelected) = 0.7

Anyway, we believe that probabilistic reasoning is very helpful to understand the world
with many uncertain situations.

L1(3) September 23, 2021 17 / 20

Probability & Random Variables 19 Mar. 6, 2023

 Summary & Review Questions

You
You build
build up the very basics of
of aa probabilistic
probabilistic model.
model.
Review Questions
What
What else
else do we need to build up?
up?

1) Explain what a probabilistic model is and why we need it.

2) What is the mathematical definition of event?
L1(3)
L1(3) September23,23,
September 2021
2021 1919 / 20
/ 20
3) What are the key elements of the probabilistic model?
4) List up the probability axioms and explain them. Are you going to choose the same
axioms to build up the probability theory?
5) Why do we need countable additivity in the probability axioms?

Probability & Random Variables 20 Mar. 6, 2023

L1(3) September 23, 2021 20 / 20
 Thank You for Your
Attention!
Prof. Bang Chul Jung
URL: https://sites.google.com/site/bangchuljung
Email: [email protected]

Probability & Random Variables 21 Mar. 6, 2023