Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

MAE206 Information Engineering :

Basics on probability and statistics

Meryem Benammar

ISAE-Supaéro, DEOS

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 1 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Context

Probability spaces

Discrete random variables and vectors

Continuous random variables and vectors

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 1 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Where is Information ? ...

Where is information contained ?

• Numerical : text, DNA strings, bank transactions, ...
• Audio : music (digital or analog), probe signals (rovers), ...
• Image : camera images, snapshots, paintings, ...
• Video : high motion videos, live streams, ...

Figure – Source : icon-icons.com

Types of information : Semantic or quantitative

We will focus on quantitative information !

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 2 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

What is Information Engineering (IE) ?

Three families of algorithms

We will focus on compression and security !

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 3 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Mathematics behind IE

Where there is uncertainty ⇒ there is information !

Information measurement :
Randomness (uncertainty) : Information theory !
Probabilities and statistics

Figure – Source : www.sci-highs.com/

Figure – Source : www.rfi.fr/

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 4 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

In this course ...

Problem Based Learning (PBL) :

• Mini-projects of 5 to 6 students
• PBL forward, intermediate and return sessions
• Autonomy, rigor, organization, ...

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 5 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

In this course ...

• List and identify basic probability distributions (discrete and continuous)

• Master the operations on random variables and probability distributions
• List and analyze basic results from statistics
• Identify the notion of information with a measure of randomness
• Model an information source with a probabilistic model
• Describe basic information engineering tools
• Evaluate information engineering tools from a probabilistic point of view
• Criticize and suggest improvements to these tools

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 6 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Context

Probability spaces

Discrete random variables and vectors

Continuous random variables and vectors

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 6 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Experiments

• Consider an experiment for which the output is unknown

• Output belongs to an alphabet set Ω :

Experiment Alphabet Ω

Ω = {H, T }

Ω = {1, 2, 3, 4, 5, 6}

Ω = {1, 2, 3, 4, 5, 6} × {1, 2, 3, 4, 5, 6}

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 7 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Events as sets : definition

Definition (Alphabet and Events)
• The set of possible outcomes Ω is called an Alphabet
• An event E is a subset of Ω : E ⊆ Ω
• An event space E is the set of all possible events

Experiment Alphabet Ω Examples of events E

• Result is 1 : E1 = {1}
• Result is even : E2 = {2, 4, 6}
Ω = {1, 2, 3, 4, 5, 6} • Result is ≥ 3 : E3 = {3, 4, 5, 6}
• Result is ≤ 6 : E4 = {1, 2, 3, 4, 5, 6}

• Result is H : E1 = {H}
• Result is T : E2 = {T }
Ω = {H, T } • Result is whatever : E3 = {H, T }
• Coin was lost : E4 = {∅}

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 8 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Events as sets : axioms

Figure – Set and event operations

Operation on event Operation on sets

Event 1 and Event 2 E1 ∩ E2
Event 1 or Event 2 E1 ∪ E2
Not event 2 E2c = Ω \ E2
Event 1 except Event 2 E1 \ E2
Null event ∅
Trivial event Ω
Table – Set axioms

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 9 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Probability measure

Definition (Probability measure)

Let Ω be an alphabet, E be its event space. A probability measure P is defined as

P:E → [0, 1]
E → P(E)

where P verifies the following axioms

X
• Exhaustivity : P(Ω) = P(w) = 1
w∈Ω
• Additivity : if two events E1 and E2 are disjoint, then

E1 ∩ E2 = ∅ ⇒ P(E1 ∪ E2 ) = P(E1 ) + P(E2 ).

Value w ∈ Ω H T Event E ⊂ Ω {H} {T } {H, T } ∅

⇒
P(w) 0.5 0.5 P(E) 0.5 0.5 1 0
Table – Example : Fair coin toss

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 10 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Probability measure : axioms

Event complement P(E c ) = 1 − P(E)

Union of events P(E1 ∪ E2 ) = P(E1 ) + P(E2 ) − P(E1 ∩ E2 )

X
Exhaustivity P(w) = 1
w∈Ω

Null event P(∅) = 0

X
Elementary measures P(E) = P(w)
w∈E

Subsets E1 ⊂ E2 ⇒ P(E1 ) ≤ P(E2 )

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 11 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Probability space

Definition (Probability space)

A probability space P = (Ω, E, P) is defined by three components
• An alphabet set Ω representing a set of all possible outputs
• An event set E consists in all possible subsets of the alphabet Ω
• A probability measure P

Experiment Probability measure P Probability space

• P1 (H) = 0.5
• P1 (T ) = 0.5 ({H, T }, P({H, T }), P1 )
Fair coin flip

• P2 (H) = 0.7
• P2 (T ) = 0.3 ({H, T }, P({H, T }), P2 )
Rigged coin flip

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 12 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Random variables

Definition
Let us consider a probability space P = (Ω, E, P) and (X , EX ) a new alphabet
with its event space. A random variable X is a mapping from Ω to X

X: Ω → X
w → X(w).
• The probability measure PX is defined, for all events Ex in EX

PX (Ex ) = P(X −1 (EX )) = P({w ∈ Ω, X(w) ∈ Ex }

• The probability space associated with X is PX = (X , EX , PX ).

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 13 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Random variables : examples

Consider a fair die throw with the probability space :

 
1 1
Ω = {1, ..., 6} , E = P(Ω) , P= ,...,
6 6

We can define a variety of random variables X : w → X(w)

Event w Random variable X(w) Alphabet X Probability

 PX 
Output w itself : 1 1
Value w X =Ω PX = ,...,
X(w) = w 6 6
Parity (even/odd) :
Value w X = {0, 1} PX = {0.5, 0.5}
X(w) = w[2]
Threshold value :
Value w X = {0, 1} PX = {0.33, 0.666}
X(w) = (w ≥ 3)?  
1 1
Value w Square : X(w) = w2 X = Ω2 PX = ,...,
6 6

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 14 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Context

Probability spaces

Discrete random variables and vectors

Continuous random variables and vectors

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 14 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Probability mass function

Definition (Probability mass function (pmf))

The probability mass function of a discrete random variable X with associated
probability measure PX is defined by

PX : X → [0 : 1]
x → PX (x) = PX (X = x) = PX ({x})

We have by definition of the pmf that

X
PX (x) = 1.
x∈X

The probability mass function PX (·) RULES them all !

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 15 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Classical discrete random variables

1. Bernoulli of parameter p (Bern(p)) : Coin toss with Head probability p

2. Discrete uniform over the interval [1 : K] (Unif ([1 : K])) : Fair dice throw

3. Binomial with parameter (n, p) (Binom(n, p)) : number of heads in n coin flips

4. Constant variable equal to K (Const(K)) : stale dice at value K

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 16 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Moments : expectation and variance

Definition (Moments)
To each random variable X with pmf PX are associated
• An expected (average) value E(X) (first order moment)
X
E(X) , x.PX (x)
x∈X

• A variance (squared standard deviation) V(X) (second order moment)

V(X) , E X 2 − E2 (X).

Examples :

Random law Expectation E(X) Variance V(X)

Bern(p) p p(1 − p)
K+1 K 2 −1
Unif ([1 : K]) 2 2
Const(K) K 0

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 17 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Pair of random variables : joint pmf

Let (X, Y ) be a pair of random variables resulting from a joint experiment.

Definition (Joint pmf)

The joint pmf PX,Y associated with the pair (X, Y ) is given by

X ×Y → [0, 1]
(x, y) → PX,Y (x, y) = P(X = x and Y = y)

Examples : Tossing two coins

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 18 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Pair of random variables : marginal pmfs

Definition (Marginal pmfs)

To the joint pmf PX,Y are associated two marginal pmfs PX and PY defined by
X X
PX (x) = PX,Y (x, y) , PY (y) = PX,Y (x, y)
y∈Y x∈X

Examples : Tossing two coins

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 19 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Pair of random variables : joint and marginal pmfs

X X X
PX,Y (x, y) = 1 and PX (x) = PY (y) = 1
(x,y)∈X ×Y x∈X y∈Y

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 20 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Pair of random variables : conditional pmfs

Definition (Conditional pmfs)
The conditional pmfs associated with PX,Y are defined by

PX,Y (x, y)
PX|Y (x|y) = = P (X = x|Y = y)
PY (y)
PX,Y (x, y)
PY |X (y|x) = = P (Y = y|X = x)
PX (x)

Examples : Tossing two coins

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 21 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Pairs of random variables : conditional pmfs

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 22 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Bayes’ formula

Definition
Let (X, Y ) be a pair of random variables with joint pmf PX,Y . Assume that we
only know PX and PY |X , then Bayes’ formulae write as

PX (x)PY |X (y|x)
PX|Y (x|y) = X
PX (x0 )PY |X (y|x0 )
x0

Very useful in machine learning, signal processing (estimation, detection),

communication engineering....

The joint pmf PX,Y RULES them all !

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 23 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Random vectors
• A binary stream of n = 10 bits iid Bern(0.5)
U k = (0, 1, 1, 0, 0, 0, 1, 0, 1, 0, 0, 1, 1)
• A random image of n = 20 × 20 gray-level iid pixels

• n = 100 consecutive iid samples of a random noise U nif [−5 : 5]

5

-1

-2

-3

-4
0 20 40 60 80 100

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 24 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Random vectors : joint pmf

Generalize to a vector of random variables (X1 , ..., Xn ).

Definition (Joint and marginal pmfs)

The joint pmf of the vector can be defined as PX1 ,...,Xn

X1 × . . . Xn → [0, 1]
(x1 , . . . , xn ) → PX1 ,...,Xn (x1 , . . . , xn ) = P(X1 = x1 , . . . , Xn = xn )

To the joint pmf PX1 ,...,Xn are associated n marginal pdfs

X
PXi (xi ) = PX1 ,...,Xn (x1 , . . . , xn )
(x1 ,...,xn )\xi

Example : Throwing a dice n consecutive times

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 25 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Random vectors : chain rule

Definition (The chain rule)
The joint pmf can be expanded using the so-called chain rule as follows
n
Y
PX1 ,...,Xn (x1 , . . . , xn ) = PXi |X1 ,...,Xi−1 (xi |x1 , . . . , xi−1 )
i=1
n
Y PX1 ,...,Xi (x1 , . . . , xi )
=
i=1
PX1 ,...,Xi−1 (x1 , . . . , xi−1 )

Example :
1. A set of variables (X1 , . . . , Xn ) are pairwise independent
n
Y
PX1 ,...,Xn (x1 , . . . , xn ) = PXi (xi ),
i=1

2. If, further, the variables are identically distributed (iid), i.e., they follow the
same law PX , then
n
Y
PX1 ,...,Xn (x1 , . . . , xn ) = PX (xi ).
i=1

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 26 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Context

Probability spaces

Discrete random variables and vectors

Continuous random variables and vectors

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 26 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Probability density function : pdf

A continuous random variables X takes values over an interval X in R.

Definition (Probability density function)

Let X be a continous random variable (X : ω → R), then the associated
probability density function is denoted by fX and verifies the following
properties :
• Positivity :
∀x ∈ R, fX (x) ≥ 0
• Exhaustivity : Z
fX (x) dx = 1
x∈X

• Interval probability :
Z b
P(a ≤ X ≤ b) = fX (x) dx
a

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 27 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Moments : expectation, variance, ...

Definition (Moments)
To each random variable X with pmf PX / pdf fX are associated
• An expected value E(X) (first order moment)
Z
x.fX (x) dx.
x∈X

• A variance V(X) (second order moment)

V(X) , E X 2 − E2 (X).

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 28 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Moments : expectation, variance, ...

Properties (Moments)

1. The expectation is linear, i.e.,

E(f (X)) = f (E(X))

for any linear transformation f .

2. For any constant α, we have that

V(α.X) = α2 V(X).

3. For any constant c, we have that

V(X + c) = V(X).

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 29 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Cumulative distribution function (cdf)

Definition (Cumulative distribution function (cdf))

Let X be a continuous random variable with associated pdf fX . Then, to X is
associated a cumulative distribution function (cdf) FX defined by
Z a
FX (a) = P(X ≤ a) = fX (x) dx
−∞

Figure – Cumulative distribution function : integration

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 30 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Cumulative distribution function (cdf)

Properties (Cumulative distribution function (cdf))
1. Exhaustivity
FX (∞) = 1
2. Increasing function
a1 ≤ a2 ⇒ FX (a1 ) ≤ FX (a2 )
3. Positivity
FX (−∞) = 0
4. Relation to pdf
∂FX
fX (x) = (x)
∂x

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 31 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Continuous random variables : examples

• Uniform over an interval [a : b] (∼ Unif [a, b])

• Gaussian with mean µ and variance σ 2 (∼ N (µ, σ 2 ))

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 32 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Continuous random vectors

Definition (Joint and marginal pds)

The joint pdf of the vector can be defined as fX1 ,...,Xn

X1 × . . . Xn → [0, 1]
(x1 , . . . , xn ) → fX1 ,...,Xn (x1 , . . . , xn )

To the joint pdf fX1 ,...,Xn are associated n marginal pdfs

Z
fXi (xi ) = fX1 ,...,Xn (x1 , . . . , xn ) dx1 ...dxn
(x1 ,...,xn )\xi

Chain rule
n
Y
fX1 ,...,Xn (x1 , . . . , xn ) = fXi |X1 ,...,Xi−1 (xi |x1 , . . . , xi−1 )
i=1
n
Y fX1 ,...,Xi (x1 , . . . , xi )
=
i=1
fX1 ,...,Xi−1 (x1 , . . . , xi−1 )

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 33 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Law of large numbers

Theorem (Law of Large Numbers (LLN))

Let (X1 , . . . , Xn ) be n iid random variables, with pmf PX /pdf fX , and let
µ = E(X) be the expectation of X.
The empirical mean X̄n of (X1 , . . . , Xn ), defined by
n
1X
X̄n , Xi
n i=1

converges in probability, as n, grow infinite, to µ, i.e.,

 
P lim X̄n = µ = 1.
n→∞

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 34 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Central limit theorem

Theorem (Central limit theorem (CLT))
Let (X1 , . . . , Xn ) be n iid random variables, with pmf PX /pdf fX , and let
µ = E(X) be the expectation√of X and σ 2 be its variance.
The random variable Zn = n(X̄n − µ) defined by
√ Xn
n
Zn = Xi − µ
n i=1

converges in distribution, as n, grow infinite, to a normal Gaussian distribution

lim P (Zn ≤ z) = Φ(z).

n→∞

where Φ(z) is the cdf of a normal Gaussian distribution.

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 35 / 36

 Context Probability spaces Discrete random variables and vectors Continuous random variables and vectors

Kullback-Leibler divergence

Definition (Kullback-Leibler (KL) divergence)

Let PX and QX be two probability distributions on X .
The KL divergence between PX and QX is defined as
Z  
X PX (x)
DKL (PX ||QX ) , PX (x) log .
QX (x)
x∈X

The KL is extensively used in statistical information engineering

Properties
The KL divergence verifies a certain set of properties :
1. Asymmetry : DKL (PX ||QX ) 6= DKL (QX ||PX ).
2. Null element : DKL (PX ||PX ) = 0.
3. Positivity : DKL (PX ||QX ) ≥ 0, for all laws (PX , QX ).

Meryem Benammar (ISAE-SUPAERO) MAE 206 : IE October 2023 36 / 36