0% found this document useful (0 votes)
53 views26 pages

Slide 5 01

Uploaded by

dan
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
53 views26 pages

Slide 5 01

Uploaded by

dan
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 26

ECE 302: Lecture 5.

1 Joint PDF and CDF

Prof Stanley Chan

School of Electrical and Computer Engineering


Purdue University

©Stanley Chan 2022. All Rights Reserved.


1 / 26
What are joint distributions?

Joint distributions are high-dimensional PDF (or PMF or CDF).

fX (x) =⇒ fX1 ,X2 (x1 , x2 ) =⇒ fX1 ,X2 ,X3 (x1 , x2 , x3 )


| {z } | {z } | {z }
one variable two variables three variables
=⇒ . . . =⇒ fX1 ,...,XN (x1 , . . . , xN ).
| {z }
N variables

Notation:
fX (x) = fX1 ,...,XN (x1 , . . . , xN ).

©Stanley Chan 2022. All Rights Reserved.


2 / 26
Why study joint distributions?

Joint distributions are ubiquitous in modern data analysis.


For example, an image from a dataset can be represented by a
high-dimensional vector x.
Each vector has certain probability to be present.
Such probability is described by the high-dimensional joint PDF fX (x).

©Stanley Chan 2022. All Rights Reserved.


3 / 26
Outline
Joint PDF and CDF
Joint Expectation
Conditional Distribution
Conditional Expectation
Sum of Two Random Variables
Random Vectors
High-dimensional Gaussians and Transformation
Principal Component Analysis
Today’s lecture
Joint PMF, PDF
Joint CDF
Marginal PDF
Independence
©Stanley Chan 2022. All Rights Reserved.
4 / 26
Joint PMF
Definition
Let X and Y be two discrete random variables. The joint PMF of X and
Y is defined as

pX ,Y (x, y ) = P[X = x and Y = y ]. (1)

Figure: A joint PMF for a pair of discrete random variables consists of an array of
impulses. To measure the size of the event A, we sum all the impulses inside A.
©Stanley Chan 2022. All Rights Reserved.
5 / 26
Example

Example 1. Let X be a coin flip, Y be a dice. Find the joint PMF.

Solution. The sample space of X is {0, 1}. The sample space of Y is


{1, 2, 3, 4, 5, 6}. The joint PMF is

Y
1 2 3 4 5 6
1 1 1 1 1 1
X=0 12 12 12 12 12 12
1 1 1 1 1 1
X=1 12 12 12 12 12 12

Or written in equation:
1
pX ,Y (x, y ) = , x = 0, 1, y = 1, 2, 3, 4, 5, 6.
12

©Stanley Chan 2022. All Rights Reserved.


6 / 26
Example

Example 2. In the previous example, define A = {X + Y = 3} and


B = {min(X , Y ) = 1}. Find P[A] and P[B].

Solution:
X
P[A] = pX ,Y (x, y ) = pX ,Y (0, 3) + pX ,Y (1, 2)
(x,y )∈A
2
=
12X
P[B] = pX ,Y (x, y )
(x,y )∈B

= pX ,Y (1, 1) + pX ,Y (1, 2) + . . . + pX ,Y (1, 5) + pX ,Y (1, 6)


6
= .
12
©Stanley Chan 2022. All Rights Reserved.
7 / 26
Joint PDF
Definition
Let X and Y be two continuous random variables. The joint PDF of X
and Y is a function fX ,Y (x, y ) that can be integrated to yield a probability:
Z
P[A] = fX ,Y (x, y )dxdy , (2)
A

for any event A ⊆ ΩX × ΩY .

©Stanley Chan 2022. All Rights Reserved.


8 / 26
Example

Example 1. Consider a uniform joint PDF fX ,Y (x, y ) defined on [0, 2]2


with fX ,Y (x, y ) = 14 . Let A = [a, b] × [c, d]. Find P[A].

Solution:

P[A] = P[a ≤ X ≤ b, c ≤ X ≤ d]
Z dZ b
= fX ,Y (x, y )dxdy
c a
Z dZ b
1 (d − c)(b − a)
= dxdy = .
c a 4 4

©Stanley Chan 2022. All Rights Reserved.


9 / 26
Example

Example 2. In the previous example, let B = {X + Y ≤ 2}. Find P[B].

Solution:
Z Z 2 Z 2−y
P[B] = fX ,Y (x, y )dxdy = fX ,Y (x, y )dxdy
B 0 0
Z 2 Z 2−y
1
= dxdy
0 0 4
2
2−y
Z
1
= dy = .
0 4 2

©Stanley Chan 2022. All Rights Reserved.


10 / 26
Review of 2D Integration

Figure: To integrate the probability P[X + Y ≤ 2], we perform a 2D integration


over a triangle.

©Stanley Chan 2022. All Rights Reserved.


11 / 26
Example

Example 3. Consider a joint PDF


(
ce −x e −y , 0 ≤ y ≤ x < ∞,
fX ,Y (x, y ) =
0, otherwise.

Find the constant c.

Solution. There are two ways to take the integration. We choose the
inner integration w.r.t. y first.
Z Z ∞Z x
fX ,Y (x, y )dxdy = ce −x e −y dydx
Ω 0 0
Z ∞
c
= ce −x (1 − e −x ) = .
0 2

Therefore, c = 2.
©Stanley Chan 2022. All Rights Reserved.
12 / 26
Review of 2D Integration

Figure: To integrate the probability P[0 ≤ Y ≤ X ], we perform a 2D integration


over a triangle.

©Stanley Chan 2022. All Rights Reserved.


13 / 26
Marginal PDF

Definition
The marginal PMF is defined as
X X
pX (x) = pX ,Y (x, y ) and pY (y ) = pX ,Y (x, y ), (3)
y ∈ΩY x∈ΩX

and the marginal PDF is defined as


Z Z
fX (x) = fX ,Y (x, y )dy and fY (y ) = fX ,Y (x, y )dx (4)
ΩY ΩX

©Stanley Chan 2022. All Rights Reserved.


14 / 26
Example
1
Example 1. Consider the joint PDF fX ,Y (x, y ) = 4 shown below. Find
the marginal PDFs.

Solution. If we integrate over x and y , then we have




1,
 if 1 < x ≤ 2,
3, if 1 < x ≤ 2,

 
2, if 2 < x ≤ 3,
fX (x) = 1, if 2 < x ≤ 3, and fY (y ) =
  1, if 3 < x ≤ 4,
0, otherwise.
 


0, otherwise.
©Stanley Chan 2022. All Rights Reserved.
15 / 26
Example
Example 2. A joint Gaussian random variable (X , Y ) has joint PDF

((x − µX )2 + (y − µY )2 )
 
1
fX ,Y (x, y ) = exp − .
2πσ 2 2σ 2

Find the marginal PDF fX (x).

Solution.

((x − µX )2 + (y − µY )2 )
Z  
1
fX (x) = 2
exp − dy
−∞ 2πσ 2σ 2
 Z ∞
(x − µX )2 (y − µY )2
  
1 1
=√ exp − · √ exp − dy
2πσ 2 2σ 2 −∞ 2πσ 2 2σ 2
(x − µX )2
 
1
=√ exp − .
2πσ 2 2σ 2
©Stanley Chan 2022. All Rights Reserved.
16 / 26
Independence

Definition
If two random variables X and Y are independent, then

pX ,Y (x, y ) = pX (x)pY (y ), and fX ,Y (x, y ) = fX (x)fY (y ).

Definition
If a sequence of random variables X1 , . . . , XN are independent, then their
joint PDF (or joint PMF) can be factorized.
N
Y
fX1 ,...,XN (x1 , . . . , xN ) = fXn (xn ). (5)
n=1

©Stanley Chan 2022. All Rights Reserved.


17 / 26
Example

Example 1. Consider two random variables with a joint PDF given by

(x − µX )2 + (y − µY )2
 
1
fX ,Y (x, y ) = exp − .
2πσ 2 2σ 2

Then, from the previous example, we know that

(x − µX )2 (y − µY )2
   
1 1
fX ,Y (x, y ) = √ exp − × √ exp −
2πσ 2σ 2 2πσ 2σ 2
| {z } | {z }
fX (x) fY (y )

Therefore, the random variables X and Y are independent.

©Stanley Chan 2022. All Rights Reserved.


18 / 26
Example

Example 2. Consider two random variables X and Y with a joint PDF


given by

fX ,Y (x, y ) ∝ exp −(x − y )2




= exp −x 2 + 2xy − y 2


= exp −x 2 exp {2xy } exp −y 2


 
| {z } | {z } | {z }
fX (x) extra term fY (y )

This PDF cannot be factorized into a product of two marginal PDFs.


Therefore, the random variables are dependent.

©Stanley Chan 2022. All Rights Reserved.


19 / 26
Independent and Identically Distributed (i.i.d.)

Definition (Independent and Identically Distributed (i.i.d.))


A collection of random variables X1 , . . . , XN are called independent and
identically distributed (i.i.d.) if
All X1 , . . . , XN are independent;
All X1 , . . . , XN have the same distribution, i.e., fX1 (x) = . . . = fXN (x).

Why is i.i.d. so important?


If a set of random variables are i.i.d., then the joint PDF can be
written as a product of PDFs.
Integrating a joint PDF is not fun. Integrating a product of
PDFs is a lot easier.

©Stanley Chan 2022. All Rights Reserved.


20 / 26
Example

Example 1. Let X1 , X2 , . . . , XN be a sequence of i.i.d. Gaussian random


variables where each Xi has a PDF
 2
1 x
fXi (x) = √ exp − .
2π 2

The joint PDF of X1 , X2 , . . . , XN is


N   2 
Y 1 x
fX1 ,...,XN (x1 , . . . , xN ) = √ exp − i
2π 2
i=1
N ( N )

1 X x2
i
= √ exp −
2π 2
i=1

©Stanley Chan 2022. All Rights Reserved.


21 / 26
Joint CDF
Definition
Let X and Y be two random variables. The joint CDF of X and Y is the
function FX ,Y (x, y ) such that

FX ,Y (x, y ) = P[X ≤ x ∩ Y ≤ y ]. (6)

Definition
If X and Y are discrete, then
X X
FX ,Y (x, y ) = pX ,Y (x ′ , y ′ ). (7)
y ′ ≤y x ′ ≤x

If X and Y are continuous, then


Z y Z x
FX ,Y (x, y ) = fX ,Y (x ′ , y ′ )dx ′ dy ′ . (8)
−∞ −∞
©Stanley Chan 2022. All Rights Reserved.
22 / 26
Example
Example 1. Let X and Y be two independent uniform random variables
Uniform(0, 1). Then, the joint CDF is
Z x Z y
′ ′
FX ,Y (x, y ) = FX (x)FY (y ) = fX (x )dx fY (y ′ )dy ′
0 0
Z x Z y

= 1dx 1dy ′ = xy .
0 0

Example 2. Let X and Y be two independent uniform random variables


Gaussian(µ, σ 2 ). Then, the joint CDF is
Z x Z y
′ ′
FX ,Y (x, y ) = FX (x)FY (y ) = fX (x )dx fY (y ′ )dy ′
−∞ −∞
   
x −µ y −µ
=Φ Φ ,
σ σ

where Φ(·) is the CDF of the standard Gaussian. ©Stanley Chan 2022. All Rights Reserved.
23 / 26
Properties

Proposition
Let X and Y be two random variables. The marginal CDF is

FX (x) = FX ,Y (x, ∞)
FY (y ) = FX ,Y (∞, y ).

Definition
Let FX ,Y (x, y ) be the joint CDF of X and Y . Then, the joint PDF can be
obtained through

∂2
fX ,Y (x, y ) = FX ,Y (x, y ).
∂y ∂x

©Stanley Chan 2022. All Rights Reserved.


24 / 26
Summary
Joint PMF and PDF:

Joint distributions are high-dimensional PDF (or PMF or CDF).

From 1D to 2D:
Replace 1D integration to 2D integration
Independence:
N
Y
fX1 ,...,XN (x1 , . . . , xN ) = fXn (xn ). (9)
n=1

A collection of random variables X1 , . . . , XN are called independent


and identically distributed (i.i.d.) if
All X1 , . . . , XN are independent;
All X1 , . . . , XN have the same distribution, i.e., fX1 (x) = . . . = fXN (x).
©Stanley Chan 2022. All Rights Reserved.
25 / 26
Questions?

©Stanley Chan 2022. All Rights Reserved.


26 / 26

You might also like