ECE 603

Probability and Random

Processes
Lessons 10-13
Chapter 5
Joint Distributions

© 2020 UMass Amherst Global. All rights reserved.

Objectives
• Explore tools for studying joint distributions of random variables.
• Compare joint distributions of discrete random variables with continuous
random variables.

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

2
Rationale
• If you were to examine the population of your community, you might notice
that each household has a different number of people. Each of those
household members has a different age, a different income, a different
number of hobbies, etc.
• Each of these results is a random variable. In this Lesson, you explore the
concept of comparing two or more random variables, because you grasp
comparing two, the extension to n random variables is straightforward.

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

3
Prior Learning
• Basic Concepts
• Counting Methods
• Random Variables
• Access to the online textbook: https://www.probabilitycourse.com/

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

4
Joint Probability Mass Function (PMF)
PMF:

Joint Probability Mass Function (PMF) for and :

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

5
Joint Probability Mass Function (PMF)
all possible value for

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

6
Joint Probability Mass Function (PMF)

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

7
Joint Probability Mass Function (PMF)
Marginal PMFs
I have , how do I find PMF of ?

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

8
Joint Probability Mass Function (PMF)
Marginal PMFs

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

9
Joint Probability Mass Function (PMF)
Example. Consider two random variables and
pleiad
with joint PMF given in
PYI
Table.
a) Find the marginal PMFs of and .
b) Find . 16 it

c) Are and independent?

Pxylxiy I
Phx on 1
yo
Px N
p Xo p yo

pl't o
Pax a
yea pain x I I I
Chapter 5 © 2020 UMass Amherst Global. All rights reserved.
10
P Yo x 4 Ply o PH txt
RIEF
independent
1
83 not
get
independent
Py x Pyly R V
Rdx x
y y
s yea

RN of Pyly Ys y i
X I
1 6

o.ge
PxCx o
Pyly o

pxyfx
independent
t I I if not
Joint Cumulative Distributive Function (CDF)
Remember that, for a random variable , we define the CDF as

The joint cumulative distribution function of two random variables

in t
and

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

11
Joint Cumulative Distributive Function (CDF)

o
p

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

12
Joint Cumulative Distributive Function (CDF)
Marginal CDFs of and :

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

13
 Joint Cumulative Distributive Function (CDF)
x o
s
Example. Toss a fair coin twice, Px x
fo 5 4 1

g a I
and are independent. Find .

Fxy 31 P xT.I YEI

IT Y
X o
Y o I PH
I P
Chapter 5 © 2020 UMass Amherst Global. All rights reserved.
14
Joint Cumulative Distributive Function (CDF)
Two discrete random variables and are independent if

Equivalently, and are independent if

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

15
Joint Cumulative Distributive Function (CDF)
So far:
Joint PMF
•
• all possible value for
• Marginal PMFs

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

16
Joint Cumulative Distributive Function (CDF)
Joint CDF:

Remember:

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

17
Joint Cumulative Distributive Function (CDF)
Lemma. For two random variables and , and real numbers
, we have
fxylx.gl Plexiyey

PI EX

Fx xx f xD

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

18
Conditioning and Independence
Conditioning:

For example

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

19
Conditioning and Independence
Px
PMF:

PI XENA
Conditional PMF:

Conditional CDF:
Fx

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

20
Conditioning and Independence
Conditional PMF and CDF:
For a discrete random variable and event , the conditional PMF of given
is defined as

Similarly, we define the conditional CDF of given as

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

21
Conditioning and Independence
Let

Conditional PMF of given :

PM
joint
Marginal
w

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

22
Conditioning and Independence
Similarly, we can define the conditional probability of given :

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

23
Conditioning and Independence
For discrete random variables and , the conditional PMFs of given
and vice versa are defined as

for any and

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

24
 Conditioning and Independence

Example. Consider two random variables and with joint PMF given in the
following Table.
Find conditional PMF of given
T
My exily
F f I

Pxly.PH 55 pIX 1iY

ply r z

Pyly 1 2
Pex pix 34,44
Chapter 5 © 2020 UMass Amherst Global. All rights reserved.
25
Conditioning and Independence
Independent Random Variables:
Two discrete random variables and are independent if

Y
in in
Equivalently Pay
Pl
Equivalently

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

26
Conditioning and Independence
Conditional Expectation:

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

27
Conditioning and Independence

Example. Consider two random variables and with joint PMF given in
Table.
Find and .

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

28
 play It ke
fi
1 2
pix 4
5 12

15
Hf
ke
ply I
e o

x 34 7 27
p
IF t

F X 7 2 1 115 a I

Vav x D Efx ya ECHED

t
Conditioning and Independence
Law of Total Probability:

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

29
Conditioning and Independence
Law of Total Probability:
If is a partition of the sample space then we have

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

30
Conditioning and Independence
Law of Total Expectation:
If is a partition of the sample space then we have

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

31
Orchestrated Conversation:
Conditioning and Independence
Example. Suppose that the number of customers visiting a fast food restaurant
in a given day is . Assume that each customer purchases a
drink with probability independently from other customers and independently
from the value of . Let be the number of customers who purchase drinks.
Find .
N N Poisson 7 PII É
Binomial Ni Pl I p i pi
x

XN
T PIN X PAIN PIN
Binomial

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

32
 Y N

EG IT Xé nip

I
P.IN
PE nine
P

p.EE in.e

YEI h o
1 nil 0 n 2
U N 4

EÉI
Mi E
Functions of Two Random Variables
Let and are two random variables, and suppose that
is random variable.
O
Law of the unconscious statistician (LOTUS) for two discrete random variables:

to

Chapter 5
33
Functions of Two Random Variables
Example.
Linearity of Expectation: For two discrete random variables and ,
show that

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

34
Functions of Two Random Variables
General scenario:

Find PMF of .
Es

1)

2) For
er

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

35
Functions of Two Random Variables
Example.
Let and be two independent
random variables and . Find PMF of .

Rx i 2 3
7 Px G Ii Pf P

Py y i pj p
Ry 3
I 1 43

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

36
 Px x
EMI F
Py y M I 314
Rz g 3 s I
t.si l.y
Pz 2 8
Egg 1,14149
xityj 2
I 3
I

Pz 2 3 17.3 4 t 4 s
El
I 3
3
t
3 32 9 39
I
3
I I a

Pecan É Me

E
e s
IT IT
Ee
E r s
3 l l
 Y
2 2

2
H zn n
Palen 3 z

3
4 2 27 3
t 3
Pelzer

212 3 3
H R D I
Conditional Expectation and Variance
Conditional Expectation as a Function of a Random Variable:

Note that depends on the value of , so we can write

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

37
Conditional Expectation and Variance
Thus, we can think of as a function of the value of the random
variable . We then write

If is a random variable with range , then

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

38
Conditional Expectation and Variance
Example. Consider two random variables and with joint PMF given in
the following Table

a) Find the Marginal PMFs of and .

b) Find the conditional PMF of given and .

c) Find the PMF of .

d) Find , and check that .

e) Find .

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

39
Conditional Expectation and Variance
Table. Joint PMF of and in example.

715
P x a

ply 1 I
plied ply il
3 5 45

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

40
 FEI
Ryu led PIPIT
s
Pyly 1 0 4 0
g
Pxly x
Hy 3 43
o

Pxly volt D
É
Pyly x 1
4 1 O

t 43 43
E X 4
03 O
t t
0
E X 4 1 o I 1 0

3 5 43
Pz O
45

E 63 3 3 3 o 8 3
l
E x 3 o t
E
Conditional Expectation and Variance
Rule: Taking out what is known.

Proof: Note that is a random variable that is a function of .

If then

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

41
Conditional Expectation and Variance
Iterated Expectations:
Let
Then,

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

42
Conditional Expectation and Variance
Iterated Expectations:

This is equal to the low of total Expectation.

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

43
Conditioning and Independence
Example. Suppose that the number of customers visiting a fast food restaurant
in a given day is . Assume that each customer purchases a
drink with probability independently from other customers and independently
from the value of . Let be the number of customers who purchase drinks.
Find .

Binomially N P
XIN
X
N N Poisson
ENG P X
EE E EX N EEN PJ P
Chapter 5 © 2020 UMass Amherst Global. All rights reserved.
44
Conditional Expectation and Variance
If and are independent random variables, then

1.

2.
P.MY
Y Ext
NY
LxPx É
II
3.
E
4.

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

45
Conditional Expectation and Variance
Conditional Variance:
Let then

Note that is a function of . We define is a function

of . That is, is a random variable whose value equals

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

46
Conditional Expectation and Variance
Example. Consider two random variables and with joint PMF given in
the following Table. Let .

a) Find the PMF of .

b) Find .

c) Check .

EN ly
END
Chapter 5 © 2020 UMass Amherst Global. All rights reserved.
47
a Vainly
Conditional Expectation and Variance
Law of Total Variance:

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

48
Conditional Expectation and Variance
Example. Let be the number of customers that visit a certain store in a
given day. Suppose that we know and . Let be the amount
that the customer spends on average. We assume ’s are independent of
each other and also independent of . We further assume they have the same
mean and variance

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

49
Conditional Expectation and Variance
Let be the store's total sales, i.e.,

Find and .

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

50
Two Continuous Random Variables
PDF:

Joint PDF:

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

51
Two Continuous Random Variables
If we choose , then

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

52
Two Continuous Random Variables
Definition. Two random variables and are jointly continuous if there
exists a nonnegative function , such that, for any set ,
we have

The function is called the joint probability density function (PDF) of

and .

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

53
Two Continuous Random Variables
CDF:

Joint CDF:

and

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

54
Two Continuous Random Variables

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

55
Two Continuous Random Variables
Example. Let and be two random variables with joint PDF given by

a) Find .
b) Find . t.it

I cannon as i.x.I of X F ie't f I

I
I

X Ei y

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

56
off off
I I fxy
i i x y dx
dy
fi e
I fxylx.gl
Two Continuous Random Variables

Chapter 5
57
Two Continuous Random Variables
The joint cumulative function of two random variables and is defined as

The joint CDF satisfies the following properties:

1) , for any (marginal CDF of );

2) , for any (marginal CDF of );

3)

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

58
Two Continuous Random Variables
4)

5)

6) if and are independent, then

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

59
Two Continuous Random Variables
Marginal PDFs:
For discrete random variables:

For continuous random variables:

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

60
Two Continuous Random Variables
Example. Let and be two random variables with joint PDF given by

Find the marginal PDFs and .

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

61
Conditioning and Independence
Conditional probability:

conditional CDF:

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

62
Conditioning and Independence
Example. Let continuous

If

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

63
Conditioning and Independence
Finally, if , then . Thus, we obtain

Then the conditional PDF of given is given by

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

64
Conditioning and Independence
In general, for a random variable and an event , we have the following:

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

65
Conditioning and Independence
Conditioning by Another Random Variable:
For discrete random variables: the conditional PMF of given is
given by

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

66
Conditioning and Independence
For continuous random variables:
The conditional PMF of given is given by
be

The conditional PMF of given is given by

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

67
Conditioning and Independence
Example. Choose a point Uniformly at random in the unit disc ,

a) Find and .
it
b) Find .
c) Find

in

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

68
 fry lag y e
4

O W
O

Sj
c er I

c t t
c I

É t en

fly y

if
ay
o

tod

f e
f fay xx dy I dy taxi
Y Y I X

FI
Fx
dy y fix

y
II ETF

fyly
fig FF
b

Iggy
Conditioning and Independence
For two jointly continuous random variables and , we have

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

69
Summary of Conditioning
Conditional probability:

conditional CDF and PDF given : (e.g., )

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

70
Summary of Conditioning
In general, for a random variable and an event , we have the following:

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

71
Summary of Conditioning
Conditioning by Another Random Variable:
For discrete random variables: the conditional PMF of given is
given by

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

72
Summary of Conditioning
For continuous random variables:
The conditional PMF of given is given by

The conditional PMF of given is given by

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

73
Summary of Conditioning
For two jointly continuous random variables and , we have

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

74
Conditioning and Independence
Law of Total Probability:

Law of Total Expectation:

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

75
Conditioning and Independence
Law of Total Variance:

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

76
Conditioning and Independence
Example. Suppose and given , then

Find i

fy fylx Fx i

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

77
 of

fry
O
O U
ftp.t
Ittf

y Ex

ya IIof tardy

t.tt

I inxtydy
If tiny dy I Ing dy

Fito
flying y

Sudo u v
Soda finydy ylny SD.tt
why i du tdy ying Y
du dy waxy
Conditioning and Independence
Independent Random Variables:
Discrete:

Continuous:

#:

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

78
Functions of Two Continuous Random Variables
LOTUS for two continuous random variables:

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

79
Functions of Two Continuous Random Variables
Example. Let and be two independent random
variables, and . Find the CDF and PDF of .

it

iii fy
Chapter 5 © 2020 UMass Amherst Global. All rights reserved.
80
 a

Fz P Eez p
xyszy go 271

p TEE o

7i

Z t

Fz z Z Zina y

Int
Fz H Z I
L dx

fl me
fz z Zf zlnyl
luz 2
f luz

Zluz
InZ
Functions of Two Continuous Random Variables
Theorem. Let and be two jointly continuous random variables. Let

, where is a
continuous one-to-one (invertible) function with continuous partial derivatives.
Let , i.e., . Then and
are jointly continuous and their joint PDF, , for is
given by

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

81
 Z XY W Y

W
X E Y
Mf
All
Fxx
W
d c

E ECE
H E w ECW

If't If
s

U ft i off
few_fz
I
w

y g

Uft Ew Afl

O o u
 ofzewei
to
fzwlz.gg O O W

f S't du
Z 4 Z
In w Inz
Z
Functions of Two Continuous Random Variables
Where is the Jacobian of defined by

11

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

82
Functions of Two Continuous Random Variables
Example.
Let and be two independent random variables with PDF
defined
we fx w

Find and . If fx Z W

fay xy fx tx fy y
Z X ty Hiii

x
et
w

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

83
Functions of Two Continuous Random Variables
If and are two jointly continuous random variables and ,
then

If and are also independent, then

In
The
Chapter 5 © 2020 UMass Amherst Global. All rights reserved.
84
Summary of Independence
Two continuous random variables and are independent if

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

85
Conditioning and Independence
Example. Determine whether and are independent:

ÉgÉÉÉé

as E é't
E
Fy E independent

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

86
Conditioning and Independence
Two continuous random variables and are independent, then we have

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

87
Conditioning and Independence
More generally: and independent

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

88
Covariance and Correlation
The covariance between and is defined as
EG EAD

Proof: War Ix E f EC xD
x u x
Guy x u

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

89
Covariance and Correlation
Example. Suppose and given , is

Find
fxx f y ly fix

1
Cov XY EXT EG E G
É ex

O o w
1 5

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

90
EAT
N N
x
If É Ey
o o w

yet dy If É me
fax
O

Edy Jey
e É exec

I
ax
fly o w
o

É
JE
dx

F LyJ
 Covariance and Correlation
Lemma. The covariance has the following properties:
1)

It
2) independent: independent Golay o

3)

4)

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

91
Covariance and Correlation
5)

6)

7)

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

92
Covariance and Correlation
More generally

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

93
Covariance and Correlation
Variance of a sum:
If , then

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

94
Covariance and Correlation
More generally, for , we conclude:

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

95
Covariance and Correlation
Correlation Coefficient:

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

96
Covariance and Correlation
Properties of the correlation coefficient:

1)

2)

3)
I ax t b

van fly o

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

97
Covariance and Correlation
Definition. Consider two random variables and :

1) If , we say that and are uncorrelated.

2) If , we say that and are positively correlated.

3) If , we say that and are negatively correlated.

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

98
Bivariate Normal Distribution
Definition. Two random variables
or jointly normal, if 00
and are said to be bivariate normal,
has a normal distribution for all

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

99
Bivariate Normal Distribution
Example. Let and be two bivariate normal random variables, i.e.,

If , find the PDF of .

ZECH 3 1 2 0 3
G 3E

b'Varly lab Colley

Cov 2 a'vaulx
9 4 4 I 2 3 2 I

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

100
Bivariate Normal Distribution
Definition. Two random variables and are said to have a bivariate
normal distribution with parameters and , if their joint PDF is
given by
é e
rata E

Where and are all constants.

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

101
Bivariate Normal Distribution

If ( are uncorrelated) :

Function of Function of

IF
Chapter 5 © 2020 UMass Amherst Global. All rights reserved.
102
Bivariate Normal Distribution
Theorem. If and are bivariate normal and uncorrelated, then they are
independent. __

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

103
Bivariate Normal Distribution
Example. Let and be two bivariate normal random variables. Find the
conditional PDF of

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

104
Bivariate Normal Distribution
Theorem. Suppose and are jointly normal random variables with
parameters and . Then, given is normally
distributed with

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

105
Bivariate Normal Distribution
Example. Let and be two bivariate normal random variables and
uncorrelated,
EG EA
We define Varix 1
02
a) Find and TI fay
b) Find EW EGHEW
c) Find FIX 411 4 f
3 764 4 9 84
d) Find C
3767 GEOFF
Chapter 5 © 2020 UMass Amherst Global. All rights reserved.
106
Post-work for Lesson
• Complete homework assignment for Lessons 10-13:

HW#6 and HW#7

Go to the online classroom for details.

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

107
To Prepare for the Next Lesson
• Read Chapter 6 in your online textbook:
https://www.probabilitycourse.com/chapter6/6_0_0_intro.php
• Complete the Pre-work for Lessons 14-16.

Visit the online classroom for details.

Chapter 5 © 2020 UMass Amherst Global. All rights reserved.

108