Lecture Notes 3

Multiple Random Variables

• Joint, Marginal, and Conditional pmfs

• Bayes Rule and Independence for pmfs

• Joint, Marginal, and Conditional pdfs

• Bayes Rule and Independence for pdfs

• Functions of Two RVs

• One Discrete and One Continuous RVs

• More Than Two Random Variables

Corresponding pages from B&T textbook: 110-111, 158-159, 164-170, 173-178, 186-190, 221-225.

EE 178/278A: Multiple Random Variables Page 3 – 1

Two Discrete Random Variables – Joint PMFs

• As we have seen, one can define several r.v.s on the sample space of a random
experiment. How do we jointly specify multiple r.v.s, i.e., be able to determine
the probability of any event involving multiple r.v.s?

• We first consider two discrete r.v.s

• Let X and Y be two discrete random variables defined on the same experiment.
They are completely specified by their joint pmf

pX,Y (x, y) = P{X = x, Y = y} for all x ∈ X , y ∈ Y

P P
• Clearly, pX,Y (x, y) ≥ 0, and x∈X y∈Y pX,Y (x, y) = 1

• Notation: We use (X, Y ) ∼ pX,Y (x, y) to mean that the two discrete r.v.s have
the specified joint pmf

EE 178/278A: Multiple Random Variables Page 3 – 2

• The joint pmf can be described by a table
Example: Consider X, Y with the following joint pmf pX,Y (x, y)

X
1 2 3 4
1 1/16 0 1/8 1/16
Y 2 1/32 1/32 1/4 0
3 0 1/8 1/16 1/16
4 1/16 1/32 1/16 1/32

EE 178/278A: Multiple Random Variables Page 3 – 3

Marginal PMFs

• Consider two discrete r.v.s X and Y . They are described by their joint pmf
pX,Y (x, y). We can also define their marginal pmfs pX (x) and pY (y). How are
these related?
• To find the marginal pmf of X , we use the law of total probability
X
pX (x) = p(x, y) for x ∈ X
y∈Y

Similarly to find the marginal pmf of Y , we sum over x ∈ X

• Example: Find the marginal pmfs for the previous example
X pY (y)
1 2 3 4
1 1/16 0 1/8 1/16
Y 2 1/32 1/32 1/4 0
3 0 1/8 1/16 1/16
4 1/16 1/32 1/16 1/32
pX (x)

EE 178/278A: Multiple Random Variables Page 3 – 4

 Conditional PMFs

• The conditional pmf of X given Y = y is defined as

pX,Y (x, y)
pX|Y (x|y) = for pY (y) 6= 0 and x ∈ X
pY (y)

Also, the conditional pmf of Y given X = x is

pX,Y (x, y)
pY |X (y|x) = for pX (x) 6= 0 and y ∈ Y
pX (x)

• For fixed x, pY |X (y|x) is a pmf for Y

• Example: Find pY |X (y|2) for the previous example

EE 178/278A: Multiple Random Variables Page 3 – 5

• Chain rule: Can write

pX,Y (x, y) = pX (x)pY |X (y|x)

• Bayes rule for pmfs: Given pX (x) and pY |X (y|x) for all (x, y) ∈ X × Y , we can
find
pX,Y (x, y)
pX|Y (x|y) =
pY (y)
pY |X (y|x)
= pX (x)
pY (y)
pY |X (y|x)
= P pX (x), by total probability
pX,Y (x′, y)
x′ ∈X

Using the chain rule, we obtain another version of Bayes rule

pY |X (y|x)
pX|Y (x|y) = P pX (x)
pX (x′)pY |X (y|x′ )
x′ ∈X

EE 178/278A: Multiple Random Variables Page 3 – 6

 Independence

• The random variables X and Y are said to be independent if for any events
A ∈ X and B ∈ Y

P{X ∈ A, Y ∈ B} = P{X ∈ A}P{Y ∈ B}

• Can show that the above definition is equivalent to saying that the r.v.s X and
Y are independent if

pX,Y (x, y) = pX (x)pY (y) for all (x, y) ∈ X × Y

• Independence implies that pX|Y (x|y) = pX (x) for all (x, y) ∈ X × Y

EE 178/278A: Multiple Random Variables Page 3 – 7

Example: Binary Symmetric Channel

• Consider the following binary communication channel

Z ∈ {0, 1}

X ∈ {0, 1} Y ∈ {0, 1}

The bit sent X ∼ Bern(p), the noise Z ∼ Bern(ǫ), the bit received
Y = (X + Z) mod 2 = X ⊕ Z , and X and Z are independent. Find
1. pX|Y (x|y),
2. pY (y), and
3. the probability of error P{X 6= Y }

EE 178/278A: Multiple Random Variables Page 3 – 8

1. To find pX|Y (x|y) we use Bayes rule
pY |X (y|x)
pX|Y (x|y) = P pX (x)
pY |X (y|x′ )pX (x′)
x′ ∈X

We know pX (x). To find pY |X , note that

pY |X (y|x) = P{Y = y|X = x}
= P{X ⊕ Z = y|X = x}
= P{Z = y ⊕ X|X = x}
= P{Z = y ⊕ x|X = x}
= P{Z = y ⊕ x}, since Z and X are independent
= pZ (y ⊕ x)

So we have
pY |X (0|0) = pZ (0 ⊕ 0) = pZ (0) = 1 − ǫ, pY |X (0|1) = pZ (0 ⊕ 1) = pZ (1) = ǫ
pY |X (1|0) = pZ (1 ⊕ 0) = pZ (1) = ǫ, pY |X (1|1) = pZ (1 ⊕ 1) = pZ (0) = 1 − ǫ

EE 178/278A: Multiple Random Variables Page 3 – 9

Plugging in the Bayes rule equation, we obtain

pY |X (0|0)
pX|Y (0|0) = pX (0)
pY |X (0|0)pX (0) + pY |X (0|1)pX (1)
(1 − ǫ)(1 − p)
=
(1 − ǫ)(1 − p) + ǫp
pY |X (0|1)
pX|Y (1|0) = pX (1) = 1 − pX|Y (0|0)
pY |X (0|0)pX (0) + pY |X (0|1)pX (1)
ǫp
=
(1 − ǫ)(1 − p) + ǫp
pY |X (1|0)
pX|Y (0|1) = pX (0)
pY |X (1|0)pX (0) + pY |X (1|1)pX (1)
ǫ(1 − p)
=
ǫ(1 − p) + (1 − ǫ)p
pY |X (1|1)
pX|Y (1|1) = pX (1) = 1 − pX|Y (0|1)
pY |X (1|0)pX (0) + pY |X (1|1)pX (1)
(1 − ǫ)p
=
ǫ(1 − p) + (1 − ǫ)p

EE 178/278A: Multiple Random Variables Page 3 – 10

2. We already found pY (y): pY (1) = ǫ(1 − p) + (1 − ǫ)p

3. Now to find the probability of error P{X 6= Y }, consider

P{X 6= Y } = pX,Y (0, 1) + pX,Y (1, 0)

= pY |X (1|0)pX (0) + pY |X (0|1)pX (1)
= ǫ(1 − p) + ǫp
=ǫ

An interesting special case is when ǫ = 1/2

Here, P{X 6= Y } = 1/2, which is the worst possible (no information is sent),
and
1 1 1
pY (0) = p + (1 − p) = = pY (1),
2 2 2
i.e., Y ∼ Bern(1/2), independent of the value of p !
Also in this case, the bit sent X and the bit received Y are independent (check
this)

EE 178/278A: Multiple Random Variables Page 3 – 11

Two Continuous Random variables – Joint PDFs

• Two continuous r.v.s defined over the same experiment are jointly continuous if
they take on a continuum of values each with probability 0. They are completely
specified by a joint pdf fX,Y such that for any event A ∈ (−∞, ∞)2 ,
Z
P{(X, Y ) ∈ A} = fX,Y (x, y) dx dy
(x,y)∈A

For example, for a rectangular area

Z dZ b
P{a < X ≤ b, c < Y ≤ d} = fX,Y (x, y) dx dy
c a
• Properties of a joint pdf fX,Y :
1. fX,Y (x, y) ≥ 0
R∞ R∞
2. −∞ −∞ fX,Y (x, y) dx dy = 1
• Again joint pdf is not a probability. Can relate it to probability as
P{x < X ≤ x + ∆x, y < Y ≤ y + ∆y} ≈ fX,Y (x, y) ∆x ∆y

EE 178/278A: Multiple Random Variables Page 3 – 12

 Marginal PDF

• The Marginal pdf of X can be obtained from the joint pdf by integrating the
joint over the other variable y
Z ∞
fX (x) = fX,Y (x, y) dy
−∞

This follows by the law of total probability. To see this, consider

x
x x + ∆x

EE 178/278A: Multiple Random Variables Page 3 – 13

P{x < X ≤ x + ∆x}

fX (x) = lim
∆x→0 ∆x
∞
1 X
= lim lim P{x < X ≤ x + ∆x, n∆y < Y ≤ (n + 1)∆y}
∆x→0 ∆x ∆y→0
n=−∞
Z ∞ Z ∞
1
= lim fX,Y (x, y) dy ∆x = fX,Y (x, y) dy
∆x→0 ∆x −∞ −∞

EE 178/278A: Multiple Random Variables Page 3 – 14

 Example

• Let (X, Y ) ∼ f (x, y), where


c if x, y ≥ 0, and x + y ≤ 1
f (x, y) =
0 otherwise

1. Find c
2. Find fY (y)
3. Find P{X ≥ 21 Y }

• Solution:
1. To find c, note that Z ∞ Z ∞
f (x, y) dx dy = 1,
−∞ −∞
thus 12 c = 1, or c = 2

EE 178/278A: Multiple Random Variables Page 3 – 15

2. To find fY (y), we use the law of total probability

Z ∞
fY (y) = f (x, y) dx
−∞
( R
(1−y)
0
2 dx 0 ≤ y ≤ 1
=
0 otherwise

2(1 − y) 0 ≤ y ≤ 1

=
0 otherwise
fY (y)
2

y
0 1

EE 178/278A: Multiple Random Variables Page 3 – 16

 3. To find the probability of the set {X ≥ 21 Y } we first sketch it
y
1 x = 12 y
2
3

x
1

from the figure we find that

  Z
1
P X≥ Y = fX,Y (x, y) dx dy
2 {(x,y): x≥ 21 y}
Z 32 Z (1−y)
2
= 2 dx dy =
0 y
2
3

EE 178/278A: Multiple Random Variables Page 3 – 17

Example: Buffon’s Needle Problem

• The plane is ruled with equidistant parallel lines at distance d apart. Throw
needle of length l < d at random. What is the probability that it will intersect
one of the lines?

d
Θ

X
l

• Solution:
Let X be the distance from the midpoint of the needle to the nearest of the
parallel lines, and Θ be the acute angle determined by a line through the needle
and the nearest parallel line

EE 178/278A: Multiple Random Variables Page 3 – 18

 (X, Θ) are uniformly distributed within the rectangle [0, d/2] × [0, π/2], thus

4
fX,Θ(x, θ) = , x ∈ [0, d/2], θ ∈ [0, π/2]
πd

The needle intersects a line iff X < 2l sin Θ

The probability of intersection is:
  Z Z
l
P X < sin Θ = fX,Θ(x, θ) dx dθ
2 {(x,θ): x< 2l sin θ}
Z π/2 Z 2l sin θ
4
= dx dθ
πd 0 0
Z π/2
4 l
= sin θ dθ
πd 0 2
2l
=
πd

EE 178/278A: Multiple Random Variables Page 3 – 19

Example: Darts

• Throw a dart on a disk of radius r. Probability on the coordinates (X, Y ) is

described by a uniform pdf on the disk:

1
if x2 + y 2 ≤ r 2

πr 2
,
fX,Y (x, y) =
0, otherwise

Find the marginal pdfs

• Solution: To find the pdf of Y (same as X ), consider

∞
1
Z Z
fY (y) = fX,Y (x, y) dx = 2 dx
−∞ πr {x: x2+y2≤r2}
Z √r2−y2  2 p
1 πr 2
r 2 − y 2 , if |y| ≤ r
= 2 √ dx =
πr − r2−y2 0, otherwise

EE 178/278A: Multiple Random Variables Page 3 – 20

 Conditional pdf

• Let X and Y be continuous random variables with joint pdf fX,Y (x, y), we
define the conditional pdf of Y given X as
fX,Y (x, y)
fY |X (y|x) = for fX (x) 6= 0
fX (x)

• Note that, for a fixed X = x, fY |X (y|x) is a legitimate pdf on Y – it is

nonnegative and integrates to 1
• We want the conditional pdf to be interpreted as:
fY |X (y|x)∆y ≈ P{y < Y ≤ y + ∆y|X = x}
The RHS can be interpreted as a limit
P{y < Y ≤ y + ∆y, x < X ≤ x + ∆x}
P{y < Y ≤ y + ∆y|X = x} = lim
∆x→0 P{x < X ≤ x + ∆x}
fX,Y (x, y) ∆x ∆y fX,Y (x, y)
≈ lim = ∆y
∆x→0 fX (x)∆x fX (x)

EE 178/278A: Multiple Random Variables Page 3 – 21

• Example: Let 
2, x, y ≥ 0, x + y ≤ 1
f (x, y) =
0, otherwise
Find fX|Y (x|y).

2(1 − y), 0 ≤ y ≤ 1
Solution: We already know that fY (y) =
0, otherwise
Therefore
1
1−y , x, y ≥ 0, x + y ≤ 1, y < 1

fX,Y (x, y)
fX|Y (x|y) = =
fY (y) 0, otherwise

fX|Y (x|y)
1
(1−y)

x
(1 − y)

EE 178/278A: Multiple Random Variables Page 3 – 22

• Chain rule: pX,Y (x, y) = pX (x)pY |X (y|x)

EE 178/278A: Multiple Random Variables Page 3 – 23

Independence and Bayes Rule

• Independence: Two continuous r.v.s are said to be independent if

fX,Y (x, y) = fX (x)fY (y) for all x, y
It can be shown that this definition is equivalent to saying that X and Y are
independent if for any two events A, B ⊂ (−∞, ∞)
P{X ∈ A, Y ∈ B} = P{X ∈ A}P{Y ∈ B}
• Example: Are X and Y in the previous example independent?
• Bayes rule for densities: Given fX (x) and fY |X (y|x), we can find
fY |X (y|x)
fX|Y (x|y) = fX (x)
fY (y)
fY |X (y|x)
= R∞ fX (x)
f
−∞ X,Y
(u, y)du
fY |X (y|x)
= R∞ fX (x)
−∞
fX (u)f Y |X (y|u) du

EE 178/278A: Multiple Random Variables Page 3 – 24

• Example: Let Λ ∼ U[0, 1], and the conditional pdf of X given Λ = λ
fX|Λ(x|λ) = λe−λx, 0 ≤ λ ≤ 1, i.e., X|{Λ = λ} ∼ Exp(λ). Now, given that
X = 3, find fΛ|X (λ|3)
Solution: We use Bayes rule
fX|Λ(3|λ)fΛ(λ)
fΛ|X (λ|3) = R 1
0 Λ
f (u)fX|Λ(3|u) du
λe−3λ
(
1 (1−4e−3 ) , 0 ≤ λ ≤ 1
= 9
0, otherwise
fΛ(λ)

λ
0 1
fΛ|X (λ|3)
1.378

0.56
1 λ
3 1

EE 178/278A: Multiple Random Variables Page 3 – 25

Joint cdf

• If X and Y are two r.v.s over the same experiment, they are completely
specified by their joint cdf

FX,Y (x, y) = P{X ≤ x, Y ≤ y} for x, y ∈ (−∞, ∞)

(x, y)

EE 178/278A: Multiple Random Variables Page 3 – 26

• Properties of the joint cdf:
1. FX,Y (x, y) ≥ 0
2. FX,Y (x1, y1) ≤ FX,Y (x2, y2) whenever x1 ≤ x2 and y1 ≤ y2
3. limx,y→∞ FX,Y (x, y) = 1
4. limy→∞ FX,Y (x, y) = FX (x) and limx→∞ FX,Y (x, y) = FY (y)
5. limy→−∞ FX,Y (x, y) = 0 and limx→−∞ F (x, y) = 0
6. The probability of any set can be determined from the joint cdf, for example,
y

c
x
a b
P{a < X ≤ b, c < Y ≤ d} = F (b, d) − F (a, d) − F (b, c) + F (a, c)

EE 178/278A: Multiple Random Variables Page 3 – 27

• If X and Y are continuous random variables having a joint pdf fX,Y (x, y), then
Z x Z y
FX,Y (x, y) = fX,Y (u, v) du dv for x, y ∈ (−∞, ∞)
−∞ −∞

Moreover, if FX,Y (x, y) is differentiable in both x and y, then

∂ 2F (x, y) P{x < X ≤ x + ∆x, y < Y ≤ y + ∆y}

fX,Y (x, y) = = lim
∂x∂y ∆x,∆y→0 ∆x∆y

• Two random variables are independent if

FX,Y (x, y) = FX (x)FY (y)

EE 178/278A: Multiple Random Variables Page 3 – 28

 Functions of Two Random Variables

• Let X and Y be two r.v.s with known pdf fX,Y (x, y) and Z = g(x, y) be a
function of X and Y . We wish to find fZ (z)
• We use the same procedure as before: First calculate the cdf of Z , then
differentiate it to find fZ (z)
• Example: Max and Min of Independent Random Variables
Let X ∼ fX (x) and Y ∼ fY (y) be independent, and define
U = max{X, Y }, and V = min{X, Y }
Find the pdfs of U and V
• Solution: To find the pdf of U , we first find its cdf
FU (u) = P{U ≤ u}
= P{X ≤ u, Y ≤ u}
= FX (u)FY (u), by independence

EE 178/278A: Multiple Random Variables Page 3 – 29

Now, to find the pdf, we take the derivative w.r.t. u

fU (u) = fX (u)FY (u) + fY (u)FX (u)
For example, if X and Y are uniformly distributed between 0 and 1,
fU (u) = 2u for 0 ≤ u ≤ 1
Next, to find the pdf of V , consider
FV (v) = P{V ≤ v}
= 1 − P{V > v}
= 1 − P{X > v, Y > v}
= 1 − (1 − FX (v))(1 − FY (v))
= FX (v) + FY (v) − FX (v)FY (v),

thus
fV (v) = fX (v) + fY (v) − fX (v)FY (v) − fY (v)FX (v)

For example, if X ∼ Exp(λ1) and Y ∼ Exp(λ2), then V ∼ Exp(λ1 + λ2)

EE 178/278A: Multiple Random Variables Page 3 – 30

 Sum of Independent Random Variables

• Let X and Y be independent r.v.s with known distributions. We wish to find

the distribution of their sum W = X + Y
• First assume X ∼ pX (x) and Y ∼ pY (y) are independent integer-valued r.v.s,
then for any integer w, the pmf of their sum
pW (w) = P{X + Y = w}
X
= P{X = x, Y = y}
{(x,y): x+y=w}
X
= P{X = x, Y = w − x}
x
X
= P{X = x}P{Y = w − x}, by independence
x
X
= pX (x)pY (w − x)
x

This is the discrete convolution of the two pmfs

EE 178/278A: Multiple Random Variables Page 3 – 31

For example, let X ∼ Poisson(λ1) and Y ∼ Poisson(λ2) be independent, then

the pmf of their sum
X∞
pW (w) = pX (x)pY (w − x)
x=−∞
w
X
= pX (x)pY (w − x) for w = 0, 1, . . .
x=0
w
X λx1 −λ1 λ2w−x −λ2
= e e
x=0
x! (w − x)!
w   x  w−x
(λ1 + λ2)w −(λ1+λ2) X w λ1 λ2
= e
w! x=0
x λ1 + λ2 λ1 + λ2
(λ1 + λ2)w −(λ1+λ2)
= e
w!
Thus W = X + Y ∼ Poisson(λ1 + λ2)
In general a Poisson r.v. with parameter λ can be written
P as the sum of any
number of independent Poisson(λi) r.v.s, so long as λi = λ. This property of
a distribution is called infinite divisibility

EE 178/278A: Multiple Random Variables Page 3 – 32

• Now, let’s assume that X ∼ fX (x), and Y ∼ fY (y) are independent continuous
r.v.s. We wish to find the pdf of their sum W = X + Y . To do so, first note
that
P{W ≤ w|X = x} = P{X + Y ≤ w|X = x}
= P{x + Y ≤ w|X = x}
= P{x + Y ≤ w}, by independence
= FY (w − x)

Thus
fW |X (w|x) = fY (w − x), a very useful result
Now, to find the pdf of W , consider
Z ∞
fW (w) = fW,X (w, x) dx
−∞
Z ∞ Z ∞
= fX (x)fW |X (w|x) dx = fX (x)fY (w − x) dx
−∞ −∞

This is the convolution of fX (x) and fY (y)

EE 178/278A: Multiple Random Variables Page 3 – 33

• Example: Assume that X ∼ U[0, 1] and Y ∼ U[0, 1] are independent r.v.s. Find
the pdf of their sum W = X + Y
Solution: To find the pdf of the sum, we convolve the two pdfs
Z ∞
fW (w) = fX (x)fY (w − x) dx
−∞
R
w
R0 dx, if 0 ≤ w ≤ 1


1
= w−1
dx, if 1 < w ≤ 2

0, otherwise


w, if 0 ≤ w ≤ 1


= 2 − w, if 1 < w ≤ 2

0, otherwise


• Example: If X ∼ N (µ1, σ12 ) and Y ∼ N (µ2 , σ22) are indepedent, then their sum
W ∼ N (µ1 + µ2, σ12 + σ22), i.e., Gaussian is also an infinitely divisible
distribution– any Gaussian r.v. can be written as the sum of any number of
independent Gaussians as long as their means sum to its mean and their
variances sum to its variance (will prove this using transforms later)

EE 178/278A: Multiple Random Variables Page 3 – 34

 One Discrete and One Continuous RVs

• Let Θ be a discrete random variable with pmf pΘ(θ)

• For each Θ = θ, such that pΘ(θ) 6= 0, let Y be a continuous random variable
with conditional pdf fY |Θ(y|θ)
• The conditional pmf of Θ given Y can be defined as a limit
P{Θ = θ, y < Y ≤ y + ∆y}
pΘ|Y (θ|y) = lim
∆y→0 P{y < Y ≤ y + ∆y}
pΘ(θ)fY |Θ(y|θ)∆y
= lim
∆y→0 fY (y)∆y
fY |Θ(y|θ)
= pΘ(θ)
fY (y)

• So we obtain yet another version of Bayes rule

fY |Θ(y|θ)
pΘ|Y (θ|y) = P ′ ′ Θ
p (θ)
θ ′ pΘ(θ )fY |Θ (y|θ )

EE 178/278A: Multiple Random Variables Page 3 – 35

Example: Additive Gaussian Noise Channel

• Consider the following communication channel model

Z ∼ N (0, N )

Θ Y

where the signal sent


+1, with probability p
Θ=
−1, with probability 1 − p,

the signal received (also called observation) Y = Θ + Z , and Θ and Z are

independent
Given Y = y is received (observed), find the a posteriori pmf of Θ, pΘ|Y (θ|y)

EE 178/278A: Multiple Random Variables Page 3 – 36

• Solution: We use Bayes rule
fY |Θ(y|θ)pΘ(θ)
pΘ|Y (θ|y) = P ′ ′
θ ′ pΘ(θ )fY |Θ (y|θ )

We know pΘ(θ). To find fY |Θ(y|θ), consider

P{Y ≤ y|Θ = 1} = P{Θ + Z ≤ y|Θ = 1}
= P{Z ≤ y − Θ|Θ = 1}
= P{Z ≤ y − 1|Θ = 1}
= P{Z ≤ y − 1}, by independence of Θ and Z

Therefore, Y |{Θ = +1} ∼ N (+1, N ). Also, Y |{Θ = −1} ∼ N (−1, N )

Thus
(y−1) 2
√ p e− 2N
2πN
pΘ|Y (1|y) = −(y−1) 2 −(y+1)2
(1−p)
√ p e 2N +√ e 2N
2πN 2πN
y
pe
= for − ∞ < y < ∞
pey + (1 − p)e−y

EE 178/278A: Multiple Random Variables Page 3 – 37

• Now, let p = 1/2. Suppose the receiver decides that the signal transmitted is 1
if Y > 0, otherwise he decides that it is a −1. What is the probability of
decision error?
• Solution: First we plot the conditional pdfs

fY |Θ(y| − 1) fY |Θ(y|1)

y
−1 1

This decision rule make sense, since you decide that the signal transmitted is 1 if
fY |Θ(y|1) > fY |Θ(y| − 1)
An error occurs if
◦ Θ = 1 is transmitted and Y ≤ 0, or
◦ Θ = −1 is transmitted and Y > 0

EE 178/278A: Multiple Random Variables Page 3 – 38

 But if Θ = 1, then Y ≤ 0 iff Z < −1, and if Θ = −1, then Y > 0 iff Z > 1
Thus the probability of error is

P{ error } = P{Θ = 1, Y ≤ 0 or Θ = −1, Y > 0}

= P{Θ = 1, Y ≤ 0} + P{Θ = −1, Y > 0}
= P{Θ = 1}P{Y ≤ 0|Θ = 1} + P{Θ = −1}P{Y > 0|Θ = −1}
1 1
= P{Z < −1} + P{Z > 1}
2 2
 
1
=Q √
N

EE 178/278A: Multiple Random Variables Page 3 – 39

Summary: Total Probability and Bayes Rule

• Law of total probability:
P
◦ events: P(B) = i P(Ai ∩ B), Ai s partition Ω
P
◦ pmf: pX (x) = y p(x, y)
R
◦ pdf: fX (x) = fX,Y (x, y) dy
P R
◦ mixed: fY (y) = θ pΘ(θ)fY |Θ(y|θ), pΘ(θ) = fY (y)pΘ|Y (θ|y)dy
• Bayes rule:
P(B|Aj )
◦ events: P(Aj |B) = P P(Aj ), Ai s partition Ω
i P(B|Ai )P(Ai )
pY |X (y|x)
◦ pmf: pX|Y (x|y) = P ′ ′ X p (x)
x Y |X (y|x )pX (x )
′ p
fY |X (y|x)
◦ pdf: fX|Y (x|y) = R f (x)
fX (x′ )fY |X (y|x′ ) dx′ X
fY |Θ (y|θ)
◦ mixed: pΘ|Y (θ|y) = P ′ ′ pΘ(θ),
θ ′ pΘ (θ )fY |Θ (y|θ )
pΘ|Y (θ|y)
fY |Θ(y|θ) = R fY (y)
fY (y ′ )pΘ|Y (θ|y ′)dy ′

EE 178/278A: Multiple Random Variables Page 3 – 40

 More Than Two RVs

• Let X1, X2, . . . , Xn be random variables (defined over the same experiment)
• If the r.v.s are discrete then they can be jointly specified by their joint pmf
pX1,X2,...,Xn (x1, x2, . . . , xn), for all (x1, x2, . . . , xn) ∈ X1 × X2×, . . . × Xn

• If the r.v.s are jointly continuous, then they can be specified by the joint pdf
fX1,X2,...,Xn (x1 , x2, . . . , xn), for all (x1, x2 , . . . , xn)

• Marginal pdf (or pmf) is the joint pdf (or pmf) for a subset of {X1, . . . , Xn};
e.g. for three r.v.s X1, X2, X3 , the marginals are fXi (xi) and fXi,Xj (xi, xj ) for
i 6= j
• The marginals can be obtained from the joint in the usual way, e.g. for the
n = 3 example
Z ∞
fX1,X2 (x1, x2) = fX1,X2,X3 (x1, x2, x3) dx3
−∞

EE 178/278A: Multiple Random Variables Page 3 – 41

• Conditional pmf or pdf can be defined in the usual way, e.g. the conditional pdf
of (Xk+1, Xk+2, . . . , Xn) given (X1, X2, . . . , Xk ) is

fX1,...,Xn (x1, . . . , xn)

fXk+1,...,Xn|X1,...,Xk (xk+1, . . . , xn|x1, . . . , xk ) =
fX1,...,Xk (x1, . . . , xk )

• Chain rule: We can write

fX1,...,Xn (x1, . . . , xn) = fX1 (x1)fX2|X1 (x2|x1)fX3|X1,X2 (x3|x1, x2) . . .

. . . fXn|X1,...,Xn−1 (xn|x1, . . . , xn−1 )

• In general X1, X2, . . . , Xn are completely specified by their joint cdf

FX1,...,Xn (x1, . . . , xn) = P{X1 ≤ x1, . . . , Xn ≤ xn}, for all (x1, . . . , xn)

EE 178/278A: Multiple Random Variables Page 3 – 42

 Independence and Conditional Independence

• Independence is defined in the usual way: X1, X2, . . . , Xn are said to be

independent iff
n
Y
fX1,X2,...,Xn (x1, x2, . . . , xn) = fXi (xi) for all (x1, x2, . . . , xn)
i=1

• Important special case, i.i.d. r.v.s: X1, X2, . . . , Xn are said to be independent
and identically distributed (i.i.d.) if they are independent and have the same
marginal, e.g. if we flip a coin n times independently we can generate
X1, X2, . . . , Xn i.i.d. Bern(p) r.v.s
• The r.v.s X1 and X3 are said to be conditionally independent given X2 iff
fX1,X3|X2 (x1, x3|x2) = fX1|X2 (x1|x2)fX3|X2 (x3|x2) for all (x1, x2, x3)

• Conditional independence does not necessarily imply or is implied by

independence, i.e., X1 and X3 independent given X2 does not necessarily mean
that X1 and X3 are independent (or vice versa)

EE 178/278A: Multiple Random Variables Page 3 – 43

• Example: Series Binary Symmetric Channels:

Z1 Z2

X2
X1 X3

Here X1 ∼ Bern(p), Z1 ∼ Bern(ǫ1), and Z2 ∼ Bern(ǫ2) are independent, and

X3 = X1 + Z1 + Z2 mod 2
◦ In general X1 and X3 are not independent
◦ X1 and X3 are conditionally independent given X2
◦ Also X1 and Z1 are independent, but not conditionally independent given X2
• Example Coin with Random Bias: Consider a coin with random bias P ∼ fP (p).
Flip it n times independently to generate the r.v.s X1, X2, . . . , Xn (Xi = 1 if
i-th flip is heads, 0 otherwise)
◦ The r.v.s X1, X2, . . . , Xn are not independent
◦ However, X1, X2, . . . , Xn are conditionally independent given P — in fact,
for any P = p, they are i.i.d. Bern(p)

EE 178/278A: Multiple Random Variables Page 3 – 44