UCSD ECE153 Handout #27

Prof. Young-Han Kim Tuesday, May 6, 2014

Solutions to Homework Set #5

(Prepared by TA Fatemeh Arbabjolfaei)

1. Neural net. Let Y = X + Z, where the signal X ∼ U[−1, 1] and noise Z ∼ N (0, 1) are
independent.

(a) Find the function g(y) that minimizes

MSE = E (sgn(X) − g(Y ))2 ,

 

where (
−1 x ≤ 0
sgn(x) =
+1 x > 0.

(b) Plot g(y) vs. y.

Solution: The minimum MSE is achieved when g(Y ) = E(sgn(X) | Y ). We have

Z ∞
g(y) = E(sgn(X) | Y = y) = sgn(x)fX|Y (x|y) dx .
−∞

To find the conditional pdf of X given Y , we use

(1
fY |X (y|x)fX (x) 2 −1 ≤ X ≤ 1
fX|Y (x|y) = , where fX (x) =
fY (y) 0 otherwise.

Since X and Z are independent,

fY |X (y|x) = fZ (y − x) ⇒ Y | {X = x} ∼ N (x, 1) .

To find fY (y) we integrate fY |X (y|x)fX (x) over x :

∞ 1 (y−x) 2
1
Z Z
fY (y) = fY |X (y|x)fX (x) dx = √ e− 2 dx
−∞ −1 2 2π
1 (x−y)2
Z ∞ (x−y) 2 ∞ (x−y) 2 
1 1 1 1 1
Z Z
= √ e− 2 dx = √ e− 2 dx − √ e− 2 dx
2 −1 2π 2 −1 2π 1 2π
= 21 (Q(−y − 1) − Q(−y + 1)) .

Combining the above results, we get

1
 (y−x)2
Z ∞ Z 1 √1 e− 2
2 2π
g(y) = sgn(x)fX|Y (x|y) dx = sgn(x) dx
−∞ −1 fY (y)
 Z 0 (x−y) 2 1 (x−y) 2 
1 1 1
Z
= − √ e− 2 dx + √ e− 2 dx
2fY (y) −1 2π 0 2π
−Q(−y − 1) + 2Q(−y) − Q(−y + 1)
= .
Q(−y − 1) − Q(−y + 1)

The plot is shown below. Note the sigmoidal shape corresponding to the common neural
network activation function.

1.5

0.5
g(y)

−0.5

−1

−1.5
−8 −6 −4 −2 0 2 4 6 8
y

2. Additive shot noise channel. Consider an additive noise channel Y = X + Z, where the signal
X ∼ N (0, 1), and the noise Z|{X = x} ∼ N (0, x2 ), i.e., the noise power of increases linearly
with the signal squared.

(a) Find E(Z 2 ).

(b) Find the best linear MSE estimate of X given Y .

Solution:

(a) Since Z|{X = x} ∼ N (0, x2 ),

E(Z|X) = 0
Var(Z|X = x) = E(Z 2 |X = x) − E(Z|X = x)2 = x2 .

Therefore,

E(Z 2 ) = E(E(Z 2 |X)) = E(X 2 ) = 1.

2
 (b) From the the best linear estimate formula,

Cov(X, Y )
X̂ = (Y − E(Y )) + E(X).
σY2

Here we have

E(X) = 0,
E(Y ) = E(X + Z) = E(X) + E(Z) = E(X) + E(E(Z|X)) = 0 + E(0) = 0,
E(XZ) = E(E(XZ|X)) = E(XE(Z|X)) = E(X · 0) = E(0) = 0,
σY 2 = E(Y 2 ) − (E(Y ))2 = E((X + Z)2 ) − 02
= E(X 2 ) + E(Z 2 ) + 2E(XZ)
= 1 + 1 + 2 × 0 − 0 = 2,
Cov(X, Y ) = E((X − E(X))(Y − E(Y ))) = E(XY )
= E(X(X + Z)) = E(X 2 ) + E(XZ) = 1 + 0 = 1.

Using all of the above, we get

1
X̂ = Y.
2

3. Estimation vs. detection. Let the signal

+1, with probability 21


X=
−1, with probability 21 ,

and the noise Z ∼ Unif[−2, 2] be independent random variables. Their sum Y = X + Z is

observed.

(a) Find the best MSE estimate of X given Y and its MSE.
(b) Now suppose we use a decoder to decide whether X = +1 or X = −1 so that the
probability of error is minimized. Find the optimal decoder and its probability of error.
Compare the optimal decoder’s MSE to the minimum MSE.

Solution:

(a) We can easily find the piecewise constant density of Y


1
 4 |y| ≤ 1


fY (y) = 81 1 < |y| ≤ 3


0 otherwise

3
 The conditional probabilities of X given Y are

0
 −3 ≤ y < −1
P{X = +1 | Y = y} = 1
−1 ≤ y ≤ +1
2
1 +1 < y ≤ +3


1
 −3 ≤ y < −1
P{X = −1 | Y = y} = 1
−1 ≤ y ≤ +1
2
0 +1 < y ≤ +3


Thus the best MSE estimate is


−1 −3 ≤ Y < −1

g(Y ) = E(X | Y ) = 0 −1 ≤ Y ≤ +1

+1 +1 < Y ≤ +3


The minimum mean square error is

EY (Var(X | Y )) = EY (E(X 2 | Y ) − (E(X | Y ))2 ) = E(1 − g(Y )2 )

Z ∞
2
= 1 − E(g(Y ) ) = 1 − g(y)2 fY (y) dy
−∞
Z −1 Z 1 Z +3 
1 1 1
=1− 1 · dy +
8 0 · dy +
4 1 · dy 8
−3 −1 +1
Z −1 Z 3
1 1 1 1 1
=1− 8 dy − 8 dy = 1 − 4 − 4 = 2 .
−3 1

(b) The optimal decoder is given by the MAP rule. The a posteriori pmf of X was found
in part (a). Thus the MAP rule reduces to

−1 −3 ≤ y ≤ −1


D(y) = ±1 −1 < y < +1


+1 +1 < y ≤ +3

Since either value can be chosen for D(y) in the center range of Y , a symmetrical decoder
is sufficient, i.e., (
−1 y < 0
D(y) =
+1 y ≥ 0
The probability of decoding error is

P{d(Y ) 6= X} = P{X = −1, Y ≥ 0} + P{X = 1, Y < 0}

= P{X = −1 | Y ≥ 0} P{Y ≥ 0} + P{X = 1 | Y < 0}P{Y < 0}
1 1 1 1 1
= 4 · 2 + 4 · 2 = 4 .

4
If we use the decoder (detector) as an estimator, its MSE is

E (d(Y ) − X)2 = 34 · 0 + 41 · 22 = 1 .

This MSE is twice that of the minimum mean square error estimator.

5
 Solutions to Additional Exercises

1. Orthogonality. Let X̂ be the minimum MSE estimate of X given Y .

(a) Show that for any function g(y), E((X − X̂)g(Y )) = 0, i.e., the error (X − X̂) and g(Y )
are orthogonal.
(b) Show that
Var(X) = E(Var(X|Y )) + Var(X̂).
Provide a geometric interpretation for this result.

Solution:

(a) We use iterated expectation and the fact that E(g(Y )|Y ) = g(Y ):
  h  i
E (X − X̂)g(Y ) = E E (X − X̂)g(Y )|Y
= E [E ((X − E(X|Y ))g(Y ))|Y )]
= E (g(Y )E ((X − E(X|Y ))|Y ))
= E (g(Y ) (E(X|Y ) − E(X|Y )))
= 0.

(b) First we write

E(Var(X|Y )) = E(X 2 ) − E((E(X|Y ))2 ),

and

Var(E(X|Y )) = E((E(X|Y ))2 ) − (E(E(X|Y )))2

= E((E(X|Y ))2 ) − (E(X))2 .

Adding the two terms completes the proof.

p
Interpretation: If we view
p p X, E(X|Y ), and X−E(X|Y ) as vectors with“norm” Var(X),
Var(E(X|Y )), and E(Var(X|Y )), respectively, then this result provides a “Pythago-
ras theorem”, where the signal, the error, and the estimate are the sides of a right triangle
(estimate and error being orthogonal).

2. Let X and Y be two random variables. Let Z = X + Y and let W = X − Y . Find the best
linear estimate of W given Z as a function of E(X), E(Y ), σX , σY , ρXY and Z.

Solution: By the theorem of MMSE linear estimate, we have

Cov(W, Z)
Ŵ = (Z − E(Z)) + E(W ).
σZ2

6
 Here we have
E(W ) = E(X) − E(Y ),
E(Z) = E(X) + E(Y ),
σZ2 = σX
2
+ σY2 + 2ρXY σX σY ,
Cov(W, Z) = E(W Z) − E(W )E(Z) = E((X − Y )(X + Y )) − (E(X) − E(Y ))(E(X) + E(Y ))
= E(X 2 ) − E(Y 2 ) − (E(X))2 + (E(Y ))2 = σX
2
− σY2 .
So the best linear estimate of W given Z is
σX2 − σ2
Y
Ŵ = 2 + σ 2 + 2ρ (Z − E(X) − E(Y )) + E(X) − E(Y ).
σX Y XY σX σY

3. Let X and Y be two random variables with joint pdf


x + y, 0 ≤ x ≤ 1, 0 ≤ y ≤ 1,
f (x, y) =
0, otherwise.
(a) Find the MMSE estimator of X given Y .
(b) Find the corresponding MSE.
(c) Find the pdf of Z = E(X|Y ).
(d) Find the linear MMSE estimator of X given Y .
(e) Find the corresponding MSE.

Solution:
(a) We first calculate the marginal pdf of Y , by a direct integration. For y < 0 or y > 1, we
integrate over 0. For 0 ≤ y ≤ 1,
Z 1
1
fY (y) = (x + y) dx = y + .
0 2
Note that the limits of the integration are derived from the definition of the joint pdf.
Thus,
y + 21 , 0 ≤ y ≤ 1

fY (y) =
0, otherwise.
Now we can calculate the conditional pdf
fXY (x, y) x+y
fX|Y (x|y) = = 1
fY (y) 2 +y
for 0 ≤ x, y ≤ 1. Therefore, for 0 ≤ y ≤ 1
1
+ y2
Z 1 1
x2 + yx
Z
3
E[X|Y = y] = xfX|Y (x|y) dx = dx = ,
0 0 y + 12 1
2 +y
hence
1
3 + Y2
E[X|Y ] = 1 .
2 +Y

7
(b) The MSE is given by
E(Var(X|Y )) = EX 2 − E((E(X|Y ))2 )
Z 1  Z 1 1 y !2 
3 + 2
 
2 1 1
= x x+ dx − 1 y+ dy
0 2 0 2 +y
2
 
5 1 ln(3) 1 ln(3)
= − + = −
12 3 144 12 144
≈ 0.0757 .
1
+ y2 1 1 5 2
(c) Since E[X|Y = y] = 3
1
+y
= 2 + 6(1+2y) , we have 9 ≤ E[X|Y = y] ≤ 3 for 0 ≤ y ≤ 1.
2
1/3+Y /2 5
From the part (a), we know Z = E[X|Y ] = 1/2+Y . Thus, 9 ≤ Z ≤ 23 .
We first find the cdf FZ (z) of Z and then differentiate it to get the pdf. Consider
n1 + Y o n 3z − 2 o
3 2
FZ (z) = P{Z ≤ z} = P 1 ≤ z = P{2 + 3Y ≤ 3z + 6zY } = P Y ≥
2 +Y
3 − 6z
n 3z − 2 o
=1−P Y < .
3 − 6z
3z−2 5
For 0 ≤ 3−6z ≤ 1, i.e., ≤ z ≤ 23 , we have
9
Z 0 0
1

Z
FZ (z) = 1 − fY (y)dx = 1 − y+ dx
3z−2 3z−2 2
3−6z 3−6z

By differentiating with respect to z, we get

3z − 2 1
d 3z − 2
−1 −1
fZ (z) = − + · =− ·
3 − 6z 2 dz 3 − 6z 6(1 − 2z) 3(1 − 2z)2
1
=
18(2z − 1)3
5
for 9 ≤ z ≤ 32 . Otherwise fZ (z) = 0.
(d) The best linear MMSE estimate is
Cov(X, Y )
X̂ = (Y − E(Y )) + E(X).
σY2
Here we have
1Z 1 1
1 7
Z Z
E(X) = x(x + y) dy dx = x(x + ) dy dx = ,
0 0 0 2 12
1
1 7
Z
E(Y ) = y(y + ) dy = ,
0 2 12
1Z 1
7 7 1
Z
Cov(X, Y ) = E(XY ) − E(X)E(Y ) = xy(x + y) dy dx − × =− ,
0 0 12 12 144
Z 1  2
1 7 11
σY2 = E(Y 2 ) − (E(Y ))2 = y 2 (y + ) dy − = .
0 2 12 144

8
 So the best linear MMSE estimate is
 
1 7 7
X̂ = − Y − + .
11 12 12

(e) The MSE of linear estimate is

2 Cov2 (X, Y )
MSE = σX − .
σY2
2 = σ2 = 11
Here by symmetry, σX Y 144 . Thus,

2 Cov2 (X, Y ) 11 1
MSE = σX − = + ≈ 0.167 .
σY2 144 11

We can check that LMSE > MSE.

4. Additive-noise channel with signal dependent noise. Consider the channel with correlated
signal X and noise Z and observation Y = 2X + Z, where
2
µX = 1 , µZ = 0 , σX = 4, σZ2 = 9 , ρX,Z = − 38 .

Find the best MSE linear estimate of X given Y .

Solution: The best linear MMSE estimate is given by the formula

Cov(X, Y )
X̂ = (Y − E(Y )) + E(X).
σY2

Here we have

E(Y ) = 2E(X) + E(Z) = 2 ,

σY2 = 4σX
2
+ σZ2 + 4ρXZ σX σZ = 16 ,
Cov(X, Y ) = E(XY ) − E(X)E(Y ) = E(2X 2 + XZ) − 1 × 2
2 23
= 2(σX + µ2X ) + (ρXZ σX σZ + µX µZ ) − 2 = .
4
So the best linear MMSE estimate is
23 23 9
X̂ = (Y − 2) + 1 = Y + .
64 64 32