ECE 154C Homework Assignment #2

(Due: Wednesday, May 8, 2019)

Reading assignment: Jack Wolf lecture notes pp. 40-68 and slides 84-147 from spring 2014

Recommended: Solve HW #3 from spring 2017 for practice (appended at the end)

Programming assignment: You can use any programming language you prefer (MATLAB,
Python, C/C++, or Julia, for example). Write down your code as clearly as possible and add
suitable comments. You should write your program from scratch. However, it is okay to use
some common functions such as sort or random generators in your code. Submit your code and
the output plots to [email protected] or turn in the hard copies during the lecture on
Monday morning.

Let 𝑋~𝑁(0,1) denote a Gaussian random variable with mean 0 and standard deviation 𝜎 = 1.
Therefore, its PDF is given by

1 𝑥 2⁄
𝑓𝑋 (𝑥) = 𝑒− 2.
√2𝜋

a) Let 𝑁 = 2,4,8, … , 210 denote the number of quantized intervals. For each value of 𝑁,
define the quantization regions by dividing the interval [𝑋𝑚𝑖𝑛 , 𝑋𝑚𝑎𝑥 ] = [−5,5] into 𝑁
10
equal sub-intervals of length ∆= , denoted by 𝑅𝑖 for 𝑖 = 1,2, … , 𝑁. Assume that if 𝑋
𝑁
falls in the region 𝑅𝑖 , the D/A converter estimates the value of 𝑋 by 𝑋̂ = 𝑌, which is the
center of 𝑖 th region. Define the approximated mean-squared error between 𝑋 and 𝑋̂ by
𝑋𝑚𝑎𝑥
2 (𝑥 − 𝑥̂)2 𝑓𝑋 (𝑥)𝑑𝑥.
𝜀 =∫
𝑋𝑚𝑖𝑛
Plot the values of 𝜀 ∆ for 𝑁 = 2,4,8, … , 210 . Does it converge? Please use log-scale on
2⁄ 2

the axis that corresponds to 𝜀 2 ⁄∆2 .

b) Using the Lloyd-Max algorithm derived in class, find the optimal quantization regions
and representation points for 𝑁 = 2,4,8, … , 210 bit quantization for 𝑋~𝑁(0,1)
with [𝑋𝑚𝑖𝑛 , 𝑋𝑚𝑎𝑥 ] = [−5,5]. You can start from the equal intervals derived in the
previous part. For the convergence check, use the following condition
𝐷𝑛 −𝐷𝑛−1
≤ 10−3 ,
𝐷𝑛
where 𝐷𝑛 denotes the MSE after 𝑛 iterations in the algorithm. Plot the values of 𝑁 2 𝜀 2
for 𝑁 = 2,4,8, … , 210 . Does it converge? Can you provide an explanation? Please use
log-scale on the axis that corresponds to 𝑁 2 𝜀 2 .
UCSD ECE154C Handout #8
Prof. Young-Han Kim Thursday, April 27, 2017

Homework Set #3
(Due: Wednesday, May 3, 2017)

1. Consider any binary tree with leaves at depth l1 , . . . , lM . Show that

X
M
2−li ≤ 1.
i=1

2. Consider an i.i.d. source with alphabet {A, B, C, D} and probabilities {0.1, 0.2, 0.3, 0.4}.

(a) Find a Tunstall code that encodes the source phrases into binary codewords of
length 4. Compute the average number of binary code symbols per source symbol
and compare it to the entropy (with the appropriate base).
(b) Now suppose the Tunstall code you found in part (a) is followed by a binary
Huffman code. Compute the average number of binary code symbols per source
symbol.
(c) Find a Tunstall code that encodes the source phrases into quaternary code words
of length 2. Compute the average number of quaternary code symbols per source
symbol and compare it to the entropy (with the appropriate base).
(d) Suppose the Tunstall code found in part (c) is followed by a quaternary Huffman
code. Compute the average number of quaternary code symbols per source symbol
and compare it to the entropy (with the appropriate base).

3. Using the Lloyd–Max algorithm derived in class, find the optimal quantization regions
and representation points for 1 and 2 bit quantization for a random variable with
probability density function
1
fX (x) = e−|x| or all x.
2
For each case, calculate the average mean squared error between the unquantized and
quantized values.

1
UCSD ECE154C Handout #11
Prof. Young-Han Kim Wednesday, May 3, 2017

Solutions to Homework Set #3

1. Consider any binary tree with leaves at depth l1 , . . . , lM . Show that

M
X
2−li ≤ 1.
i=1

Solution: First note that for any binary tree with leaves at depth l1 , . . . , lM , there is a
corresponding binary prefix code with codeword length l1 , . . . , lM . We briefly prove that for
any binary prefix code, the codeword lengths l1 , . . . , lM must satisfy the inequality
M
X
2−li ≤ 1. (1)
i=1

Define the length of the longest codeword as lmax = maxi li . Now consider all nodes of the
tree at depth lmax . A codeword at depth lmax has 2lmax −li leaves. Since the total number of
leaves at depth lmax is 2lmax . We must have
M
X
2lmax −li ≤ 2lmax .
i=1

Therefore, (1) is satisfied.

2. Consider an i.i.d. source with alphabet {A, B, C, D} and probabilities {0.1, 0.2, 0.3, 0.4}.

(a) Find a Tunstall code that encodes the source phrases into binary codewords of length 4.
Compute the average number of binary code symbols per source symbol and compare it
to the entropy (with the appropriate base).
(b) Now suppose the Tunstall code you found in part (a) is followed by a binary Huffman
code. Compute the average number of binary code symbols per source symbol.
(c) Find a Tunstall code that encodes the source phrases into quaternary code words of
length 2. Compute the average number of quaternary code symbols per source symbol
and compare it to the entropy (with the appropriate base).
(d) Suppose the Tunstall code found in part (c) is followed by a quaternary Huffman code.
Compute the average number of quaternary code symbols per source symbol and compare
it to the entropy (with the appropriate base).

1
 Solution:

(a) Refer to scanned pages for procedure. The codeword mappings can be given as the
following.

A → 0000 BA → 0001 BB → 0010 BD → 0011

BC → 0100 DA → 0101 DB → 0110 DC → 0111
CA → 1000 CB → 1001 CD → 1010 CC → 1011
DDA → 1100 DDB → 1101 DDD → 1110 DDC → 1111

(b) In order to encode the Tunstall code given in the previous problem into a binary Huffman
code, we sort the Tunstall codewords with respect to their probabilities and perform
Huffman coding on those probabilities. Continued on scanned pages.
(c) The total number of codewords needed for a quaternary code of length 2 is M = 42 = 16.
Hence we can use the results found in part (a) with different codeword mappings. The
codeword mappings can be given as the following.

A → 00 BA → 01 BB → 02 BD → 03
BC → 10 DA → 11 DB → 12 DC → 13
CA → 20 CB → 21 CD → 22 CC → 23
DDA → 30 DDB → 31 DDD → 32 DDC → 33

Continued on scanned pages.

(d) In order to encode the Tunstall code given in the previous problem into a binary Huffman
code, we sort the Tunstall codewords with respect to their probabilities and perform
Huffman coding on those probabilities. Continued on scanned pages.

3. Using the Lloyd–Max algorithm derived in class, find the optimal quantization regions and
representation points for 1 and 2 bit quantization for a random variable with probability
density function
1
fX (x) = e−|x| or all x.
2
For each case, calculate the average mean squared error between the unquantized and quan-
tized values.

Solution: We need the following integrations for this problem.

Z Z
−x −x
xe dx = −(x + 1)e + C, x2 e−x dx = −(x2 + 2x + 2)e−x + C.

For 1 bit quantization, by symmetry the optimal boundary should be located at 0, and the
quantization points should be {−a, a} for some a > 0. Now, using the optimality condition,
a should be the centroid of the region [0, ∞), that is,
R∞ R ∞ −x
0 xfX (x)dx xe dx
a = R∞ = R0 ∞ −x = 1.
0 fX (x)dx 0 e dx

2
Hence, the mean squared error for this quantizer is
Z ∞
1
2
E[(X − X̂) ] = 2 (x − 1)2 e−x dx = 2 − 2 + 1 = 1.
0 2

For 2 bit quantization, again by symmetry the optimal boundary should be located at
{−b, 0, b} for some b > 0, and the quantization points should be {−a2 , −a1 , a1 , a2 } for some
a1 , a2 > 0. From the optimality condition, the quantization point should be the centroid of
the each region.
Rb
xfX (x)dx 1 − (b + 1)e−b
a1 = R0 b = ,
1 − e−b
0 fX (x)dx
R∞
xfX (x)dx
a2 = Rb ∞ = b + 1.
b fX (x)dx

Also, by the Voronoi condition, we require that

a1 + a2
= b.
2
Putting these together, we would get

1 − (b + 1)e−b
2b = a1 + a2 = + b + 1,
1 − e−b
or equivalently
b = 2(1 − e−b ).
This means that the optimal b needs to satisfy the equation. However, this is not analytically
solvable, so we need to do the Lloyd-Max algorithm to find the optimal points or solve the
equation numerically. If you run the Lloyd-Max algorithm with your computer, then you
would get
b ≈ 1.5936, a1 ≈ 0.5936, a2 ≈ 2.5936.
Accordingly, one can find the mean squared error as follows:
Z b Z ∞ 
2 2 1 −x 2 1 −x
E[(X − X̂) ] = 2 (x − a1 ) e dx + (x − a1 ) e dx
0 2 b 2
Z b Z ∞
= (x2 − 2a1 x + a21 )e−x dx + (x2 − 2a2 x + a22 )e−x dx
0 b
Z ∞ Z b Z ∞ Z b Z ∞
2 −x −x −x −x
= x e dx − 2a1 xe dx − 2a2 xe dx + a1 2 2
e dx + a2 e−x dx
0 0 b 0 b
−b −b
= 2 − 2a1 (1 − (b + 1)e ) − 2a2 (b + 1)e + a21 (1 − e−b ) + a22 e−b
= 0.3524

3
4
5
6