EE5143 Problem Set 1: Entropy and Source Coding

Instructor: Dr. Andrew Thangaraj

Jan - May 2024
Entropy

1. Entropy of functions. (EIT 2.2) Let X be a random variable taking on a finite number of
values. What is the (general) inequality relationship between H(X) and H(Y ) if

(a) Y = 2X ?
(b) Y = cos X?

2. Entropy of a disjoint mixture. (EIT 2.10) Let X1 and X2 be discrete random variables
drawn according to probability mass functions p1 (·) and p2 (·) over the respective alphabets
X1 = {1, 2, ..., m} and X2 = {m + 1, ..., n}. Let
(
X1 w.p. α
X=
X2 w.p. 1 − α

(a) Find H(X) in terms of H(X1 ), H(X2 ), and α.

(b) Maximize over α to show that 2H(X) ≤ 2H(X1 ) +2H(X2 ) and interpret using the notion
that 2H(X) is the effective alphabet size.

3. Entropy of a sum. (EIT 2.14) Let X and Y be random variables that take on values
x1 , x2 , . . . , xr and y1 , y2 , . . . , ys , respectievely. Let Z = X + Y .

(a) Show that H(Z|X) = H(Y |X). Argue that if X, Y are independent, then H(Y ) ≤
H(Z) and H(X) ≤ H(Z). Thus, the addition of independent random variables adds
uncertainty.
(b) Give an example of (necessarily dependent) random variables in which H(X) > H(Z)
and H(Y ) > H(Z).
(c) Under what conditions does H(Z) = H(X) + H(Y )?

4. Mixing increases entropy. Refer to EIT 2.28.

5. Infinite Entropy. (EIT 2.19) ThisPproblem shows that the entropy of a discrete random
variable can be infinite. Let A = ∞ 2 −1
n=2 (n log n) . It is easy to show that A is finite by
bounding the infinite sum by the integral of (x log2 x)−1 .

Now, show that the integer-values random variable X defined by P r(X = n) = (An log2 n)−1
for n = 2, 3, . . . has H(X) = +∞.

6. Joint Entropy. Let X and Y be random variables with joint pmf p(x, y) as shown in the
table below. Find H(X), H(Y ), H(Y |X), and H(X, Y ).

X; Y 0 1 2 3
0 1/16 1/8 1/32 1/32
1 1/8 1/16 1/32 1/32
2 1/16 1/16 1/16 1/16
3 1/4 0 0 0

1
 7. The value of a question. (EIT 2.38) Let X ∼ p(x), x = 1, 2, . . . , m. We are given a set
S ⊆ {1, 2. . . . , m}. We ask whether X ∈ S and receive the answer
(
1 if X ∈ S,
Y =
0 if X ∈/ S.

Suppose that P r{X ∈ S} = α. Find the decrease in uncertainty H(X) − H(X|Y ). Ap-
parently, any set S with a given α is as good as any other.

8. Entropy inequalities. For random variables X, Y, Z, show the following:

(a) H(X, Y ) + H(X, Y, Z) ≤ H(X) + 2H(Y ) + H(X, Z)

(b) H(X, Y ) + H(Y, Z) + H(X, Z) ≥ 2H(X, Y, Z)

9. Coin Flips. Refer to EIT 2.1

Optional Practice Problems.

2.4, 2.5, 2.7, 2.8, 2.13, 2.18, 2.33, 2.35, 2.36, 2.47 from EIT text book.

Source Coding

1. Huffman Coding.

(a) (EIT 5.5) Find the binary Huffman code for the source with probabilities 13 , 51 , 15 , 15 2 2


, 15 .
Argue that this code is also optimal for the source with probabilities 15 , 15 , 15 , 51 , 15 .

(b) (EIT 5.15) Construct a binary Huffman code for the following distribution on five
symbols: p = (0.3, 0.3, 0.2, 0.1, 0.1). What is the average length of this code? Next,
construct a probability distribution p′ on five symbols for which the code that you
constructed previously has an average length (under p′ ) equal to its entropy H(p′ ).
(c) (EIT 5.14) Find the ternary Huffman codes for the P
random variable X with proba-
1 2 3 4 5 6


bilities p = 21 , 21 , 21 , 21 , 21 , 21 and calculate L = pi li .
1 1 1
(d) (EIT 5.44*) Find the word lengths of the optimal binary encoding of p = ( 100 , 100 , . . . 100 ).
(e) Now, consider a black box containing one of two possible sources A or B. Source
1/4 1/4 1/4 1/4
A emits symbols with the distribution PA = { 0 , 1 , 2 , 3 }, while Source B has
1/2 1/4 1/8 1/8
distribution PB = { 0 , 1 , 2 , 3 }.
i. Design binary Huffman codes CA and CB for Sources A and B, respectively.
What are the average lengths?
ii. A prefix binary source code C is designed to encode the symbols {0, 1, 2, 3}
coming out of the black box when the source inside it is unknown.
A. Suppose C is chosen to be CA when Source B is actually in the black box.
What is the average length? Denote this as LPB (CA ).
B. Suppose C is chosen to be CB when Source A is actually in the black box.
What is the average length? Denote this as LPA (CB ).
C. Can you construct a binary, prefix code C such that L(C) = 21 LPA (C) +
1
2 LPB (C) is minimized? What is the minimum L(C)?

2
2. Codes. (EIT 5.37) For the codes C1 , C2 , C3 , C4 shown below, indicate which of the following
are,

C1 = {00, 01, 0}
C2 = {00, 01, 100, 101, 11}
C3 = {0, 10, 110, 1110, . . .}
C4 = {0, 00, 000, 0000}

(a) Uniquely decodable?

(b) Instantaneous?

3. Uniquely decodable codes. Determine which of the following codes are uniquely decodable:

(a) {0, 10, 11}

(b) {0, 01, 10}
(c) {00, 01, 10, 11}
(d) {110, 10, 11, 00, 100}

4. Huffman vs. Shannon. (EIT 5.33) A random variable X takes on three values with
probabilities 0.6, 0.3 and 0.1.

(a) What are the lengths of the binaryHuffman codewords

 for X? What are the lengths
1
of the binary Shannon codewords l(x) = ⌈log p(x) ⌉ for X?
(b) What is the smallest integer D such that the expected Shannon codeword length
with a D−ary alphabet equals the expected Huffman codeword length with a D−ary
alphabet?

5. Bad wine. (EIT 5.32) One is given 6 bottles of wine. It is known that precisely one bottle
has gone bad (tastes terrible). From inspection of the bottles it is determined that the
7 6 4 4 3 3
probability pi that the ith bottle is bad is given by (p1 , p2 , . . . p6 ) = ( 26 , 26 , 26 , 26 , 26 , 26 ).
Tasting will determine the bad wine.
Suppose you taste the wines one at a time. Choose the order of tasting to minimize the
expected number of tastings required to determine the bad bottle. Remember, if the first
5 wines pass the test you don’t have to taste the last.

(a) What is the expected number of tastings required?

(b) Which bottle should be tasted first?

Now you get smart. For the first sample, you mix some of the wines in a fresh glass and
sample the mixture. You proceed, mixing and tasting, stopping when the bad bottle has
been determined.

(a) What is the minimum expected number of tastings required to determine the bad
wine?
(b) What mixture should be tasted first?

3
 6. Lossless compression. Consider a sequence {X1 , X2 , . . . XN } of i.i.d. Bernoulli random
variables where Xi ∼ Ber(0.5). This sequence of random variables can be compressed and
represented as a sequence of ‘repeats’ or ‘run-lengths’. For example, 0011100000 . . . has
the repeats ‘00’, ‘111’ and ‘00000’. Let us assume that the P5sequence has 5 repeats and
denote the k-th repeat length by Rk . It follows that N = k=1 Rk .
In particular, the sequence {X1 , X2 , . . . XN } can be compressed as {X1 , R1 , R2 , . . . , R5 }.
Here are some examples,

• (100100 . . .) 7→ (1, 1, 2, 1, 2, . . .)
• (111001011 . . .) 7→ (1, 3, 2, 1, 1, 2, . . .)
• (0011110111 . . .) 7→ (0, 2, 4, 1, 3, . . .)

(a) Calculate the entropy of the first repeat R1 .

(b) Calculate the joint entropy of (X1 , R1 , R2 , . . . , R5 ).
(c) Design an optimal prefix code for R1 and calculate the expected code length.

7. The Games.

(a) Lower bound on number of questions. Arjun plays a game with Aryan. Arjun chooses
some tool from a Mechanical tool kit, and Aryan attempts to identify the tool Arjun
picked with a series of yes-no questions. Suppose Aryan is clever enough to use the
code achieving the minimal expected length with respect to Arjun’s distribution. We
observe that Aryan requires an average of 9 questions to determine the tool Arjun
has picked. Find a rough lower bound to the number of tools in the tool kit.
(b) The game of Hi-Lo (EIT 5.19)
i. A computer generates a random number N according to a known pmf p(n), n ∈
{1, 2, . . . , 100}. Seetha asks a yes-no question, “Is N = n?” and is told yes, you
are too high (or you are too low). She continues for a total of six questions. If
she is right (i.e., she receives the answer yes) during the six question sequence,
she receives a prize of value v(N ). How should she proceed to maximize her
expected winnings?
ii. Consider the following variation: N ∼ p(n), prize = v(n), p(n) known, as before.
But arbitrary yes-no questions are asked sequentially until N is determined.
Questions cost 1 unit each. How should Seetha proceed? What is the expected
payoff? Note that “Determined” doesn’t mean that a yes answer is received.
iii. Continuing the previous part, what if v(n) is fixed but p(n) can be chosen by the
computer (and then announced to Seetha)? The computer wishes to minimize
the Seetha’s expected return. What should p(n) be? What is the expected
return to the Seetha?

Optional Practice Problems. 5.6, 5.8, 5.10, 5.17, 5.27, 5.40 From EIT text book.

Optional Challenging Problems:

1. Axiomatic definition of entropy. (EIT 2.46) A function f (x1 , x2 , ..., xm ) is said to be

symmetric if it produces the same value irrespective of the order of its arguments. Now,
let us assume a sequence of symmetric functions Hm (p1 , p2 , ..., pm ) satisfies the following
properties:

4
 • Normalization: H2 ( 12 , 21 ) = 1,
• Continuity: H2 (p, 1 − p) is a continuous function of p,
• Grouping: Hm (p1 , p2 , ..., pm ) = Hm−1 (p1 + p2 , p3 , ..., pm ) + (p1 + p2 )H2 ( p1p+p
1
, p2 )
2 p1 +p2
Then show that Hm must be of the form:
m
X
Hm (p1 , p2 , ..., pm ) = − pi log(pi ) m = 2, 3, ...
i=1

2. Combinatorial meaning of entropy, Yury Polyanskiy. Fix n ≥ 1 and 0 ≤ k ≤ n. Let p = nk

and define Tp ⊂ {0, 1}n to be the set of all binary sequences with p fraction of ones. Show
that if k ∈ [1, n − 1] then
s
1
|Tp | = exp{nh(p)}C(n, k)
np(1 − p)

where C(n, k) is bounded by two universal constants C0 ≤ C(n, k) ≤ C1 , and h(·) is the
binary entropy. Conclude that for all 0 ≤ k ≤ n we have

log |Tp | = nh(p) + O(log n).

Hint: Use Stirling’s approximation:

1 n! 1
e 12n+1 ≤ √ ≤ e 12n , n≥1
2πn(n/e) n