CS 527 / ECE 599 Error Correcting Codes

Assignment No. 2, Due: Friday, January 31, 2025

Question1: A fair coin is tossed. What is the mutual information between

the TOP and the BOTTOM faces?
Solution: Let X be the top face and Y be the bottom face of a fair coin.
Since the coin is fair, the possible outcomes are:
1
P (X = H, Y = T ) = P (X = T, Y = H) =
2
Each face is equally likely to be heads or tails:
1 1
P (X = H) = P (X = T ) = , P (Y = H) = P (Y = T ) =
2 2
The mutual information is given by:
X P (x, y)
I(X; Y ) = P (x, y) log
x,y
P (x)P (y)

Substituting the values:

1 1
1 2 1 2
I(X; Y ) = log 1 1 + log 1 1
2 2 × 2
2 2 × 2

1 1 1 1
= log 2 + log 2 = (1) + (1) = 1 bit
2 2 2 2
Question 2:. Erasure Channel. Consider the discrete memoryless channel
shown in Figure 1. Assuming P (X = 0) = 23 , P (X = 1) = 13 , and p = 14 , find:
(a) H(X), H(Y )

(b) H(Y |X), H(X|Y )

(c) H(X, Y )
(d) I(X; Y )

Solution:
(a) Compute H(X) and H(Y )
 
X 2 2 1 1
H(X) = − P (X) log P (X) = − log + log
3 3 3 3

1
 For Y :
1 1 1
P (Y = 0) = , P (Y = 1) = , P (Y = e) =
2 4 4

 
X 1 1 1 1 1 1
H(Y ) = − P (Y ) log P (Y ) = − log + log + log
2 2 4 4 4 4

(b) Compute H(Y |X) and H(X|Y )

1 1 3 3
H(Y |X) = H(p) = −p log p − (1 − p) log(1 − p) = − log − log
4 4 4 4

1 1 1 1 1 1 1
H(X|Y ) = H(X|Y = 0)+ H(X|Y = 1)+ H(X|Y = e) = H(X|Y = e) = H( , )
2 4 4 4 4 2 2
(c) Compute H(X, Y )

H(X, Y ) = H(X) + H(Y |X)

(d) Compute I(X; Y )

I(X; Y ) = H(X) − H(X|Y ) = H(Y ) − H(Y |X)

Question 3: Can we construct a tree code with the following parameters?
If so, show the code tree; if not, explain why it is not possible:

(a) Binary code with lengths 2, 2, 2, 3, 3

(b) Ternary code with lengths 1, 2, 2, 3, 3, 3, 3

(c) Ternary code with lengths 1, 1, 2, 2, 2, 3

Solution:
To determine if we can construct a tree code with the given parameters,
we need to check if the Kraft inequality is satisfied for each case. The Kraft
inequality states that for a set of binary (or D-ary) code lengths, the sum of the
reciprocals of the D-ary powers of the code lengths should be less than or equal
to 1. The Kraft inequality is given by:
n
X
D−li ≤ 1
i=1

where li is the length of the i-th code, and D is the base of the code (2 for
binary, 3 for ternary). Let’s check each case:
(a) Binary code with lengths 2, 2, 2, 3, 3
The Kraft inequality is:

2
 1 1 1 1 1
+ 2 + 2 + 3 + 3 =1≤1
22 2 2 2 2
Since the inequality is satisfied, we can construct a binary code with these
lengths.
(b) Ternary code with lengths 1, 2, 2, 3, 3, 3, 3
The Kraft inequality is:
1 1 1 1 1 1 1 19
+ 2+ 2+ 3+ 3+ 3+ 3 = ≤1
31 3 3 3 3 3 3 27
Since the inequality is satisfied, we can construct a ternary code with these
lengths.
(c) Ternary code with lengths 1, 1, 2, 2, 2, 3
The Kraft inequality is:
1 1 1 1 1 1 28
1
+ 1+ 2+ 2+ 2+ 3 = >1
3 3 3 3 3 3 27
Since the inequality is not satisfied, we cannot construct a ternary code
with these lengths.
Question 4: Huffman coding. Consider the random variable X given by:

X = {X1 , X2 , X3 , X4 , X5 , X6 , X7 }

P (X) = {0.49, 0.26, 0.12, 0.04, 0.04, 0.03, 0.02}

(a) Find the binary Huffman code for X.

(b) Find the expected code length for this encoding.
(c) Find a 4-ary Huffman code for X and calculate the average length.

Solution:

Figure 1: Assignment 24

3
 Question 5:. A D-ary tree has N leaf nodes. If every internal node has D
children, prove that N mod (D − 1) = 1.
Solution: Let’s prove this statement by induction on the height of the D-ary
tree.
Base Case (h = 0)
A tree with a height of 0 has only one node, a leaf node. In this case, N = 1
and D = 1. Therefore, the equality holds.
Inductive Step
Assume that the statement is true for a tree of height h, where:

N mod (D − 1) = 1
We want to prove that it is true for a tree of height h + 1.
For height h, we have:

N = Dh , N mod (D − 1) = 1
which means:

Dh = Q · (D − 1) + 1
for some integer Q.
For height h + 1, we have:

N = D(h+1)
Substituting the previous equation:

D(h+1) = D · (Q · (D − 1) + 1) = Q · D · (D − 1) + D
Taking modulo (D − 1):

N mod (D − 1) = D mod (D − 1) = 1
Thus, by the principle of mathematical induction, the statement holds for
all heights h.
Question 6: Optimal codeword lengths. Although the codeword lengths
of an optimal variable length code are complicated functions of the message
probabilities {p1 , p2 , . . . , pm }, it can be said that less probable symbols are en-
coded into longer codewords. Suppose that the message probabilities are given
in decreasing order p1 ≥ p2 ≥ · · · ≥ pm .

(a) Prove that for any binary Huffman code, if the most probable message
symbol has probability p > 3/5, then that symbol must be assigned a
codeword of length 1.
(b) Prove that for any binary Huffman code, if the most probable message
symbol has probability p < 1/4, then that symbol must be assigned a
codeword of length ≥ 2.

4
 Solution:
(a) Proof by Contradiction
For binary Huffman codes, for p1 to have a code length of 1, it must be that p1
is the only minimum left together with the sum probability p2 +p3 +p4 +· · ·+pn .
In contradiction, if we can find a p1 > 53 such that it coexists with two other
probabilities py and psum , where py is the probability of a character Y and psum
is the sum of probabilities of all the other characters.
Since p1 is the largest probability in the list, we know that: py ≤ p1 . Fur-
thermore, for p1 to be encoded with length greater than 1, it must also be one of
the two minimum probabilities. Since the combination psum , p1 can only happen
if py is greater than both, it must be the case that:

py ≤ p1 < psum .
We also have:

p1 + py + psum = 1.
Since psum > p1 and p1 > 53 , it follows that:
3
psum > .
5
3
Thus, since both psum > 5 and p1 > 35 , we get:

p1 + py + psum > 1.
This contradicts the assumption that the sum of probabilities must be ex-
actly 1. By contradiction, p1 must have length 1.
(b) Another Contradiction Argument
Similarly, assume that another py exists such that the set {py , psum } is the
minimum so that p1 can have length 1. The inequalities are set up as follows:

p1 + py + psum = 1,

py , psum < p1 .
Since py < p1 and psum < p1 , we assume:
1
p1 < .
4
Thus, it follows that:
1 1
psum < , py < .
4 4
Adding them together:

p1 + py + psum < 1.

5
 This again contradicts the assumption that the sum of these probabilities
must be equal to 1. By contradiction, the codeword for p1 must have length
≥ 2.
7. Shannon code. Consider the following method for generating a code for
a random variable X which takes on m values {1, 2, . . . , m} with probabilities
p1 , p2 , . . . , pm . Assume that the probabilities are ordered so that p1 ≥ p2 ≥
· · · ≥ pm . Define
i−1
X
Fi = pk
k=1

(a) Construct the code for the probability distribution (0.5, 0.25, 0.125, 0.125).
(b) Show that the code constructed by this process is prefix free and the
average length satisfies:

H(X) ≤ L < H(X) + 1

Solution: (a)

Pi Fi li Code
0.5 0 1 0
0.25 0.5 2 10
0.125 0.75 3 110
0.125 0.875 3 111
(b) Entropy and Average Code Length
We know that:
n   n
X 1 X
H(X) = pi log = pi li
i=1
pi i=1

The average length of this code is:

n
X
L= pi li
i=1

Since the average length requires each code word to have an integer length,
li is always rounded up to the nearest integer ⌈li ⌉, such that ⌈li ⌉ ≥ li . Thus, we
get:

H(X) ≤ L
We also observe that:
   
1 1
⌈log ⌉ < log +1
pi pi
which implies:

6
 ⌈li ⌉ < li + 1
Summing over all probabilities:
n
X n
X
pi ⌈li ⌉ < pi (li + 1)
i=1 i=1
n
X n
X
L< pi li + pi
i=1 i=1
Pn
Since i=1 pi = 1, we obtain:
n
X
L< pi li + 1
i=1
Thus:

H(X) ≤ L < H(X) + 1

Consider the nature of Shannon coding based on negative powers of 2. For
two characters Xi and Xj (i < j) with probabilities pi and pj , before binary
conversion, their codes are given by:
i−1
X
Fi = pk
k=1
j−1
X
Fj = pk
k=1
Due to the nature of prefix codes, the longer code Fj cannot contain Fi .
That is, the first li characters of Fj must be different from Fi . This can be
written as the inequality:

Fj − Fi > 2−li
Expanding the left-hand side:

Fj − Fi = pi + pi+1 + · · · + pj−1 + pj
For the right-hand side:
 
1
− log2
2−li = 2 pi

= 2log2 (pi ) = pi
Thus, we have:

Fj − Fi > 2−li
which is always true. Therefore, Shannon coding is a prefix code.