Scribd
Upload
89 views2 pages

EECS 554 hw3

This document is the homework assignment for EECS 554. It contains 9 questions about data compression topics like prefix codes, entropy, Huffman coding, and typical sequences. Students are asked to prove properties, design codes for data sources, and derive bounds related to these compression concepts.

Uploaded by

Fengxing Zhu
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
89 views2 pages

EECS 554 hw3

This document is the homework assignment for EECS 554. It contains 9 questions about data compression topics like prefix codes, entropy, Huffman coding, and typical sequences. Students are asked to prove properties, design codes for data sources, and derive bounds related to these compression concepts.

Uploaded by

Fengxing Zhu
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 2

EECS 554

Homework 3
Due: 10/7

1 Compression
1.1 In the lecture we stated two equivalent conditions for a prefix free (PF) code to be “full”.
Condition 1: No additional codeword can be added without destroying the prefix-free prop-
erty
Condition 2: No codeword can be shortened without destroying the prefix-free property
Show that these two conditions are not equivalent for non-binary codes. (You can con-
sider ternary codes and come up with a counterexample)
1.2 In the proof of the converse of the Kraft inequality we constructed a prefix free (PF)
code by first sorting the given codeword lengths in increasing order l1 ≤ l2 ≤ · · · ≤ lN .
Is this necessary for the construction? In other words, do you see any problem if the
construction is done without the lengths being sorted?
1.3 Show that the entropy of a DMS is a non-negative number and is zero if and only if the
source is deterministic.
∑ ∑M
1.4 Let p1 , . . . , pM and q1 , . . . , qM be arbitrary positive numbers with Mi=1 pi = i=1 qi = 1.
Show that
∑ M
1 ∑ M
1
pi log2 ≤ pi log2
i=1
pi i=1
qi
with equality if and only if pi = qi for all i.
Hint: Use the inequality ln x ≤ x − 1 for x = qi /pi .
1.5 Using the above result, show that the entropy of a DMS with an M-ary alphabet is at
most log2 M .
1.6 In the derivation in class of the bounds for the optimal average length of prefix free
codes we used a necessary condition for optimality, without showing that it is also sufficient.
Using the inequality ln x ≤ x − 1, show that H ≤ L̄ for every prefix free code. Did you use
the fact that the code is full?
1.7 A source has an alphabet {a1 , a2 , a3 , a4 } with corresponding probabilities {0.1, 0.2, 0.3, 0.4}.
1. Design a Huffman code for the source and evaluate the average length of the code.
2. One can group two consecutive symbols into symbol pairs (or super-symbols) and design
a Huffman code for the source emitting such super-symbols. There are 16 such super-
symbols. Find the probability of each such super-symbols and design a Huffman code.
What is the average codeword length? What is the average required bits per each source
letter ? Compare with the previous answer (for a fair comparison, both lengths are
expressed in bits per single letter).

1
1.8 A source emits one of four messages randomly every 1 microsec with probabilities 0.5,
0.3, 0.1, and 0.1. Design a Huffman code for this source.

1. What is its average length? What is the average transmission rate?

2. Now suppose that we want to encode this source using not a binary (i.e., {0, 1}) but a
ternary (i.e., {0, 1, 2}) alphabet. Design a ternary Huffman code using the same ideas
as in binary codes, but with ternary trees instead of binary trees. What is the average
code length (in “trits/symbol”) and data rate (in “trits”/sec)?

1.9 Consider a DMS with finite alphabet A and probabilities p(a) for a ∈ A. For any
sequence x ∈ An and for any a ∈ A, let Nx (a) denote the average number of occurences of
the symbol a in x. Given ϵ > 0 and n, define the set of strongly typical sequences of length
n as
Tϵn = {x ∈ An : |Nx (a) − p(a)| < ϵ, ∀a ∈ A}.
def

Find a lower bound on the size |Tϵn | of this set that is asymptotically tight with the upper
bound derived in class.

You might also like