Solutions Manual

Introduction to Data Compression

Third Edition
Z. Li and K. Sayood
p
r
a
v
e
n
d
r
a
Chapter 2 1
Chapter 2
Problem 1.
H(X) =
M

i=1
P(x
i
) log
2
1
P(x
i
)

M

i=1
P(x
i
) log
2
1
1
= 0
H(X) log
2
M =
M

i=1
P(x
i
) log
2
1
P(x
i
)

M

i=1
P(x
i
) log
2
M
=
M

i=1
P(x
i
)(log
2
1
P(x
i
)
log
2
M)
=
M

i=1
P(x
i
) log
2
1
MP(x
i
)
= E[log
2
1
MP(X)
]
Now we use Jensens inequality which states that if g(X) is a convex

function E[g(X)]
g(E[X]).
E[log
2
1
MP(x
i
)
] log
2
E[
1
MP(X)
]
= log
2
(
M

i=1
P(x
i
)
1
MP(x
i
)
)
= 0
or
H(X) log
2
M
Problem 2.
Without loss of generality we can assume that the source alphabet is 1, 2, , M. Let H
1
represent the rst order entropy and let H be the entropy of the source. Dene
G
K
=
M

i
1
=1

M

i
K
=1
P(X
1
= i
1
, X
2
= i
2
X
K
= i
K
) log
2
P(X
1
= i
1
, X
2
= i
2
X
K
= i
K
)
then by denition
H = lim
K
1
K
G
K
If the X
i
are independent we can write
P(X
1
= i
1
, X
2
= i
2
X
K
= i
K
) = P(X
1
= i
1
)P(X
2
= i
2
) P(X
K
= i
K
)
p
r
a
v
e
n
d
r
a
Chapter 2 2
and
G
K
=
M

i
1
=1

M

i
K
=1
P(X
1
= i
1
)P(X
2
= i
2
) P(X
K
= i
K
)[log
2
P(X
1
= i
1
)
+log
2
P(X
2
= i
2
) + log
2
P(X
K
= i
K
)]
=
M

i
2
=1
P(X
2
= i
2
)
M

i
K
=1
P(X
K
= i
K
)
M

i
1
=1
P(X
1
= i
1
) log
2
P(X
1
= i
1
)

i
1
=1
P(X
1
= i
1
)
M

i
3
=1
P(X
3
= i
3
)
M

i
K
=1
P(X
K
= i
K
)
M

i
2
=1
P(X
2
= i
2
) log
2
P(X
2
= i
2
)
.
.
.

i
1
=1
P(X
1
= i
1
)
M

i
K1
=1
P(X
K1
= i
K1
)
M

i
K
=1
P(X
K
= i
K
) log
2
P(X
K
= i
K
)
The rst summations in each term sum to 1 and the last summation (assuming identical
distributions) are equal to H
1
, and
G
K
= KH
1
Therefore,
H = lim
K
1
K
KH
1
= H
1
Problem 3.
a) 2 bits.
b) 1.75 bits.
c) 1.739818 bits.
Problem 4.
H
Q
H
P
=
m

i=1
q
i
log
2
q
i
+
m

i=1
p
i
log
2
p
i
= q
j1
log
2
q
j1
q
j
log
2
q
j
+p
j1
log
2
p
j1
+p
j
log
2
p
j
Given a function
f
a
(x) = x log x (a x) log(a x)
we can easily show that f
a
(x) is maximum for x =
a
2
Let
p
j1
+p
j
= c
p
r
a
v
e
n
d
r
a
Chapter 2 3
then
q
j1
= q
j
=
c
2
Then
H
Q
H
P
=
c
2
log
2
c
2

c
2
log
2
c
2
+p
j
log
2
p
j
+ (c p
j
) log
2
(c p
j
) (1)
= f
c
(
c
2
) f
c
(p
j
)
0
Therefore H
Q
H
P
.
Problem 7.
a) Start with the list of codewords
0, 01, 11, 111
0 is a prex for 01 generating a dangling sux of 1. 11 is a prex to 111 also generating
a dangling sux of 1. Augment the codeword list with the dangling sux.
0, 01, 11, 111, 1
Now 1 is a prex to 111 generating a dangling sux of 11. As 11 is a codeword the
code is not uniquely decodeable.
b) Start with the list of codewords
0, 01, 110, 111
0 is a prex for 01 generating a dangling sux of 1. Augment the codeword list with
the dangling sux.
0, 01, 110, 111, 1
1 is a prex of 110 and 111 generating the dangling suxes 10 and 11. As neither is
a codeword we augment the list and continue.
0, 01, 110, 111, 1, 10, 11
At this point we can get no new dangling suxes therefore this code is uniquely
decodeable.
c) Start with the list of codewords
0, 10, 110, 111
No codeword is a prex of any other codeword. Therefore, this code is uniquely
decodeable.
d) Start with the list of codewords
1, 10, 110, 111
1 is a prex of 110 which generates a dangling sux of 10 which is a codeword.
Therefore this code is not uniquely decodeable.
p
r
a
v
e
n
d
r
a
Chapter 3 4
Chapter 3
Problem 4.
a)
H =
5

i=1
P(a
i
) log P(a
i
) = 1.817684bits
b) If we sort the probabilities in descending order we can see that the two letters with
the lowest probabilities are a
2
and a
4
. These will become the leaves on the lowest level
of the binary tree. The parent node of these leaves will have a probability of 0.9. If
we consider parent node as a letter in a reduced alphabet the it will be one of the two
letters with the lowest probability: the other one being a
1
. Continuing in this manner
we get the binary tree shown in Figure 1. and the code is
a a a a a
1 2 3 4 5
(0.04)
(0.05) 0.15) (0.26) (0.50)
(0.09)
(0.24)
(0.5)
1
1
1 0
0
0
Figure 1: Human code for the ve letter alphabet.
a
1
110
a
2
1111
a
3
10
a
4
1110
a
5
0
c)

l = 0.15 3 + 0.04 4 + 0.26 2 + 0.05 4 + 0.5 1 = 1.83bits/symbol

Problem 5.
p
r
a
v
e
n
d
r
a
Chapter 3 5
a
a
a
a
4
2
3
1
(.35)
(.30)
(.25)
(.10)
(.35)
(.65)
(1.0)
Figure 2: Human code for the four letter alphabet in Problem 5.
a) The Human code tree is shown in Figure 2. The code is
a
1
011
a
2
01
a
3
010
a
4
1
The average length of the code is 0.13+0.32+0.253+0.351 = 2 bits/symbol.
b) The Human code tree is shown in Figure 3. The code is
a
a
a
a
4
3
(.35)
(.30)
(.25)
(.10)
(.35)
(.65)
(1.0)
1
2
Figure 3: Minimum variance Human code for the four letter alphabet in Problem 5.
a
1
01
a
2
11
a
3
00
a
4
10
The average length of the code is obviously 2 bits/symbol.
p
r
a
v
e
n
d
r
a
Chapter 3 6
While the average length of the codeword is the same for both codes, that is they are both
equally ecient in terms of rate. However, the second code has a variance of zero for the
codelengths. This means that we would not have any problems with buer control if we
were using this code in a communication system. We cannot make the same assertion about
the rst code.
Problem 6. Examining the Human code generated in Problem 4 (not 3!) along with the
associated probabilities, we have
a
1
110 0.15
a
2
1111 0.04
a
3
10 0.26
a
4
1110 0.05
a
5
0 0.50
The proportion of zeros in a given sequence can be obtained by rst computing the proba-
bility of observing a zero in a codeword

5
k=1
P(0[a
k
)P(a
k
) and then dividing that by the
average length of a codeword. The probability of observing a zero in a codeword is
1 0.15 + 0 0.04 + 1 0.26 + 1 0.05 + 1 0.50 = 0.96
0.96/1.83 = 0.52. Thus the proportion of zeros is close to a half. If we examine Human
codes for sources with dyadic probabilities we would nd that the proportion is exactly a
half. Thus the use of a Human code will not lead to inecient channel usage.
p
r
a
v
e
n
d
r
a
Chapter 3 7
Problem 8.
3
5
2
1
2
1
1
49
51
50
47
48
45
43 44
NYT
v
d
0 1
r
a
[aardv]
4
7
2
1
3
2
1
49
51
50
47
48
45
43 44
NYT
v
0 1
r
a
[aardvar]
3
6
2
1
3
1
1
49
51
50
47
48
45
43 44
NYT
v
d
0 1
r
a
[aardva]
d
5
8
3
3
2
49
51
50
47
48
r
a
[aardvark]
46
1
44
v
45
1
d
1 0
43
NYT k
41 42
Figure 4: Figure for Problem 8 in Chapter 3.
Problem 7. If we combine three letters in the rst and second steps we get the following
code:
a
1
01
a
2
12
a
3
02
a
4
10
a
5
00
a
6
11
Thus the average length of the code is two ternary digits per symbol. If we calculate the
average length of the code in Example 3.3.1 we obtain an average length of 1.7 ternary digits
per symbol.
Problem 10
p
r
a
v
e
n
d
r
a
Chapter 3 8
a)
Message a
2
a
1
a
3
a
2
a
1
a
2
Transmitted binary sequence 1010001011
Received binary sequence 0010001011
Decoded sequence a
4
a
4
a
2
a
2
Depending on how you count the errors ve characters are received in error before the
rst correctly decoded character.
b)
Message a
2
a
1
a
3
a
2
a
1
a
2
Transmitted binary sequence 001011001000
Received binary sequence 101011001000
Decoded sequence a
1
a
1
a
3
a
2
a
1
a
2
Only a single character is received in error.
c)
Message a
2
a
1
a
3
a
2
a
1
a
2
Transmitted binary sequence 1010001011
Received binary sequence 1000001011
Decoded sequence a
2
a
3
a
4
a
2
a
2
four characters are received in error before the rst correct character.
For the minimum variance code the situation is dierent
Message a
2
a
1
a
3
a
2
a
1
a
2
Transmitted binary sequence 001011001000
Received binary sequence 000011001000
Decoded sequence a
2
a
2
a
3
a
2
a
1
a
2
Again, only a single character is received in error.
Problem 12.
Lets rst do the preprocessing: Note that y
max
= 41 and because we are going to assume a
zero for the prediction of the rst element of the sequence y
min
= 0.
y
i
32 33 35 39 37 38 39 40 40 40 40 39 40 40 41 40
y
i
0 32 33 35 39 37 38 39 40 40 40 40 39 40 40 41
d
i
32 1 2 4 -2 1 1 1 0 0 0 -1 1 0 1 -1
T
i
0 9 8 6 2 4 3 2 1 1 1 1 2 1 1 0
x
i
32 2 4 8 3 2 2 2 0 0 0 1 2 0 2 1
As J = 8 we divide the sequence of sixteen into two packets. The rst packet consists of the
numbers:
32 2 4 8 3 2 2 2
p
r
a
v
e
n
d
r
a
Chapter 3 9
There is signicant variation in the values in this packet so we would be best served with
one of the split sample options. The split sample option with m = 1 would require 43 bits,
m = 2 would require 35 bits, m = 3 would require 39 bits and for m > 3 would require more
than 40 bits so we pick m = 2. In the table we show the translation for each number in the
packet
Number binary representation code
32 100000 00000000001
2 000010 101
4 000100 0001
8 001000 00001
3 000011 111
2 000010 101
2 000010 101
2 000010 101
Stringing the codes together with an identier for the code option gives us the code for the
rst packet.
The second packet looks very dierent.
0 0 0 1 2 0 2 1
It looks like a candidate for the low entropy option. However, as we do not have the lookup
tables available we can just use the fundamental sequence option which gives us the following
code for the packet.
11101001100101
Problem 13.
First iteration:
Letter Probability
a
1
0.7
a
2
0.2
a
3
0.1
Second iteration:
Letter Probability
a
2
0.2
a
3
0.1
a
1
a
1
0.49
a
1
a
2
0.14
a
1
a
3
0.07
p
r
a
v
e
n
d
r
a
Chapter 3 10
Final iteration:
Letter Code
a
2
000
a
3
001
a
1
a
2
010
a
1
a
3
011
a
1
a
1
a
1
100
a
1
a
1
a
2
101
a
1
a
1
a
3
110
p
r
a
v
e
n
d
r
a
Chapter 4 11
Chapter 4
Problem 1. Given a number a [0, 1) with binary representation [b
1
b
2
. . . b
n
]
a = b
1
2
1
+b
2
2
2
+ +b
n
2
n
If b has a binary representation with [b
1
b
2
. . . b
n
] as prex, then
b = b
1
2
1
+b
2
2
2
+ +b
n
2
n
+b
n+1
2
(n+1)
+
Therefore,
b a = b
n+1
2
(n+1)
+
Obviously b a 0 and b a. To show b < a +
1
2
n
we note that
b a = b
n+1
2
(n+1)
+b
n+2
2
(n+2)
+
2
(n+1)
+ 2
(n+2)
+
<
1
2
n
Problem 4.
Count(1) = 40 Cum Count(0) = 0 Scale3 = 0
Count(2) = 1 Cum Count(1) = 40
Count(3) = 9 Cum Count(2) = 41
Total Count = 50 Cum Count(3) = 50
If m = 6
l
(0)
= 000000; u
(0)
= 111111 = 63
The rst element of the sequence to be encoded is 1.
l
(1)
= 0 +
64 Cum Count(0)
50
| = 0 = (00000000)
2
u
(1)
= 0 +
64 Cum Count(1)
50
| 1 = 50 = (110010)
2
The next element of the sequence is 3.
l
(2)
= 0 +
51 Cum Count(2)
50
| = 41 = (101001)
2
u
(2)
= 0 +
51 Cum Count(3)
50
| 1 = 50 = (110010)
2
p
r
a
v
e
n
d
r
a
Chapter 4 12
The MSBs of l
(2)
and u
(2)
are both 1. Therefore, we shift this value out and send it to the
decoder. All other bits are shifted left by one bit, giving
l
(2)
= (010010)
2
= 18
u
(2)
= (100101)
2
= 37.
Notice that while the MSBs of the limits are dierent, the second MSB of the upper limit
is zero, while the second MSB of the lower limit is one. This is the condition for the E
3
mapping. We complement the second MSB of both limits and shift one bit to the left,
shifting in a zero as the LSB of l
(2)
and a one as the LSB of u
(2)
. This gives us
l
(2)
= (000100)
2
= 4
u
(2)
= (101011)
2
= 43.
We also increment Scale3 to a value of one.
The next element in the sequence is 2. Updating the limits, we have
l
(3)
= 4 +
40 Cum Count(1)
50
| = 36 = (100100)
2
u
(3)
= 4 +
40 Cum Count(2)
50
| 1 = 35 = (100011)
2
and the lower limit is greater than the upper limit. Clearly an unacceptable situation.
Therefore we cannot use m = 6.
Problem 5
Letter Probability cdf
a
1
.2 F
X
(1) = 0.2
a
2
.3 F
X
(2) = 0.5
a
3
.5 F
X
(3) = 1.0
l
(0)
= 0
u
(0)
= 1
First letter is a
1
l
(1)
= 0 + (1 0) 0 = 0
u
(1)
= 0 + (1 0) .2 = .2
Second letter is a
1
l
(2)
= 0 + (.2 0) 0 = 0
u
(2)
= 0 + (.2 0) .2 = .04
p
r
a
v
e
n
d
r
a
Chapter 4 13
Third letter is a
3
l
(3)
= 0 + (.04 0) 0.5 = 0.02
u
(3)
= 0 + (.04 0) 1.0 = 0.04
Fourth letter is a
2
.
l
(4)
= 0.02 + (.04 0.02) 0.2 = 0.024
u
(4)
= 0.02 + (.04 0.02) 0.5 = 0.03
Fifth letter is a
3
l
(5)
= 0.024 + (.03 0.024) 0.5 = 0.027
u
(5)
= 0.024 + (.03 0.024) 1.0 = 0.03
Sixth letter is a
1
l
(6)
= 0.027 + (.03 0.027) 0.0 = 0.027
u
(6)
= 0.027 + (.03 0.027) 0.2 = 0.0276
Therefore a possible tag value is 0.0273.
Problem 6. The tag decodes to the following sequence:
a
3
a
2
a
2
a
1
a
2
a
1
a
3
a
2
a
2
a
3
Problem 7
Letter Count Cum Count
Cum Count[0] = 0
a(=1) 37 Cum Count[1] = 37
b(=2) 38 Cum Count[2] = 75
c(=3) 25 Cum Count[3] = 100
a) Total Count = 100
m = log
2
(Total Count)| + 2 = 7 + 2 = 9bits
b)
l
(0)
= (000000000)
2
= (0)
10
u
(0)
= (111111111)
2
= (511)
10
p
r
a
v
e
n
d
r
a
Chapter 4 14
First letter is a(1)
l
(1)
= 0 +
512 0
100
| = (0)
10
= (000000000)
2
u
(1)
= 0 +
512 37
100
| 1 = (188)
10
= (010111100)
2
Transmit 0.
l
(1)
= (000000000)
2
= (0)
10
u
(1)
= (101111001)
2
= (377)
10
Input = b (2)
l
(2)
= 0 +
378 37
100
| = (139)
10
= (010001011)
2
u
(2)
= 0 +
378 75
100
| 1 = (282)
10
= (100011010)
2
(2)
This is an E3 condition scale3=1
l
(2)
= (000010110)
2
= (22)
10
u
(2)
= (100110101)
2
= (309)
10
(3)
Input=a (1)
l
(3)
= 22 +
288 0
100
| = (22)
10
= (000010110)
2
u
(3)
= 22 +
288 37
100
| 1 = (105)
10
= (001101001)
2
(4)
Transmit 0
Transmit 1. Set scale3=0
Transmit 0
l
(3)
= (001011000)
2
= (88)
10
u
(3)
= (110100111)
2
= (423)
10
input = c (3)
l
(4)
= 88 +
336 75
100
| = (340)
10
= (101010100)
2
u
(4)
= 88 +
336 100
100
| 1 = (423)
10
= (110100111)
2
p
r
a
v
e
n
d
r
a
Chapter 4 15
Transmit 1
l
(4)
= (010101000)
2
u
(4)
= (101001111)
2
E3 condition Set scale3=1.
l
(4)
= (001010000)
2
= (80)
10
u
(4)
= (110011111)
2
= (415)
10
Input = a (1)
l
(5)
= 80 +
336 0
100
| = (80)
10
= (001010000)
2
u
(5)
= 80 +
336 37
100
| 1 = (203)
10
= (011001011)
2
Transmit 0
Transmit 1. Set scale3 = 0.
l
(5)
= (010100000)
2
= (160)
10
u
(5)
= (110010111)
2
= (407)
10
Input = b (2)
l
(6)
= 160 +
248 37
100
| = (251)
10
= (011111011)
2
u
(6)
= 160 +
248 75
100
| 1 = (345)
10
= (101011001)
2
E3 condition Set scale3=1.
l
(6)
= (011110110)
2
= (246)
10
u
(6)
= (110110011)
2
= (435
10
Input=b (2)
l
(7)
= 246 +
190 37
100
| = (316)
10
= (100111100)
2
u
(7)
= 246 +
190 75
100
| 1 = (388)
10
= (110000100)
2
Transmit 1
Transmit 0. Scale3 = 0
l
(7)
= (001111000)
2
u
(7)
= (100001001)
2
p
r
a
v
e
n
d
r
a
Chapter 4 16
To terminate the encoding send the bits in the lower limit.
Transmit 00111100.
c) Received sequence: 001010110001111000.
l
(0)
= (000000000)
2
= (0)
10
u
(0)
= (111111111)
2
= (511)
10
t = (001010010)
2
= (86)
10
86 100 1
512
= 16
Compare this against the Cum Counts
k = 0 16 Cum Count[0] = 0
k = 1 16 < Cum Count[1] = 37
Decode k=1 a.
First letter is a. Update upper and lower limits.
l
(1)
= 0 +
512 0
100
| = (0)
10
= (000000000)
2
u
(1)
= 0 +
512 37
100
| 1 = (188)
10
= (010111100)
2
(5)
Shift out 0 from upper and lower limit and the tag and read in one more bit for the
tag
l
(1)
= (000000000)
2
= (0)
10
u
(1)
= (101111001)
2
= (377)
10
t = (010101100)
2
= (172)
10
172 100 1
378
= 45
k = 0 45 Cum Count[0] = 0
k = 1 45 Cum Count[1] = 37
k = 2 45 < Cum Count[2] = 75
p
r
a
v
e
n
d
r
a
Chapter 4 17
Decode k = 2 b
Update upper and lower limits
l
(2)
= 0 +
378 37
100
| = (139)
10
= (010001011)
2
u
(2)
= 0 +
378 75
100
| 1 = (282)
10
= (100011010)
2
(6)
This is an E3 condition. Complement second msb on upper and lower limits and on
tag and shift left. Read in a 0 into lsb of the lower limit, a 1 into lsb of the upper
limit, and a bit from the bitstream for the tag.
l
(2)
= (000010110)
2
= (22)
10
u
(2)
= (100110101)
2
= (309)
10
t = (001011000)
2
= (88)
10
(88 22 + 1) 100 1
288
= 23
k = 0 23 Cum Count[0] = 0
k = 1 23 < Cum Count[1] = 37
Decode k = 1 a
Update upper and lower limits:
l
(3)
= 22 +
288 0
100
| = (22)
10
= (000010110)
2
u
(3)
= 22 +
288 37
100
| 1 = (105)
10
= (001101001)
2
(7)
Shift out two 0s from the upper and lower limits and the tag. Shift in 0s into the
lsbs of the lowerlimit, 1s into the lsbs of the upper limit and read two bits from the
received bitstream into the lsbs of the tag.
l
(3)
= (001011000)
2
= (88)
10
u
(3)
= (110100111)
2
= (423)
10
t = (101100011)
2
= (355)
10
p
r
a
v
e
n
d
r
a
Chapter 4 18
(355 88 + 1) 100 1
336
= 79
k = 0 79 Cum Count[0] = 0
k = 1 79 Cum Count[1] = 37
k = 2 79 Cum Count[2] = 75
k = 3 79 < Cum Count[3] = 100
Decode k = 3 c
Update upper and lower limits
l
(4)
= 88 +
336 75
100
| = (340)
10
= (101010100)
2
u
(4)
= 88 +
336 100
100
| 1 = (423)
10
= (110100111)
2
Shift out 1 from upper and lower limits and tag.
l
(4)
= (010101000)
2
u
(4)
= (101001111)
2
t = (011000111)
2
E3 condition. Complement second msb of upper and lower limits and tag and shift
left.
l
(4)
= (001010000)
2
= (80)
10
u
(4)
= (110011111)
2
= (415)
10
t = (010001111)
2
= (143)
10
(143 80 + 1) 100 1
336
= 19
k = 0 19 0
k = 1 19 < 37
p
r
a
v
e
n
d
r
a
Chapter 4 19
Decode k = 1 a.
Update upper and lower limits:
l
(5)
= 80 +
336 0
100
| = (80)
10
= (001010000)
2
u
(5)
= 80 +
336 37
100
| 1 = (203)
10
= (011001011)
2
Shift 0 out of upper and lower limits and tag.
l
(5)
= (010100000)
2
= (160)
10
u
(5)
= (110010111)
2
= (407)
10
t = (100011110)
2
= (286)
10
(286 160 + 1) 100 1
248
= 51
k = 0 51 0
k = 1 51 37
k = 2 51 < 75
Decode k = 2 b.
Update upper and lower limits:
l
(6)
= 160 +
248 37
100
| = (251)
10
= (011111011)
2
u
(6)
= 160 +
248 75
100
| 1 = (345)
10
= (101011001)
2
E3 condition.
l
(6)
= (011110110)
2
= (246)
10
u
(6)
= (110110011)
2
= (435
10
t = (100111100)
2
= (316)
10
(316 246 + 1) 100 1
190
= 37
p
r
a
v
e
n
d
r
a
Chapter 4 20
k = 0 37 0
k = 1 37 37
k = 2 37 < 75
Decode k = 2 b.
Assuming we know that only seven symbols were transmitted we can stop decoding.
p
r
a
v
e
n
d
r
a
Chapter 5 21
Chapter 5
Problem 3.
Index Codebook entry
1 a
2 b
3 r
4 y
5 ,b
6 a,b
7 ,bb
8 ba
9 ar
10 r,b
11 ,ba
12 arr
13 ra
14 ay
15 y,b
16 ,bby
17 y,bb
18 bar
19 rr
20 ray
21 ya
22 ar,b
23 ,ba
Transmitted sequence is
1, 5, 2, 1, 3, 5, 9, 3, 1, 4, 7, 15, 8, 3, 13, 4, 9, 7, 14
p
r
a
v
e
n
d
r
a
Chapter 5 22
Problem 4.
Index Codebook entry
1 a
2 ,b
3 h
4 i
5 s
6 t
7 th
8 hi
9 is
10 s,b
11 ,bh
12 ha
13 at
14 t,b
15 ,bi
16 is,b
17 ,bhi
18 is,bh
19 hat
20 t,bi
21 it
22 t,bis
23 s,bh
24 his
25 s,bha
26 at
The decoded message is this hat is his hat it is his hat
p
r
a
v
e
n
d
r
a
Chapter 5 23
Problem 5.
Index Codebook entry
1 a
2 ,b
3 r
4 t
5 ra
6 at
7 ta
8 ata
9 atat
10 t,b
11 ,ba
12 a,b
13 ,br
14 rat
15 t,ba
16 at,b
17 ,bab
18 ,bra
19 at
The decoded message is ratatatat a rat at a rat.
Problem 6.
< 0, 0, C(b) >< 0, 0, C(a) >< 0, 0, C(r) >< 1, 1, C(a) >< 0, 0, C(y) >< 5, 2, C(,b) >
< 9, 3, C(,b) >< 4, 1, C(y) >< 7, 4, C(r) >< 3, 1, C(y) >< 12, 4, C(a) >
Problem 7
The decoded sequence is ratatatat a rat at a rat.
Problem 8.
Initial dictionary:
Index Entry
1 S
2 ,b
3 I
4 T
5 H
p
r
a
v
e
n
d
r
a
Chapter 5 24
Received sequence: 4, 5, 3, 1, 2, 8, 2, 7, 9, 7, 4
Received Decode Update Dictionary
4 T Entry 6: T . . .
5 H Entry 6: TH Entry 7: H . . .
3 I Entry 7: HI Entry 8: I . . .
1 S Entry 8: IS Entry 9: S . . .
2 ,b Entry 9: S,b Entry 10:,b . . .
8 IS Entry 10: ,bI Entry 11: IS . . .
2 ,b Entry 11: IS,b Entry 12:,b . . .
7 HI Entry 12: ,bH Entry 13: HI . . .
9 S,b Entry 13:HIS Entry 14:S,b
7 HI Entry 14: S,bH Entry 15: HI . . .
4 T Entry 15: HIT Entry 16:T . . .
Final Dictionary:
Index Entry
1 S
2 ,b
3 I
4 T
5 H
6 TH
7 HI
8 IS
9 S,b
10 ,bI
11 IS,b
12 ,bH
13 HIS
14 S,bH
15 HIT
16 T . . .
p
r
a
v
e
n
d
r
a
Chapter 6 25
Chapter 6
Problem 3(a)
Lets rst obtain all cyclical shifts of the message to be encoded
e t a ,b c e t a ,b a n d ,b b e t a ,b c e t a
a e t a ,b c e t a ,b a n d ,b b e t a ,b c e t
t a e t a ,b c e t a ,b a n d ,b b e t a ,b c e
e t a e t a ,b c e t a ,b a n d ,b b e t a ,b c
c e t a e t a ,b c e t a ,b a n d ,b b e t a ,b
,b c e t a e t a ,b c e t a ,b a n d ,b b e t a
a ,b c e t a e t a ,b c e t a ,b a n d ,b b e t
t a ,b c e t a e t a ,b c e t a ,b a n d ,b b e
e t a ,b c e t a e t a ,b c e t a ,b a n d ,b b
b e t a ,b c e t a e t a ,b c e t a ,b a n d ,b
,b b e t a ,b c e t a e t a ,b c e t a ,b a n d
d ,b b e t a ,b c e t a e t a ,b c e t a ,b a n
n d ,b b e t a ,b c e t a e t a ,b c e t a ,b a
a n d ,b b e t a ,b c e t a e t a ,b c e t a ,b
,b a n d ,b b e t a ,b c e t a e t a ,b c e t a
a ,b a n d ,b b e t a ,b c e t a e t a ,b c e t
t a ,b a n d ,b b e t a ,b c e t a e t a ,b c e
e t a ,b a n d ,b b e t a ,b c e t a e t a ,b c
c e t a ,b a n d ,b b e t a ,b c e t a e t a ,b
,b c e t a ,b a n d ,b b e t a ,b c e t a e t a
a ,b c e t a ,b a n d ,b b e t a ,b c e t a e t
t a ,b c e t a ,b a n d ,b b e t a ,b c e t a e
Now lets sort the cyclically shifted sequences:
p
r
a
v
e
n
d
r
a
Chapter 6 26
,b a n d ,b b e t a ,b c e t a e t a ,b c e t a
,b b e t a ,b c e t a e t a ,b c e t a ,b a n d
,b c e t a ,b a n d ,b b e t a ,b c e t a e t a
,b c e t a e t a ,b c e t a ,b a n d ,b b e t a
a ,b a n d ,b b e t a ,b c e t a e t a ,b c e t
a ,b c e t a ,b a n d ,b b e t a ,b c e t a e t
a ,b c e t a e t a ,b c e t a ,b a n d ,b b e t
a e t a ,b c e t a ,b a n d ,b b e t a ,b c e t
a n d ,b b e t a ,b c e t a e t a ,b c e t a ,b
b e t a ,b c e t a e t a ,b c e t a ,b a n d ,b
c e t a ,b a n d ,b b e t a ,b c e t a e t a ,b
c e t a e t a ,b c e t a ,b a n d ,b b e t a ,b
d ,b b e t a ,b c e t a e t a ,b c e t a ,b a n
e t a ,b a n d ,b b e t a ,b c e t a e t a ,b c
e t a ,b c e t a ,b a n d ,b b e t a ,b c e t a
e t a ,b c e t a e t a ,b c e t a ,b a n d ,b b
e t a e t a ,b c e t a ,b a n d ,b b e t a ,b c
n d ,b b e t a ,b c e t a e t a ,b c e t a ,b a
t a ,b a n d ,b b e t a ,b c e t a e t a ,b c e
t a ,b c e t a ,b a n d ,b b e t a ,b c e t a e
t a ,b c e t a e t a ,b c e t a ,b a n d ,b b e
t a e t a ,b c e t a ,b a n d ,b b e t a ,b c e
The sequence consisting of the last letters of the cyclically sorted set is
a d a a t t t t ,b ,b ,b ,b n c a b c a e e e e
If we now begin with the index assignments
,b 0
a 1
b 2
c 3
d 4
e 5
n 6
t 7
The move to front indices are
1 4 1 0 7 0 0 0 3 0 0 0 7 6 4 6 2 2 7 0 0 0
p
r
a
v
e
n
d
r
a
Chapter 6 27
and the original sequence has an index of 14 (the rst sequence in the list has an index of
0).
Problem 3(b)
In order to decode we rst need to obtain the sequence of last letters from the move-to-front
indices. Using the initial index assignment the rst letter (with index of 1) is decoded to be
a. The index assignment changes to
a 0 d 4
,b 1 e 5
b 2 n 6
d 3 t 7
The next index is 4 and the element with an index of 4 is d. The index assignment now
becomes
d 0 d 4
a 1 e 5
,b 2 n 6
b 3 t 7
nd the next decoded letter is a. Continuing in this fashion the sequence of index assignments
and decoded letters are:
a 0 d 4
d 1 e 5
,b 2 n 6
b 3 t 7
Decode 0 as a.
Decode 7 as t and change index assignment to:
t 0 b 4
a 1 c 5
d 2 e 6
,b 3 n 7
Decode 0 as t.
Decode 0 as t.
Decode 0 as t.
Decode 3 as ,b and change the index assignment to:
p
r
a
v
e
n
d
r
a
Chapter 6 28
,b 0 b 4
t 1 c 5
a 2 e 6
d 3 n 7
Decode 0 as ,b.
Decode 0 as ,b.
Decode 0 as ,b.
Decode 7 as n and change index assignment to:
n 0 d 4
,b 1 b 5
t 2 c 6
a 3 e 7
Decode 6 as c and change index assignment to
c 0 a 4
n 1 d 5
,b 2 b 6
t 3 e 7
Decode 4 as a and change the index assignment to
a 0 t 4
c 1 d 5
n 2 b 6
,b 3 e 7
Decode 6 as b and change the index assignment to:
b 0 ,b 4
a 1 t 5
c 2 d 6
n 3 e 7
Decode 2 as c and change the index assignment to:
c 0 ,b 4
b 1 t 5
a 2 d 6
n 3 e 7
p
r
a
v
e
n
d
r
a
Chapter 6 29
Decode 2 as a and change the index assignment to:
a 0 ,b 4
c 1 t 5
b 2 d 6
n 3 e 7
Decode 7 as e and change the index assignment to:
e 0 n 4
a 1 ,b 5
c 2 t 6
b 3 d 7
Decode 0 as e.
Decode 0 as e.
Decode 0 as e.
Thus the decoded sequence of last letters becomes:
a d a a t t t t ,b ,b ,b ,b n c a b c a e e e e
In order to get the transformation array we need to get the sequence of rst letters. This is
simply the sorted version of our decoded sequence.
L: a d a a t t t t ,b ,b ,b ,b n c a b c a e e e e
F: ,b ,b ,b ,b a a a a a b c c d e e e e n t t t t
From this we can construct T as
4 12 5 6 18 19 20 21 0 1 2 3 17 10 7 9 11 8 13 14 15 16
Now we begin decoding the original sequence. The last letter of the original sequence is in
location 14 (starting the count with 0) which is an a. T[14] = 7 therefore the preceding
letter is t. T[7] = 21, therefore the letter preceding t is e. T[21] = 16, therefore the letter
preceding e is c. T[16] = 11 therefore the letter preceding c is ,b. Continuing in this fashion
T[11] = 3 a (8)
T[3] = 6 t
T[6] = 20 e
T[20] = 15 b
p
r
a
v
e
n
d
r
a
Chapter 6 30
T[15] = 9 ,b
T[9] = 1 d
T[1] = 12 n
T[12] = 17 a
T[17] = 8 ,b
T[8] = 0 a
T[0] = 4 t
T[4] = 18 e
T[18] = 13 c
T[13] = 10 ,b
T[10] = 2 a
T[2] = 5 t
T[5] = 19 e
and we obtain the decoded sequence.
p
r
a
v
e
n
d
r
a
Chapter 7 31
Chapter 7
Problem 1
Let w(n) and b(n) represent white and black runs of length n respectively.
The lines of the image are w(16)
w(4), b(1), w(2), b(1), w(1), b(1), w(6)
w(4), b(1), w(2), b(1), w(8)
w(4), b(4), w(1), b(1), w(6)
w(4), b(1), w(2), b(1), w(1), b(1), w(6)
w(4), b(1), w(2), b(1), w(1), b(1), w(6)
w(16)
w(16)
Replacing the runlengths with the corresponding modied Human codes:
101010
1011, 010, 0111, 010, 000111, 010, 1110
1011, 010, 0111, 010, 10011
1011, 011, 000111, 010, 1110
1011, 010, 0111, 010, 000111, 010, 1110
1011, 010, 0111, 010, 000111, 010, 1110
101010
101010
Problem 4 a
Number of bits in 15 seconds: 9600 15 = 144, 000.
This corresponds to
144,000
8
= 18, 000 pixels.
At 512 pixels per line this is a little more than 35 lines which corresponds to about 6.9% of
the image.
Problem 4 b
There are
512512
88
= 4, 096 8 8 blocks in the image. If each block is represented by a single
pixel, this corresponds to 32,768 bits which would be transmitted in 3.4133 seconds over a
9600 bps line. The second approximation would require three times as many pixel or 10.24
seconds. Together the rst two approximations would require about 13.65 seconds.
p
r
a
v
e
n
d
r
a
Chapter 8 32
Chapter 8
Problem 2.
a)
M = 1 : R = 1 D = 0;
M = 2 : R =
1
2
We have four possibilities with the following probabilities and distortions:
Message Probability Distortion
00 P(0)
2
0
01 P(0)(1 P(0))
1
2
10 P(0)(1 P(0))
1
2
11 (1 P(0))
2
0
Total distortion D = 0.16
Similarly (with more calculations) we obtain
M = 4 : R = 0.25 D = 0.1856
M = 8 : R = 0.125 D = 0.1970
M = 16 : R = 0.0625 D = 0.199780
For a binary source
R =
_
H
b
(p) H
b
(D) D < min p, 1 p
0 otherwise
(9)
Comparing with the rate distortion function
Distortion Rate from scheme Optimal rate
0.0 1.0 0.721928
0.16 0.5 0.087619
0.1856 0.25 0.029752
0.1970 0.125 0.006041
0.199780 0.0625 0.000440
b) If we encoded the output of the encoder using an entropy code which achieves a
performance equal to the entropy the comparison is not as lopsided
p
r
a
v
e
n
d
r
a
Chapter 8 33
Distortion Rate from entropy coded scheme Optimal rate
0.0 0.721928 0.721928
0.16 0.12 0.087619
0.1856 0.045 0.029752
0.1970 0.01043 0.006041
0.199780 0.0010 0.000440
Problem 4.
H(X
d
[Y
d
) =

j=

i=
P(x
i
[y
j
)P(y
j
) log P(x
i
[y
j
)
=

j=

i=
f
X|Y
(x
i
[y
j
)f
Y
(y
j
)log f
X|Y
(x
i
[y
j
)
=

j=

i=
f
X|Y
(x
i
[y
j
)f
Y
(y
j
)log f
X|Y
(x
i
[y
j
)

j=

i=
f
X|Y
(x
i
[y
j
)f
Y
(y
j
)log
=

j=

i=
f
X|Y
(x
i
[y
j
)f
Y
(y
j
)log f
X|Y
(x
i
[y
j
) log
(10)
Problem 5.
H(X[Y ) H(X) =
N1

i=0
M1

j=0
P(x
i
, y
j
) log P(x
i
[y
j
) +
N1

i=0
M1

j=0
P(x
i
, y
j
) log P(x
i
)
=
N1

i=0
M1

j=0
P(x
i
, y
j
) log
P(x
i
)
P(x
i
[y
j
)
= E
_
log
P(x
i
)
P(x
i
[y
j
)
_
Now we use Jensens inequality
H(X[Y ) H(X) = E
_
log
P(X)
P(X[Y )
_
log
_
E
_
P(X)
P(X[Y )
__
p
r
a
v
e
n
d
r
a
Chapter 8 34
= log
_
_
N1

i=0
M1

j=0
P(x
i
, y
j
)
P(x
i
)
P(x
i
[y
j
)
_
_
= log
_
_
N1

i=0
M1

j=0
P(y
j
)P(x
i
)
_
_
= 0
where equality is achieved when P(x
i
) = P(x
i
[y
j
) or in other word the random variables X
and Y are independent.
Problem 6
I(X; Y ) =
N1

i=0
M1

j=0
P(x
i
, y
j
) log
P(x
i
[y
j
)
P(x
i
)
=
N1

i=0
M1

j=0
P(x
i
, y
j
) log
P(x
i
, y
j
)
P(x
i
)P(y
j
)
=
N1

i=0
M1

j=0
P(x
i
, y
j
) log
P(y
j
[x
i
)
P(y
j
)
= I(Y ; X)
Problem 7
I(X; Y ) = H(X) H(X[Y )
H(X) = p log p (1 p) log(1 p) = H
b
(p)
H(X[Y ) = P(0[0)P(Y = 0) log P(0[0) P(0[1)P(Y = 1) log P(0[1) P(1[0)P(Y = 0) log P(1[0) P(1[1)P(Y = 1) log P(1[1)
= (1 D)P(Y = 0) log(1 D) DP(Y = 1) log D DP(Y = 0) log D (1 D)P(Y = 1) log(1 D)
= [(1 D) log(1 D)][P(Y = 0) +P(Y = 1)] [Dlog D][P(Y = 0) +P(Y = 1)]
= (1 D) log(1 D) Dlog D
= H
b
(D)
Therefore
I(X; Y ) = H
b
(p) H
b
(D)
p
r
a
v
e
n
d
r
a
Chapter 9 35
Chapter 9
Problem 1. Leibnitz rule is given by

t
_
b(t)
a(t)
f(x, t)dx =
_
b(t)
a(t)

t
f(x, t)dx +f(b(t), t)

t
b(t) f(a(t), t)

t
a(t)
where a(t) and b(t) are monotonic functions of t.
The quantization noise power is given by:

2
q
= 2
M
2
1

i=1
_
i
(i1)
_
x
2i 1
2

_
2
f
X
(x)dx + 2
_

(
M
2
1)
_
x
M 1
2

_
2
f
X
(x)dx
Taking a derivative with respect to and using Leibnitz rule we obtain:

2
q
= 2
M
2
1

i=1
_
2(
(2i 1)
2
)
_
i
(i1)
_
x
2i 1
2

_
f
X
(x)dx +
i
2
4
f
X
(i)
(i 1)
2
4
f
X
((i 1))
_
+
_
2(
(M 1)
2
)
_

(
M
2
1)
_
x
M 1
2

_
f
X
(x)dx
(
M
2
1)
2
4
f
X
((
M
2
1))
_
Setting this to zero and noticing that
M
2
1

i=1
_
i
2
4
f
X
(i)
(i 1)
2
4
f
X
((i 1))
_
=
(
M
2
1)
2
4
f
X
((
M
2
1))
we obtain the desired result.
Problem 6.
Input c(x) Quantize c
1
(x) Uniformly quantized
-0.8 -1.6 -1.5 -0.75 -0.5
1.2 2.13 2.5 1.75 1.5
0.5 1.0 0.5 0.25 0.5
0.6 1.2 1.5 0.75 0.5
3.2 3.47 3.5 3.25 3.5
-0.3 -0.6 -0.5 -0.25 -0.5
p
r
a
v
e
n
d
r
a
Chapter 10 36
Chapter 10
Problem 1
Decrease in mse due to the inclusion of an output point at (, 0) is 0.0038. By symmetry,
the same is true for the inclusion of outputs at the other three points. The increased mse
because of the removal of one of the corner points is 0.00026

2
q,original
= 10

SNR
10
= 0.07178

2
q,new
= 0.05768
SNR
new
= 12.39dB
Therefore the increase in SNR is 0.95 dB
Problem 5
a) Referring to Figure 5:
P
A
=
_
3
2
e

2x
dx
_
4
3
e

2y
dy = 0.00059
P
B
=
_
_
4
3
e

2x
dx
_
2
= 0.00021
Total increase in overload probability is 8P
A
+ 4P
B
= 0.0056.
b)
P
C
=
_

0
1

2
e

2x
dx
_
5
4
1

2
e

2y
dy = 0.00166
Decrease of probability of overload is 8P
C
= 0.01329.
p
r
a
v
e
n
d
r
a
Chapter 10 37
A
A
B
C C
A B
A
C
C
C
C
A
A B
B
A
A
Figure 5: Figure for Problem 5 in Chapter 10.
p
r
a
v
e
n
d
r
a
Chapter 12 38
Chapter 12
Problem 1. Let X = a
1
, a
2
, , a
N
then
V = v : v =
N

i=1
c
i
a
i
, c
i
1
where 1 denotes the set of real numbers.
1a. Let v
1
, v
2
V . Then by denition
v
1
=
N

i=1

i
a
i
, and v
2
=
N

i=1

i
a
i
for
i
,
i
1, and
v
1
+v
2
=
N

i=1

i
a
i
+
N

i=1

i
a
i
=
N

i=1
(
i
+
i
)a
i
Dene
i
=
i
+
i
, then
i
1 and
v
1
+v
2
=
N

i=1

i
a
i
and v
1
+v
2
V
1b. By picking c
i
to be zero for all i, we obtain a vector which by denition is in V and
for which v
i
+ = v
i
for all v
i
V . Thus is the additive identity and it is contained in V .
1c. Let x =

N
i=1

i
a
i
. Dene
x =
N

i=1
(
i
)a
i
If
i
1 then (
i
) 1 and x V
Problem 2. Let F() = T [f(t)] and assume all integrals exist and we can switch the order
of integration.
_

[f(t)[
2
dt =
_

f(t)f

(t)dt
=
_

_
1
2
_

F()e
jt
d
_
f

(t)dt
=
1
2
_

F()
__

(t)e
jt
dt
_
d
=
1
2
_

F()
__

f(t)e
jt
dt
_

d
=
1
2
_

F()F

()d
=
1
2
_

[F()[
2
d
p
r
a
v
e
n
d
r
a
Chapter 12 39
Problem 3. Let F() = T [f(t)].
T
_
f(t)e
jot
_
=
_

f(t)e
jot
e
jt
dt
=
_

f(t)e
j(o)t
dt
= F(
o
)
Problem 4. Let F() = T [f(t)], F
1
() = T [f
1
(t)], and F
2
() = T [f
2
(t)], and assume
that all the integrals exist and we can change the order of integration.
F() =
_

f(t)e
jt
dt
=
_

f
1
()f
2
(t )de
jt
dt
=
_

f
1
()f
2
(t )e
jt
ddt
Let u = t , then t = u + and dt = du.
F() =
_

f
1
()
_

f
2
(u)e
j(u+)
dud
=
_

f
1
()e
j
__

f
2
(u)e
ju
du
_
d
=
_

f
1
()e
j
F
2
()d
= F
1
()F
2
()
Problem 5. We will make use of the fact that as the inverse Fourier transform of the delta
function is given by
T
1
[()] =
1
2
()e
jt
d =
1
2
Therefore, by uniqueness of the Fourier transform
T
_
1
2
_
= ()
We will also make use of the modulation property of the Fourier transform.
First note that f(t) =

n=
(t nT) is a periodic function with period T and therefore
has a Fourier series representation
f(t) =

n=
c
n
e
jnot
,
o
=
2
T
p
r
a
v
e
n
d
r
a
Chapter 12 40
c
n
=
1
T
_ T
2

T
2

n=
(t nT)dt =
1
T
_ T
2

T
2
(t)dt =
1
T
Therefore,
f(t) =
1
T

n=
e
jnot
= =

o
2

n=
e
jnot
=
o

n=
1
2
e
jnot
Now using the fact that T
_
1
2
_
= () and the modulation property
T
_
1
2
e
jnot
_
= ( n
o
)
F() = T [f(t)]
= T
_

n=
1
2
e
jnot
_
=
o

n=
T
_
1
2
e
jnot
_
=
o

n=
(
o
)
Problem 6.
(a).
H(z) =

n=0
2
n
z
n
=

n=0
(2
1
z
1
)
n
=
1
1 2
1
z
1
[2
1
z
1
[ < 1
=
2z
2z 1
[z[ >
1
2
(b).
H(z) =

n=0
(n
2
n)3
n
z
n
p
r
a
v
e
n
d
r
a
Chapter 12 41
=

n=0
n(n 1)3
n
z
n
= 3
2

n=0
n(n 1)(3
1
)
n2
z
n
Setting m = 2 in Example 11.9.2
H(z) =
1
9
2z
(z
1
3
)
2
(c).
H(z) =

n=0
(n2
n
+ 0.6
n
)z
n
=

n=0
n2
n
z
n
+

n=0
0.6
n
z
n
=
1
2

n=0
n(2
1
)
n1
z
n
+

n=0
0.6
n
z
n
=
1
2
z
(z
1
2
)
2
+
z
z 0.6
[z[ > 0.6
Problem 7.
(a).
y
n
= 0.6y
n1
+ 0.5x
n
+ 0.2x
n1
Y (z) = 0.6z
1
Y (z) + 0.5X(z) + 0.2z
1
X(z)
Therefore,
Y (z)(1 0.6z
1
) = (0.5 + 0.2z
1
)X(z)
and
H(z) =
Y (z)
X(z)
=
0.5 + 0.2z
1
1 0.6z
1
=
5z + 2
10z 6
(b).
H(z) =
5z + 2
10z 6
Performing long division we obtain the series
0.5 + 0.5z
1
+ 0.3z
2
+ 0.18z
3
+ 0.108z
4
+ 0.0648z
5
+
p
r
a
v
e
n
d
r
a
Chapter 12 42
At this time we can see the pattern. From the third term on the nth element of the series
is of the form 0.3(0.6)
n2
. Therefore,
h
n
=
_
.5 n = 0, 1
0.3(0.6)
n2
n > 1
Problem 8.
(a)
H(z) =
5
z 2
We have a pole outside the unit circle therefore this is the Z-transform of a left handed
sequence. Therefore, h
n
is of the form
h
0
+h
1
z +h
2
z
2
+
Using the long division method we get the series

5
2

5
4
z
5
8
z
2

5
16
z
3

Therefore,
h
n
=
_
5(2)
(n+1)
n = 0, 1, 2,
0 n = 1, 2, 3,
(b)
H(z) =
z
z
2
0.25
=
z
(z 0.5)(z + 0.5)
Lets expand this using partial fraction expansion:
H(z)
z
=
,z
,z(z 0.5)(z + 0.5)
=
A
1
z 0.5
+
A
2
z + 0.5
To nd the coecients A
1
and A
2
A
1
=
1
z + 0.5
[
z=0.5
= 1
A
2
=
1
z 0.5
[
z=0.5
= 1
Therefore,
H(z) =
z
z 0.5

z
z + 0.5
and
h
n
= 0.5
n
u[n] (0.5)
n
u[n]
p
r
a
v
e
n
d
r
a
Chapter 12 43
(c)
H(z) =
z
z 0.5
By inspection,
h
n
= 0.5
n
u[n]
p
r
a
v
e
n
d
r
a
Chapter 13 44
Chapter 13
Problem 1
[X
1
X
2
[
2
= (X
1
X
2
)
T
(X
1
X
2
)
= (A
T

1
A
T

2
)
T
(A
T

1
A
T

2
)
=
T
1
AA
T

T
1
AA
T

T
2
AA
T

1
+
T
2
AA
T

2
=
T
1

T
1

T
2

1
+
T
2

2
= [
1

2
[
2
Problem 2a.
If we transform
10 11 12 11 12 13 12 11
we obtain the coecients
32.527 -1.281 -1.307 0.450 -1.414 -0.301 0.541 0.255
Notice that the DC coecient has a rather high value while the other coecients have
relatively low values.
When we transform the vector
10 -10 8 -7 8 -8 7 -7
we obtain the coecients
0.354 4.251 0.350 5.046 2.475 8.384 1.769 20.388
This time the higher frequency coecients have larger magnitudes than the low frequency
coecients.
Concatenating the coecients we obtain the plot shown in Figure 6.
Problem 2b. If we concatenate the two vectors and take a 16 point DCT the results are
23.25 21.78 -3.92 -6.96 -0.68 6.04 -3.25 -2.57 0.75 3.16 -6.14 1.61 1.63 5.50 -14.24 13.56
The results are plotted in Figure 7
Problem 2c. Comparing the plots shown in Figures 6 and 7 we can see that we obtain
a larger number of coecients with relatively large magnitudes when we use the 16 point
DCT. This is due to the diering natures of the two vectors. As more large magnitude
coecients mean more bits required, it would be better from a compression perspective to
use two 8 points DCT rather than one 16 point DCT in this example.
p
r
a
v
e
n
d
r
a
Chapter 13 45
0 2 4 6 8 10 12 14 16
15
10
5
0
5
10
15
20
25
30
35
8

p
o
i
n
t

D
C
T

c
o
e
f
f
i
c
i
e
n
t
s
Figure 6: Plot of the 8 point coecients of the two vectors. The rst eight coecients belong
to the rst vector while the second eight coecients are the result of an 8 point DCT of the
second vector.
p
r
a
v
e
n
d
r
a
Chapter 13 46
0 2 4 6 8 10 12 14 16
15
10
5
0
5
10
15
20
25
30
35
1
6

p
o
i
n
t

D
C
T

c
o
e
f
f
i
c
i
e
n
t
s
Figure 7: Plot of the 16 point DCT coecients obtained by taking the 16 point DCT of the
concatenation of the two vectors.
p
r
a
v
e
n
d
r
a
Chapter 13 47
Problem 3
a) By rows:
_

_
10 2 4 0
7 1 3 1
5 1 1 1
4 0 0 0
_

_
Then by columns:
_

_
26 4 8 2
4 2 2 0
8 2 6 0
2 0 0 2
_

_
b) By columns rst:
_

_
10 7 5 4
2 1 1 0
4 3 1 0
0 1 1 0
_

_
Then by rows:
_

_
26 4 8 2
4 2 2 0
8 2 6 0
2 0 0 2
_

_
c) Results of a) and b) are identical.
p
r
a
v
e
n
d
r
a
Chapter 14 48
Chapter 14
Problem 1 The impulse response h
n
of a system is the output when the input is
n
where

n
=
_
1 n = 0
0 otherwise
By the second property if the input to the system is a
n
then the output is y
n
= h
n
.
Lets write this as

n
h
n

We can write the input sequence x

n
as
x
n
= x
0

n
+x
1

n1
+x
2

n2
+
Using both properties we can see that
x
0

n
x
0
h
n

x
1

n1
x
1
h
n1

x
2

n2
x
2
h
n2

.
.
.
.
.
.
x
M

nM
x
M
h
nM

.
.
.
.
.
.
Adding the left side we obtain the input sequence. By the second property the corresponding
output is the sum of the right side or
y
n
=

x
k
h
nk
Problem 2.
(a) Lets use a very simple impulse response:
h
1,0
=
1
2
, h
1,1
=
1
2
, h
1,2
=
1
2
, h
1,3
=
1
2
This satises the requirement that [h
1,j
[ = [h
1,k
[. Furthermore,
3

i=0
h
2
1,i
= 4
1
4
= 1
and
3

i=0
h
1,i
h
1,i+2
= h
1,0
h
1,2
+h
1,1
h
1,3
=
1
4

1
4
= 0
and the orthonormality requirement is satised.
p
r
a
v
e
n
d
r
a
Chapter 14 49
(b) The magnitude and phase response of this lter are plotted in Figure 8. It is not a very
good lter.
(c) Using Equation (13.23) we obtain the following coecients for the high pass lter:
h
2,0
=
1
2
, h
2,1
=
1
2
, h
1,2
=
1
2
, h
1,3
=
1
2
(d) The magnitude and phase response of th high pass lter are plotted in Figure 9.
Problem 3.
x
0
x
1
x
2
x
3
x
4
x
5
x
6
x
7
x
8
x
9
x
10

1 -1 1 -1 1 -1 1 -1 1 -1 1
(a)
y
n
= h
0
x
n
+h
1
x
n1
=
1

2
x
n
+
1

2
x
n1
y
0
=
1

2
1 +
1

2
0 =
1

2
y
1
=
1

2
(1) +
1

2
1 = 0
y
2
=
1

2
1 +
1

2
(1) = 0
y
3
=
1

2
(1) +
1

2
1 = 0
.
.
.
.
.
.
Or
y
n
=
_
1

2
n = 0
0 n > 0
(b)
w
n
= h
0
x
n
+h
1
x
n1
=
1

2
x
n

2
x
n1
w
0
=
1

2
1
1

2
0 =
1

2
p
r
a
v
e
n
d
r
a
Chapter 14 50
w
1
=
1

2
(1)
1

2
1 =

2
w
2
=
1

2
1
1

2
(1) =

2
w
3
=
1

2
(1)
1

2
1 =

2
.
.
.
.
.
.
Or
w
n
=
_
1

2
n = 0

2(1)
n
n > 0
(c) The lter response in part (a) corresponds to a low pass lter and the lter response in
part (b) corresponds to a high pass lter. After the initial transient the output of the low
pass lter is zero while the output of the high pass lter is an amplied version of the input.
Thus the input is a high pass signal.
Problem 4.
x
0
x
1
x
2
x
3
x
4
x
5
x
6
x
7
x
8
x
9
x
10

1 1 1 1 1 1 1 1 1 1 1
(a)
y
n
= h
0
x
n
+h
1
x
n1
=
1

2
x
n
+
1

2
x
n1
y
0
=
1

2
1 +
1

2
0 =
1

2
y
1
=
1

2
1 +
1

2
1 =

2
y
2
=
1

2
1 +
1

2
1 =

2
y
3
=
1

2
1 +
1

2
1 =

2
.
.
.
.
.
.
Or
y
n
=
_
1

2
n = 0

2 n > 0
p
r
a
v
e
n
d
r
a
Chapter 14 51
(b)
w
n
= h
0
x
n
+h
1
x
n1
=
1

2
x
n

2
x
n1
w
0
=
1

2
1
1

2
0 =
1

2
w
1
=
1

2
1
1

2
1 = 0
w
2
=
1

2
1
1

2
1 = 0
w
3
=
1

2
1
1

2
1 = 0
.
.
.
.
.
.
Or
w
n
=
_
1

2
n = 0
0 n > 0
(c) After the initial transient the output of the high pass lter is zero while the output of
the low pass lter is an amplied version of the input. Thus the input is a low pass signal.
p
r
a
v
e
n
d
r
a
Chapter 14 52
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
400
300
200
100
0
Normalized Frequency ( rad/sample)
P
h
a
s
e

(
d
e
g
r
e
e
s
)
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
8
6
4
2
0
2
4
Normalized Frequency ( rad/sample)
M
a
g
n
i
t
u
d
e

(
d
B
)
Figure 8: Magnitude and phase response of a lter with impulse response 0.5,0.5,0.5,-0.5.
p
r
a
v
e
n
d
r
a
Chapter 14 53
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0
50
100
150
200
Normalized Frequency ( rad/sample)
P
h
a
s
e

(
d
e
g
r
e
e
s
)
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
8
6
4
2
0
2
4
Normalized Frequency ( rad/sample)
M
a
g
n
i
t
u
d
e

(
d
B
)
Figure 9: Magnitude and phase response of a lter with impulse response -0.5,-0.5,0.5,-0.5.
p
r
a
v
e
n
d
r
a
Chapter 15 54
Chapter 15
Problem 1
(a)
f(t) = sin(2t) =
e
j2t
e
j2t
2j
The Fourier transform is given by
F() = T
_
e
j2t
e
j2t
2j
_
=
1
2j
T
_
e
j2t
_

1
2j
T
_
e
j2t
_
by linearity
=
1
2j
[( 2) ( + 2)]
(b)
f
1
(t) =
_
sin(2t) 2 t 2
0 otherwise
F
1
() =
_
2
2
sin(2t)e
jt
dt
=
_
2
2
_
e
j2t
e
j2t
2j
_
e
jt
dt
=
1
2j
__
2
2
e
j(w2)t
dt
_
2
2
e
j(w+2)t
dt
_
=
2
j
_
sin(2( 2))
2( 2)

sin(2( + 2))
2( + 2)
_
(c)
f
2
(t) =
_
(1 + cos(

2
t)) sin(2t) 2 t 2
0 otherwise
F
2
() =
_
2
2
(1 + cos(

2
t)) sin(2t)e
jt
dt
=
_
2
2
sin(2t)e
jt
dt +
_
2
2
cos(

2
t) sin(2t)e
jt
dt
The rst integral is identical to part (b). Evaluating the second integral we obtain
1
j
_
sin(2(w

2
2))
2(w

2
2)

sin(2(w

2
+ 2))
2(w

2
+ 2)
+
sin(2(w +

2
2))
2(w +

2
2)

sin(2(w +

2
+ 2))
2(w +

2
+ 2)
_
p
r
a
v
e
n
d
r
a
Chapter 15 55
(d) The magnitudes of F(), F
1
(), and F
2
() are plotted in Figure 10, Figure 11 and
Figure 12.
While both short term Fourier transforms broaden out the peak, the use of the rectangular
window introduces a number of extraneous peaks of signicant size. The use of the raised
cosine window considerably reduces the size of these extraneous peaks.
Problem 2.
f(t) =
_
1 + sin(2t) 0 t 1
sin(2t) otherwise
(t) =
_
1 0 t <
1
2
1
1
2
t < 1
c
j,k
=
_
f(t)
j,k
(t)dt
and

j,k
(t) = 2
j/2
(2
j
t k)
c
0,0
=
_
1
0
f(t)(t)dt
=
_ 1
2
0
(1 + sin(2t)
_
1
1
2
(1 + sin(2t))dt
=
1
2
[cos(0) + 2 cos(1) cos(2)]
c
0,1
=
_
2
1
f(t)(t 1)dt
=
_ 3
2
1
sin(2t)dt
_
2
3
2
sin(2t))dt
=
1
2
[cos(2) + 2 cos(3) cos(4)]
c
0,2
=
_
3
2
f(t)(t 2)dt
=
_ 5
2
2
sin(2t)dt
_
3
5
2
sin(2t))dt
=
1
2
[cos(4) + 2 cos(5) cos(6)]
We can see a pattern here:
p
r
a
v
e
n
d
r
a
Chapter 15 56
c
0,k
=
1
2
[cos(2k) + 2 cos(2k + 1) cos(2k + 2)]
To nd c
1,k
we use
c
1,k
=

2
_
(2t k)f(t)dt
So
c
1,0
=

2
_ 1
4
0
(1 + sin(2t))dt

2
_ 1
2
1
4
(1 + sin(2t))dt
=

2
2
_
cos(0) + 2 cos(
1
2
cos(1)
_
c
1,1
=

2
_ 3
4
1
2
(1 + sin(2t))dt

2
_
1
3
4
(1 + sin(2t))dt
=

2
2
_
cos(1) + 2 cos(
3
2
cos(2)
_
c
1,2
=

2
_ 5
4
1
sin(2t)dt

2
_ 3
2
5
4
sin(2t)dt
=

2
2
_
cos(2) + 2 cos(
5
2
cos(3)
_
Again, we can see the pattern:
c
1,k
=

2
2
_
cos(k) + cos(
2k + 1
2
) cos(k + 1)
_
Continuing for c
2,k
we nd that
c
2,k
=
2
2
_
cos(
k
2
) + cos(
2k + 1
4
) cos(
k + 1
2
)
_
Looking at the expressions for c
0,k
, c
1,k
and c
2,k
we can see a common pattern:
c
j,k
=
2
j
2
2
_
cos(
2k
2
j
) + 2 cos(
2k + 1
2
j
) cos(
2k + 2
2
j
)
_
Problem 3. We assume the following:
p
r
a
v
e
n
d
r
a
Chapter 15 57
We have a bit budget of 30 bits.
We can send the initial threshold value T
o
separately without eecting the bit budget.
We use the following codes
Zerotree root zr 00
Signicant positive sp 11
Signicant negative sn 01
Isolated zero iz 10
21 6 15 12
-6 3 6 3
3 -3 0 -3
3 0 0 0
T
0
= 16
Dominant Pass:
21 sp [11]
6 zr [00]
6 zr [00]
3 zr [00]
L
s
= 21
Renement Pass:
21 24 = 3 Correction term = 4 [0]
Second Pass:
T
0
= 8
Dominant Pass:
6 iz [10]
6 zr [00]
3 zr [00]
15 sp [11]
12 sp [11]
6 iz [10]
3 iz [10]
L
s
= 21, 15, 12
p
r
a
v
e
n
d
r
a
Chapter 15 58
Renement Pass:
21 20 = 1 Correction term = 2 [1]
15 12 = 3 Correction term = 2 [1]
12 12 = 0 Correction term = 2 [0]
Third Pass:
Pass:
T
0
= 4
Dominant Pass:
6 sp [11]
6 sn [01]
And we have used up thirty bits.
The transmitted bitstream is
110000000100000111110101101101
(b)
The received bitstream is
110000000100000111110101101101
and we begin with knowledge of the initial threshold and size of the image. For sake reference
lets represent the image as
a b e f
c d g h
i j m n
k l o p
T
o
= 16 and the rst codeword is 11 or sp. Therefore
a = 24
The next three codewords are 00 corresponding to a zerotree root. Therefore,
b = c = d = e = f = g = h = i = j = k = l = m = n = o = p = 0
There are no tree roots that are signicant so this is the end of the rst dominant pass. At
the end of this pass L
s
= a. In the renement pass we get a 0 corresponding to a negative
correction for a. Therefore,
a = 24 4 = 20
p
r
a
v
e
n
d
r
a
Chapter 15 59
The next bits are from the second dominant pass.
T
o
= 8 and 10 means an isolated zero. Thus b < T
o
but some of the descendants of b are
greater than T
o
.
The next codeword is 00 therefore, c is a zerotree root.
The next codeword is 00 therefore, d is a zerotree root.
As b was an isolated zero the next bits correspond to its descendants.
11 means e is signicant, therefore, e = 12
11 means f is signicant, therefore, f = 12
10 means g is insignicant, therefore, g = 0
10 means h is insignicant, therefore, h = 0
The signicance list now becomes L
s
= a, e, f
1 means a positive correction, therefore, a = 20 + 2 = 22.
1 means a positive correction, therefore, e = 12 + 2 = 14.
0 means a negative correction, therefore, f = 12 2 = 10.
Now we start the third dominant pass with T
o
= 4.
11 means b is signicant, therefore, b = 6
01 means c is signicant negative, therefore, c = 6.
and we have reached the end of the received bitstream.
The reconstructed image is as follows
22 6 14 10
-6 0 0 0
0 0 0 0
0 0 0 0
Problem 4. We assume the following:
We have a bit budget of 30 bits.
We can send the initial value of n separately without eecting the bit budget.
21 6 15 12
-6 3 6 3
3 -3 0 -3
3 0 0 0
p
r
a
v
e
n
d
r
a
Chapter 15 60
First Pass n = 4
LIP : (0, 0), (0, 1), (1, 0), (1, 1)
LIS : (0, 1)T, (1, 0)T, (1, 1)T
LSP :
First examine the contents of LIP.
(0, 0) 21 [21[ 16 Transmit 11 and move to LSP
(0, 1) 6 [6[ < 16 Transmit 0
(1, 0) 6 [ 6[ < 16 Transmit 0
(1, 1) 3 [3[ < 16 Transmit 0
Now examine the contents of LIS
(0, 1)Tis not signicant Transmit 0
(1, 0)Tis not signicant Transmit 0
(1, 1)Tis not signicant Transmit 0
LSP is empty.
Transmitted bits from the rst pass:
11000000
Second Pass n = 3
LIP : (0, 1), (1, 0), (1, 1)
LIS : (0, 1)T, (1, 0)T, (1, 1)T
LSP : (0, 0)
First examine the contents of LIP.
(0, 0) 21 [21[ 16 Transmit 11 and move to LSP
(0, 1) 6 [6[ < 8 Transmit 0
(1, 0) 6 [ 6[ < 8 Transmit 0
(1, 1) 3 [3[ < 8 Transmit 0
p
r
a
v
e
n
d
r
a
Chapter 15 61
Now examine the contents of LIS
(0, 1)Tis signicant Transmit 1
(0, 2) 15 [15[ 8 Transmit 11 and move to LSP
(0, 3) 12 [12[ 8 Transmit 11 and move to LSP
(1, 2) 6 [6[ < 8 Transmit 0 and move to LIP
(1, 3) 3 [3[ < 8 Transmit 0 and move to LIP
(1, 0)Tis not signicant Transmit 0
(1, 1)Tis not signicant Transmit 0
Finally, for the second pass, examine the contents of LSP
(0, 0) 21 n = 3 Transmit 0
Transmitted bits in the second pass.
0001111100000
Third Pass n = 2
LIP : (0, 1), (1, 0), (1, 1), (1, 2), (1, 3)
LIS : (1, 0)T, (1, 1)T
LSP : (0, 0), (0, 2), (0, 3)
First examine the contents of LIP.
(0, 1) 6 [6[ 4 Transmit 11 and move to LSP
(1, 0) 6 [ 6[ 4 Transmit 10 and move toLSP
(1, 1) 3 [3[ < 4 Transmit 0
(1, 2) 6 [6[ 4 Transmit 11 and move to LSP
(1, 3) 3 [3[ < 4 Transmit 0
Now examine the contents of LIS
(1, 0)Tis not signicant Transmit 0
And we have used up the thirty bits.
p
r
a
v
e
n
d
r
a
Chapter 15 62
The transmitted bits are
110000000001111100000111001100
(b)
The received bits are
110000000001111100000111001100
Begin with n = 4.
LIP : (0, 0), (0, 1), (1, 0), (1, 1)
LIS : (0, 1)T, (1, 0)T, (1, 1)T
LSP :
First LIP
11 (0, 0) 16, move to LSP
0 (0, 1) 0
0 (1, 0) 0
0 (1, 1) 0
Then to LIS
0 (0, 1)T is insignicant
0 (1, 0)T is insignicant
0 (1, 1)T is insignicant
LSP is empty.
Second pass, n = 3.
LIP : (0, 1), (1, 0), (1, 1)
LIS : (0, 1)T, (1, 0)T, (1, 1)T
LSP : (0, 0)
Begin with LIP
0 (0, 1) 0
0 (1, 0) 0
0 (1, 1) 0
p
r
a
v
e
n
d
r
a
Chapter 15 63
Then to LIS
1 (0, 1)T is signicant
11 (0, 2) 8 move to LSP
11 (0, 3) 8 move to LSP
0 (0, 2) 0 move to LIP
0 (0, 2) 0 move to LIP
0 (1, 0)T is insignicant
0 (1, 1)T is insignicant
(11)
On to LSP.
0 (0, 0) 16
Third pass, n = 2.
LIP : (0, 1), (1, 0), (1, 1), (1, 2), (1, 3)
LIS : (1, 0)T, (1, 1)T
LSP : (0, 0), (0, 2), (0, 3)
Begin with LIP
11 (0, 1) 4 move to LSP
10 (1, 0) 4 move to LSP
0 (0, 1) 0
11 (1, 2) 4 move to LSP
0 (1, 3) 0
Now to LIS
0 (1, 0)T is not signicant
We have reached the end of transmission. The reconstructed image is
16 4 8 8
-4 0 4 0
0 0 0 0
0 0 0 0
p
r
a
v
e
n
d
r
a
Chapter 15 64
8 6 4 2 0 2 4 6 8
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
|
F
(
w
)
|
w
Figure 10: Magnitude of the Fourier transform of the function sin(2t).
p
r
a
v
e
n
d
r
a
Chapter 15 65
8 6 4 2 0 2 4 6 8
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
|
F
1
(
w
)
|
w
Figure 11: Magnitude of the short term Fourier transform of the function sin(2t) with a
rectangular window.
p
r
a
v
e
n
d
r
a
Chapter 15 66
8 6 4 2 0 2 4 6 8
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
|
F
2
(
w
)
|
w
Figure 12: Magnitude of the short term Fourier transform of the function sin(2t) with a
raised cosine window.