Homework 6 Solutions

7.19 Capacity of the carrier pigeon channel. Consider a commander of an army besieged a fort
for whom the only means of communication to his allies is a set of carrier pigeons. Assume that each
carrier pigeon can carry one letter (8 bits), and assume that pigeons are released once every 5 minutes,
and that each pigeon takes exactly 3 minutes to reach its destination.
(a) Assuming all the pigeons reach safely, what is the capacity of this link in bits/hour?
(b) Now assume that the enemies try to shoot down the pigeons, and that they manage to hit a fraction
α of them. Since the pigeons are sent at a constant rate, the receiver knows when the pigeons are
missing. What is the capacity of this link?
(c) Now assume that the enemy is more cunning, and every time they shoot down a pigeon, they send
out a dummy pigeon carrying a random letter (chosen uniformly from all 8-bit letters). What is the
capacity of this link in bits/hour?
Solution:
(a) The channel sends 8 bits every 5 minutes, or 96 bits/hour.
(b) This is equivalent of an erasure channel with an input alphabet of 8 bit symbols, i.e., 256 different
symbols. For any symbols sent, a fraction α of them are received as an erasure. We would expect that
the capacity of this channel is (1 − α)8 bits/pigeon. We will justify it more formally by mimicking the
derivation for the binary erasure channel.
Consider an erasure channel with 256 symbol inputs and 257 symbol output - the extra symbol is
the erasure symbol, which occurs with probability α. Then
I(X; Y ) = H(Y ) − H(Y |X) = H(Y ) − H(α)
since the probability of erasure is independent of the input.
However, we cannot get H(Y ) to attain its maximum value, i.e., log 257, since the probability of the
erasure channel is α independent of our input distribution. the probability of the erasure channel is α
independent of our input distribution. However, if we let E be the erasure event, then
H(Y ) = H(Y, E) = H(E) + H(Y |E) = H(α) + α × 0 + (1 − α)H(Y |E = 0) (1)
and we can maximize H(Y ) by maximizing H(Y |E = 0). However, H(Y |E = 0) is just the entropy
of the input distribution, and this is maximized by the uniform. Thus the maximum value of H(Y ) is
H(α) + (1 − α) log 256, and the capacity of this channel is (1 − α) log 256 bits/pigeon, or (1 − α)96
bits/hour, as we might have expected from intuitive arguments.
(c) In this case, we have a symmetric channel with 256 inputs and 256 oupts. With probability (1 − α) +
α/256, the output symbol is equal to the input, and with probability α/256, it is transformed to any of the
other 255 symbols. This channel is symmetric in the sense of Section 7.2, and therefore the capacity of
the channel is
C = log |Y| − H(r)
= log 256 − H(1 − α + α/256, α/256, . . . , α/256)
255 255
= 8 − H(1 − α) − αH(1/255, . . . , 1/255)
256 256
255 255
= 8 − H(1 − α) − α log 255 (2)
256 256
We have to multiply this by 12 to get the capacity in bits/hour.

7.21 Tall, fat people. Suppose that average height of people in a room is 5 feet. Suppose the
average weight is 100 lbs.

1
This study source was downloaded by 100000887045271 from CourseHero.com on 06-07-2024 21:08:26 GMT -05:00

https://www.coursehero.com/file/56203116/Solution-Chapter7-HW6pdf/
 (a) Argue that no more than 1/3 of the population is 15 feet tall.
(b) Find an upper bound on the fraction of 300 lb, 10 footers in the room.
Solution:
(a) The average height of the individuals in the population is 5 feet. So n1
P
hi = 5 where n is the
population size and hi is the height of the ith person. If more than 1/3 of the population is at least 15
feet tall, then the average will be greater than 31 × 15 = 5 feet since each person is at least 0 feet tall.
Thus no more than 13 of the population is 15 feet tall.
(b) By the same reasoning as in part (a), at most 12 of the population is 10 feet tall and at most 31 of the
population weights 300 lbs. Therefore at most 13 are both 10 feet tall and weight 300 lbs.

7.25 Bottleneck channel. Suppose a signal X ∈ X = {1, 2, . . . , m} goes through an intervening

transition X → V → Y , where x = {1, 2, . . . , m}, y = {1, 2, . . . , m}, and v = {1, 2, . . . , k}. Here

X V Y

p(v|x) p(y|v)

P
p(v|x) and p(y|v) are arbitrary and the channel has transition probability p(y|x) = v p(v|x)p(y|v).
Show C ≤ log k .
Solution:
The capacity of the cascade of channels is C = maxp(x) I(X; Y ). By the data processing inequality,

I(X; Y ) ≤ I(V ; Y ) = H(V ) − H(V |Y ) ≤ H(V ) ≤ log k.

7.29 Binary multiplier channel.

(a) Consider the discrete memoryless channel Y = XZ where X and Z are independent binary random
variables that take on values 0 and 1. Let P (Z = 1) = α. Find the capacity of this channel and the
maximizing distribution X .
(b) Now suppose the receiver can observe Z as well as Y . What is the capacity?
Solution:
(a) Let P (X = 1) = p. Then P (Y = 1) = P (X = 1)P (Z = 1) = αp

I(X; Y ) = H(Y ) − H(Y |X)

= H(Y ) − P (X = 1)H(Z)
= H(αp) − pH(α) (3)
H(α)
We find that p∗ = 1
α(2H(α)/α +1)
maximizes I(X; Y ). The capacity is calculated to be log(2 α + 1) −
H(α)
α
(b) Let P (X = 1) = p. Then

I(X; Y, Z) = I(X; Z) + I(X; Y |Z)

= H(Y |Z) − H(Y |X, Z)
= H(Y |Z) = αH(α) (4)

The expression is maximized for p = 1/2, resulting in C = α. Intuitively, we can only get X through
when Z is 1, which happens α of the time.

2
This study source was downloaded by 100000887045271 from CourseHero.com on 06-07-2024 21:08:26 GMT -05:00

https://www.coursehero.com/file/56203116/Solution-Chapter7-HW6pdf/
 7.36 Channel with memory. Consider the discrete memoryless channel Yi = Zi Xi with input
alphabet Xi ∈ {−1, 1}.
(a) What is the capacity of this channel when {Zi } is i.i.d. with

1, p = 0.5
Zi = ? (5)
−1, p = 0.5
Now consider the channel with memory. Before transmission begins, Z is randomly chosen and fixed
for all time. Thus Yi = ZXi .
(b) What is the capacity if

1, p = 0.5
Z= ? (6)
−1, p = 0.5
Solution:
(a) This is a BSC with cross over probability 0.5, so C = 1 − H(p) = 0.
(b) Consider the coding scheme of sending X n = (1, b1 , b2 , . . . , bn−1 ) where the first symbol is always a
zero and the rest of the n − 1 symbols are ±1 bits. For the first symbol Y1 = Z , so the receiver knows Z
exactly. After that the receiver can recover the remaining bits error free. So in n symbol transmissions
n bits are sent, for a rate R = n−1 n → 1. The capacity C is bounded by log |X | = 1, therefore the
capacity is 1 bit per symbol.

7.37 Joint typicality.

Let (Xi , Yi , Zi ) be i.i.d. according to p(x, y, z). We will say that (xn , y n , z n ) is jointly typical (written
(n)
(xn , y n , z n ) ∈ A ) if
• p(xn ) ∈ 2−n(H(X)±)
• p(y n ) ∈ 2−n(H(Y )±)
• p(z n ) ∈ 2−n(H(Z)±)
• p(xn , y n ) ∈ 2−n(H(X,Y )±)
• p(xn , z n ) ∈ 2−n(H(X,Z)±)
• p(y n , z n ) ∈ 2−n(H(Y,Z)±)
• p(xn , y n , z n ) ∈ 2−n(H(X,Y,Z)±)
Now suppose (X̃ n , Ỹ n , Z̃ n ) is drawn according to p(xn )p(y n )p(z n n
). Thus X̃ n , Ỹ n , Z̃ n have
o the same
(n)
marginals as p(xn , y n , z n ) but independent. Find (bounds on) P r (X̃ n , Ỹ n , Z̃ n ) ∈ A in terms of
the entropies H(X), H(Y ), H(Z), H(X, Y ), H(X, Z), H(Y, Z) and H(X, Y, Z).
Solution:
n o X
P r (X̃ n , Ỹ n , Z̃ n ) ∈ A(n)
 = p(xn )p(y n )p(z n )
(n)
(xn ,y n ,z n )∈A
X
≤ 2−n(H(X)+H(Y )+H(Z)−3)
(n)
(xn ,y n ,z n )∈A
−n(H(X)+H(Y )+H(Z)−3)
≤ |A(n)
 |2
≤ 2n(H(X,Y,Z)+) 2−n(H(X)+H(Y )+H(Z)−3)
≤ 2n(H(X,Y,Z)−H(X)−H(Y )−H(Z)−4) (7)

3
This study source was downloaded by 100000887045271 from CourseHero.com on 06-07-2024 21:08:26 GMT -05:00

https://www.coursehero.com/file/56203116/Solution-Chapter7-HW6pdf/
 n o X
P r (X̃ n , Ỹ n , Z̃ n ) ∈ A(n)
 = p(xn )p(y n )p(z n )
(n)
(xn ,y n ,z n )∈A
X
≥ 2−n(H(X)+H(Y )+H(Z)+3)
(n)
(xn ,y n ,z n )∈A
−n(H(X)+H(Y )+H(Z)+3)
≥ |A(n)
 |2
≥ (1 − )2n(H(X,Y,Z)−) 2−n(H(X)+H(Y )+H(Z)+3)
≥ (1 − )2n(H(X,Y,Z)−H(X)−H(Y )−H(Z)+4) (8)

Note that the upper bound is true for all n, but the lower bound only hold for n large.

4
This study source was downloaded by 100000887045271 from CourseHero.com on 06-07-2024 21:08:26 GMT -05:00

https://www.coursehero.com/file/56203116/Solution-Chapter7-HW6pdf/
Powered by TCPDF (www.tcpdf.org)