Scribd
Upload
155 views

Assignment 1

This document contains an assignment for an information theory course. It lists 10 questions related to concepts like entropy, joint entropy, conditional entropy, and information. Students are instructed to only answer questions 4, 6, 9, and 10, which are about entropy calculations for outcomes of a World Series, run-length coding, the value of random questions, and limitations on dependence due to a bottleneck in a Markov chain. The answers are due on January 30, 2011.

Uploaded by

Udeeptha_Indik_6784
Copyright
© Attribution Non-Commercial (BY-NC)
Available Formats
Download as PDF, TXT or read online on Scribd
155 views

Assignment 1

This document contains an assignment for an information theory course. It lists 10 questions related to concepts like entropy, joint entropy, conditional entropy, and information. Students are instructed to only answer questions 4, 6, 9, and 10, which are about entropy calculations for outcomes of a World Series, run-length coding, the value of random questions, and limitations on dependence due to a bottleneck in a Markov chain. The answers are due on January 30, 2011.

Uploaded by

Udeeptha_Indik_6784
Copyright
© Attribution Non-Commercial (BY-NC)
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 3

THE UNIVERSITY OF MORATUWA Department of Electronic and Telecommunication Engineering EN5254-Information Technology Assignment # 1

The assignment is due on Sunday, January 30, 2011. You have to submit the answers only to questions 4, 6, 9 and 10.

1. Example of joint entropy Let p(x, y) be given by P X=0 X=1 Y=0 1/3 0 Y=1 1/6 1/2

Find (a) H(X), H(Y ). (b) H(X|Y ), H(Y |X). (c) H(X, Y ). (d) H(Y ) H(Y |X). (e) I(X; Y ). (f) Draw a Venn diagram for the quantities in (a) through (e). 2. Coin ips A fair coin is ipped until the rst head occurs. Let X denote the number of ips required. (a) Find the entropy H(X) in bits. The following expressions may be useful:

rn =
n=1

r , 1r

nrn =
n=1

r 2 . 1r

(b) A random variable X is drawn according to this distribution. Find an ecient sequence of yes-no questions of the form, Is X contained in the set S? Compare H(X) to the expected number of questions required to determine X. 3. Entropy of functions of a random variable Let X be a discrete random variable. Show that the entropy of a function of X is less than or equal to the entropy of X by justifying the following steps: H(X, g(X)) = H(X) + H(g(X)|X) = H(X); H(X, g(X)) = H(g(X)) + H(X|g(X)) H(g(X)). Thus H(g(X)) H(X). 1

4. World Series The World Series is a seven-game series that terminates as soon as either team wins four games. Let X be the random variable that represents the outcome of a World Series between teams A and B; possible values of X are AAAA, BABABAB, and BBBAAAA. Let Y be the number of games played, which ranges from 4 to 7. Assuming that A and B are equally matched and that the games are independent, calculate H(X), H(Y ), H(Y |X), and H(X|Y ). 5. Average entropy Let H(p) = p log2 p (1 p) log2 (1 p) be the binary entropy function. (a) Evaluate H(1/4) using the fact that log2 3 1.584. Hint: Consider an experiment with four equally likely outcomes, one of which is more interesting than the others. (b) Calculate the average entropy H(p) when the probability p is chosen uniformly in the range 0 p 1. (c) Calculate the average entropy H(p1 , p2 , p3 ) where (p1 , p2 , p3 ) is a uniformly distributed probability vector. Generalize to dimension n. 6. Run-length coding Let X1 , X2 , . . . , Xn be (possibly dependent) binary random variables. Suppose that one calculates the run lengths R = (R1 , R2 , . . .) of this sequence (in order as they occur). For example, the sequence X = 000 11 00 1 00 yields run lengths R = (3, 2, 2, 1, 2). Compare H(X1 , X2 , . . . , Xn ), H(R), and H(Xn , R). Show all equalities and inequalities, and bound all the dierences. 7. Entropy of a sum Let X and Y be random variables that take on values x1 , x2 , . . . , xr and y1 , y2 , . . . , ys , respectively. Let Z = X + Y . (a) Show that H(Z|X) = H(Y |X). Argue that if X, Y are independent, then H(Y ) H(Z) and H(X) H(Z). Thus the addition of independent random variables adds uncertainty. (b) Give an example (of necessarily dependent random variables) in which H(X) > H(Z) and H(Y ) > H(Z). (c) Under what conditions does H(Z) = H(X) + H(Y )? 8. A measure of correlation Let X1 and X2 be identically distributed, but not necessarily independent. Let =1 (a) Show = I(X1 ; X2 )/H(X1 ). (b) Show 0 1. (c) When is = 0? (d) When is = 1? H(X2 |X1 ) . H(X1 )

9. Random questions One wishes to identify a random object X p(x). A question Q r(q) is asked at random according to r(q). This results in a deterministic answer A = A(x, q) {a1 , a2 , . . .}. Suppose that X and Q are independent. Then I(X; Q, A) is the uncertainty in X removed by the question - answer (Q, A). (a) Show that I(X; Q, A) = H(A|Q). Interpret. (b) Now suppose that two i.i.d. questions Q1 , Q2 , r(q) are asked, eliciting answers A1 and A2 . Show that two questions are less valuable than twice a single question in the sense that I(X; Q1 , A1 , Q2 , A2 ) 2I(X; Q1 , A1 ). 10. Bottleneck Suppose a (non-stationary) Markov chain starts in one of n states, necks down to k < n states, and then fans back to m > k states. Thus X1 X2 X3 , X1 {1, 2, . . . , n}, X2 {1, 2, . . . , k}, X3 {1, 2, . . . , m}. (a) Show that the dependence of X1 and X3 is limited by the bottleneck by proving that I(X1 ; X3 ) log k. (b) Evaluate I(X1 ; X3 ) for k = 1, and conclude that no dependence can survive such a bottleneck.

You might also like