EE376A/STATS376A Information Theory Lecture 3 - 01/16/2018

Lecture 3: Entropy, Relative Entropy, and Mutual Information

Lecturer: Tsachy Weissman Scribe: Yicheng An, Melody Guan, Jacob Rebec, John Sholar

In this lecture, we will introduce certain key measures of information, that play crucial roles in theoretical
and operational characterizations throughout the course. These include the entropy, the mutual information,
and the relative entropy. We will also exhibit some key properties exhibited by these information measures.

1 Notation
A quick summary of the notation
1. Discrete Random Variable: U
2. Alphabet: U = {u1 , u2 , ..., uM } (An alphabet of size M)
3. Specific Value: u, u1 , etc.
For discrete random variables, we will write (interchangeably) P (U = u), PU (u) or most often just, p(u)
Similarly, for a pair of random variables X, Y we write P (X = x | Y = y), PX|Y (x | y) or p(x | y)

2 Entropy
Definition 1. “Surprise” Function:
1
S(u) , log (1)
p(u)
A lower probability of u translates to a greater “surprise” that it occurs.
Note here that we use log to mean log2 by default, rather than the natural log ln, as is typical in some other
contexts. This is true throughout these notes: log is assumed to be log2 unless otherwise indicated.
Definition 2. Entropy: Let U a discrete random variable taking values in alphabet U. The entropy of U
is given by:
  
1   X
H(U ) , E[S(U )] = E log = E − log (p(U )) = − p(u) log p(u) (2)
p(U ) u

Where U represents all u values possible to the variable.

The entropy is a property of the underlying distribution PU (u), u ∈ U that measures the amount of
randomness or surprise in the random variable.
Lemma 3. Jensen’s Inequality: Let Q denote a function on a random variable X. Jensen’s inequality
states that ∀Q that are convex:

E[Q(X)] ≥ Q(E[X]). (3)

Further, if Q is strictly convex, equality holds iff X is deterministic.

Conversely, if Q is a concave function, then

E[Q(X)] ≤ Q(E[X]). (4)

1
 Proof:

If Q is a convex function then its graph {(X, Q(X)) : X ∈ R} can be seen as an upper bound on the set
of affine functions that lie below it. Written more concretely,

Q(X) = sup L(X)

L∈L

where
L = {L : L(u) = au + b ≤ Q(u) for all − ∞ < u < ∞}
Thus, by linearity:

E[Q(X)] = E[sup L(x)] (5)

L∈L
≥ sup E[L(X)] (6)
L∈L
= sup L(E[X]) (7)
L∈L
= Q(E[X]) (8)

(6) holds because of the monotonicity of E.

The same argument for concave functions can be done using the infimum instead of the supremum.

2.1 Properties of Entropy

Suppose U = {u1 , u2 , ..., uM }
1
1. H(U ) ≤ log M , with equality iff U is uniformly distributed i.e. p(u) = M ∀u
Proof:
 
1
H(U ) = E log (9)
p(U )
 
1
≤ log E (By Jensen’s inequality because log is concave) (10)
p(U )
X 1
= log p(u) · (11)
u
p(u)
= log M. (12)
1 1
Equality by Jensen’s inequality iff p(U ) is deterministic, iff p(u) = M ∀u ∈ U

2. H(U ) ≥ 0, with equality iff U is deterministic.

Proof:

 
1 1
H(U ) = E log ≥ 0 because log ≥0 (13)
p(U ) p(U )
1
The equality occurs iff log p(u) = 0 with probability 1 so U must be deterministic.

2
3. For a PMF q define   X
1 1
Hq (U ) , E log = p(u) log . (14)
q(U ) q(u)
u∈U

Then:
H(U ) ≤ Hq (U ), (15)
with equality iff q = p.
Proof:
   
1 1
H(U ) − Hq (U ) = E log − E log (16)
p(u) q(u)
 
q(u)
H(U ) − Hq (U ) = E log (17)
p(u)
 
q(u)
≤ log E (18)
p(u)
X q(u)
= log p(u) (19)
p(u)
u∈U
X
= log q(u) (20)
u∈U
= log 1 (21)
=0 (22)

Thus,
H(U ) − Hq (U ) ≤ 0.
q(u)
Equality only holds when p(u) is deterministic, which occurs when q = p (distributions are identical).

Definition 4. Relative Entropy1 An measure of distance between probability distributions is relative

entropy:  
X p(u) p(u)
D(p k q) , p(u) log = E log (23)
q(u) q(u)
u∈U

Note that by property 3, the relative entropy is always greater than or equal to 0, with equality iff
q = p. For now, relative entropy can be thought of as a measure of discrepancy between two probability
distributions. We will soon see that it is central to information theory.

4. If X1 , X2 , . . . , Xn are independent random variables, then

n
X
H(X1 , X2 , . . . , Xn ) = H(Xi ) (24)
i=1

Note: H(X1 , X2 , . . . , Xn ) is called the joint entropy of X1 , X2 , . . . , Xn .

1 Some students may be familiar with relative entropy as Kullback-Leibler (KL) divergence

3
 Proof:

H(X1 , X2 , . . . , Xn ) = E [− log p(x1 , x2 , . . . , xn )] (25)

" n
#
Y
= E − log p(xi ) (26)
i=1
" n
#
X
=E − log p(xi ) (27)
i=1
n
X
= E [− log p(xi )] (28)
i=1
Xn
= H(Xi ). (29)
i=1

5.
Definition 5. Conditional Entropy of X given Y
 
1
H(X | Y ) , E log (30)
p(X | Y )
X 1
= p(x, y) log (31)
x,y
p(x | y)
" #
X X 1
= p(y) p(x | y) log (32)
y x
p(x | y)
X
= p(y)H(X | Y = y). (33)
y

H(X | Y ) ≤ H(X) with equality iff X and Y are independent.

Proof:
   
1 1
H(X) − H(X | Y ) = E log − E log (34)
p(X) p(X|Y )
 
p(X | Y ) p(Y )
= E log (35)
p(X) p(Y )
 
p(X, Y )
= E log (36)
p(X)p(Y )
X P (x, y)
= P (x, y) log (37)
x,y
P (x)P (y)
= D(Px,y k Px × Py ) (38)
≥0 (39)

D(Px,y k Px ×Py ) ≥ 0 because relative entropy can never be negative. Equality holds iff Px,y ≡ Px ×Py ,
(X and Y are independent).

4
 6. Chain Rule:
 1 
H(X, Y ) , E log (40)
P (X, Y )
 1
= E log ] (41)
P (Y )P (X | Y )
= H(Y ) + H(X | Y ) (42)

We can take this one step further with (5):

H(X, Y ) = H(Y ) + H(X | Y ) ≤ H(X) + H(Y ), (43)

with equality holding iff X and Y are independent.

Definition 6. Mutual information between X and Y

We now define the mutual information between random variables X and Y distributed according to
the joint PMF P (x, y):

I(X; Y ) , D(Px,y k Px × Py ) (44)

= H(X) − H(X|Y ) (45)
= H(X) + H(Y ) − H(X, Y ) (46)

(May find any of these in the literature) The mutual information tells how helpful one variable is at
reducing uncertainty in the other.
Note: while relative entropy is not symmetric, mutual information is.

3 Exercises
1. “Data processing decreases entropy” (note that this statement only applies to deterministic functions)
Y = f (X) ⇒ H(Y ) ≤ H(X) with equality when f is one-to-one.
Note: Proof is part of homework 1.
2. “Data processing on side information increases entropy”
Y = f (X) ⇒ H(Z|X) ≤ H(Z|Y )
True more generally:
whenever Y − X − Z (Markov Relation), i.e., p(Z|X, Y ) = p(Z|X), then H(Z|X) ≤ H(Z|Y )
Note: Proof is part of homework 1.
3.
Definition 7. Conditional mutual information

I(X; Y |Z) , H(X|Z) − H(X|Y, Z) (47)

Show that: I(X; Y1 , Y2 ) = I(X; Y1 ) + I(X; Y2 |Y1 )

Proof:

I(X; Y1 , Y2 ) = H(X) − H(X|Y1 , Y2 ) (48)

= H(X) − H(X|Y1 , Y2 ) − H(X|Y1 ) + H(X|Y1 ) (49)
= [H(X) − H(X|Y1 )] + [H(X|Y1 ) − H(X|Y1 , Y2 )] (50)
= I(X; Y1 ) + I(X; Y2 |Y1 ) (51)