EE 376A: Information Theory

Lecture Notes

Prof. Tsachy Weissman

TA: Idoia Ochoa, Kedar Tatwawadi

January 6, 2016
Contents

1 Introduction 1
1.1 Lossless Compression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Channel Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Lossy Compression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2 Entropy, Relative Entropy, and Mutual Information 6

2.1 Entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2 Conditional and Joint Entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3 Mutual Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3 Asymptotic Equipartition Properties 12

3.1 Asymptotic Equipartition Property (AEP) . . . . . . . . . . . . . . . . . . . . . . . . 12
3.2 Fixed Length (Near) Lossless Compression . . . . . . . . . . . . . . . . . . . . . . . . 15

4 Lossless Compression 18
4.1 Uniquely decodable codes and prefix codes . . . . . . . . . . . . . . . . . . . . . . . . 18
4.2 Prefix code for dyadic distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.3 Shannon codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.4 Average codelength bound for uniquely decodable codes . . . . . . . . . . . . . . . . 22
4.5 Huffman Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.6 Optimality of Huffman codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

5 Communication and Channel Capacity 28

5.1 The Communication Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
5.2 Channel Capacity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
5.2.1 Channel Capacity of various discrete channels . . . . . . . . . . . . . . . . . . 30
5.2.2 Recap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5.3 Information Measures for Continuous Random Variables . . . . . . . . . . . . . . . . 32
5.3.1 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5.3.2 Gaussian Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
5.4 Channel Capacity of the AWGN Channel (Additive White Gaussian Noise) . . . . . 34
5.4.1 Channel Coding Theorem for this Setting . . . . . . . . . . . . . . . . . . . . 35
5.4.2 An Aside: Cost Constraint . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
5.4.3 The Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
5.4.4 Rough Geometric Interpretation (Picture) . . . . . . . . . . . . . . . . . . . . 36

i
 5.5 Joint Asymptotic Equipartition Property (AEP) . . . . . . . . . . . . . . . . . . . . 38
5.5.1 Set of Jointly Typical Sequences . . . . . . . . . . . . . . . . . . . . . . . . . 38
5.6 Direct Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5.7 Fano’s Inequality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
5.8 Converse Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5.9 Some Notes on the Direct and Converse Theorems . . . . . . . . . . . . . . . . . . . 42
5.9.1 Communication with Feedback: Xi (J, Y i−1 ) . . . . . . . . . . . . . . . . . . . 42
5.9.2 Practical Schemes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
5.9.3 Pe vs. Pmax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

6 Method of Types 45
6.1 Method of Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
6.1.1 Recap on Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
6.2 A Version of Sanov’s Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

7 Conditional and Joint Typicality 52

7.1 Typical Set (again) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
7.2 δ–strongly typical set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
7.3 δ–jointly typical set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
7.4 δ–conditional typicality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
7.5 Encoding – Decoding Schemes for Sending Messages . . . . . . . . . . . . . . . . . . 56
7.6 Joint Typicality Lemma . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

8 Lossy Compression & Rate Distortion Theory 59

8.1 Definitions and main result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
8.2 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
8.3 Proof of Direct Part R(D) ≤ R(I) (D) . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
8.3.1 An Equivalent Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
8.3.2 Two Useful Lemmas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
8.3.3 Proof of the Equivalent Statement . . . . . . . . . . . . . . . . . . . . . . . . 63
8.4 Proof of the converse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
8.5 Geometric Interpretation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

9 Joint Source Channel Coding 68

9.1 Joint Source Channel Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
9.2 Source – Channel Separation Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . 69
9.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

EE376A COURSE NOTES ii

Chapter 1

Introduction

Information theory is the science of operations on data such as compression, storage, and com-
munication. It is among the few disciplines fortunate to have a precise date of birth: 1948, with
the publication of Claude E. Shannon’s paper entitled A Mathematical Theory of Communication.
Shannon’s Information theory had a profound impact on our understanding of the concepts in
communication.
In this introductory chapter, we will look at a few representative examples which try to give
a flavour of the problems which can be addressed using information theory. However note that,
communication theory, is just one of the numerous fields which had a dramatic shift in the under-
standing due to information theory.

1.1 Lossless Compression

Consider a source that emits a sequence of symbols U1 , U2 , . . . with Ui ∈ {a, b, c}. The Ui are i.i.d
(independently and identically distributed) according to the probability mass function

P (U = a) = 0.7
P (U = b) = P (U = c) = 0.15

Our task is to encode the source sequence into binary bits (1s and 0s). How should we do so?
The naive way is to use two bits to represent each symbol, since there are three possible
symbols. For example, we can use 00 to represent a, 01 to represent b and 10 to represent c. This
scheme has an expected codeword length of 2 bits per source symbol. Can we do better? One
natural improvement is to try to use fewer bits to represent symbols that appear more often. For
example, we can use the single bit 0 to represent a since a is the most common symbol, and 10 to
represent b and 11 to represent c since they are less common. Note that this code satisfies the prefix
condition, meaning no codeword is the prefix of another codeword, which allows us to decode a
message consisting of stream of bits without any ambiguity. Thus, if we see the encoded sequence,
001101001101011, we can quickly decode it as follows:

0 |{z}
|{z} 0 |{z}
11 |{z}
0 |{z}
10 |{z}
0 |{z}
11 |{z}
0 |{z}
10 |{z}
11
a a c a b a c a b c

1
 If we use this encoding scheme, then L, which denotes the expected number of bits we use per
source symbol, is

L = 1 × P (U = a) + 2 × (P (U = b) + P (U = c)) = 1 × 0.7 + 2 × (0.15 + 0.15) = 1.3.

This is a significant improvement over our first encoding scheme. But can we do even better? A
possible improvement is to encode two values at a time instead of encoding each value individually.
For example, the following table shows all the possibilities we can get if we look at 2 values, and
their respective probabilities (listed in order of most to least likely pairs). A possible prefix coding
scheme is also given.

source symbols probability encoding

aa 0.49 0
ab 0.105 100
ac 0.105 111
ba 0.105 101
ca 0.105 1100
bb 0.0225 110100
bc 0.0225 110101
cb 0.0225 110110
cc 0.0225 110111

Note that this scheme satisfies the two important properties: 1) the prefix condition and 2)
more common source symbol pairs have shorter codewords. If we use the above encoding scheme,
then the expected number of bits used per source symbol is

L = 0.5 × (0.49 × 1 + 0.105 × 4 + 0.105 × 3 × 3 + 0.0225 × 6 × 4) = 1.1975.

It can be proven that if we are to encode 2 values at a time, the above encoding scheme achieves
the lowest average number of bits per value (*wink* Huffman encoding *wink*).
Generalizing the above idea, we can consider a family of encoding schemes indexed by an integer
k. Given an integer k, we can encode k values at a time with a scheme that satisfies the prefix
condition and assigns shorter codewords to more common symbols. Under some optimal encoding
scheme, it seems reasonable that the expected number of bits per value will decrease as k increases.
We may ask, what is the best we can do? Is there a lower bound on L? Shannon proved that
given any such source, the best we can do is H(U ), which is called the Entropy of the source . By
definition, the source entropy is
X 1
H(U ) , p(u) log2 (1.1)
u∈U
p(u)

Thus, Shannon proved the following statement1 ,

Theorem 1. ∀ families of encoding schemes, the average codeword length, L ≥ H(U ).

1
Note that the statements of the theorems here will be informal; they will be made rigorous in later lectures.

EE376A COURSE NOTES 2

 For our example, the lower bound is thus
1 1
0.7 × log2
+ 2 × 0.15 × log2 ≈ 1.181
0.7 0.15
We will also show an upper bound, namely,

Theorem 2. ∀ε > 0, ∃ family of schemes, such that the average codeword length, L ≤ H(U ) + ε.

1.2 Channel Coding

Suppose we have a source that emits a stream of bits U1 , U2 , . . . The Ui ∈ {0, 1} are i.i.d. Bernoulli
random variables with parameter 0.5, or fair coin flips.
We want to transmit the bits Ui through a channel. Suppose the bits that are transmitted are
X1 , X2 , . . . The channel is noisy and flips each bit with probability q < 1/2. Therefore, if Y1 , Y2 , . . .
is the sequence of bits that we receive, we have

Yi = Xi ⊕ Wi , Wi ∼ Ber(q)
where ⊕ is the XOR operator.
We want to know how accurate we can be when transmitting the bits. The simplest approach
is to let Xi = Ui , and to decode the received bit by assuming Yi = Xi . Let pe be the probability of
error per source bit. Then in this case, pe = q < 1/2.
Can we decrease pe ? One approach may be to use repetition encoding, i.e., send each bit k
times for some k, and then decode the received bit as the value that appeared most among the k
received symbols. For example, if k = 3, then pe is simply the probability that the channel flipped
2 or more of the bits, which is

pe = 3(1 − q)q 2 + q 3 < q.

However, we need to send 3 times as many bits. To quantify this, we introduce the notion of bit
rate, denoted by R, which is the ratio of the number of bits sent to the units of channel space
used. For this scheme, our bit rate is 31 , whereas our bit rate in the previous example was 1.
Generalizing the above example, we see that as we increase k, our error rate pe will tend to 0,
but our bit rate R (which is 1/k) tends to 0 as well. Is there some scheme that has a significant
positive bit rate and yet allows us to get reliable communication (error rate tends to 0)? Again,
Shannon provides the answer.

Theorem 3. ∃C > 0 and ∃ family of schemes with R < C satisfying pe → 0.

In fact, the largest such C is known as the channel capacity of a channel, which represents
the largest bit rate ( the largest C ) that still allows for reliable communication. This was a very
significant and a startling revelation for the world of communication, as it was thought that zero
error probability is not achievable with a non-zero bit rate.
As examples, we will consider the channel capacity of the binary symmetric channel and the
additive white gaussian noise channel.

EE376A COURSE NOTES 3

Binary Symmetric Channel
The channel capacity of a binary symmetric channel with bit-flipping probability q is

1 − H(X), X ∼ Ber(q). (1.2)

Moreover, if we let X ∼ Ber(q) and Y ∼ Ber(pe ), we will see that a bit rate R such that

1 − H(X)
R< , (1.3)
1 − H(Y )

is achievable, whereas
1 − H(X)
R> , (1.4)
1 − H(Y )
is unachievable.

Additive White Gaussian Noise (AWGN) Channel

Suppose we have a source that emits a sequence of bits U1 , U2 , . . . , UN , where each Ui is i.i.d.
according to U ∼ Ber( 12 ).
However, we can only transmit real numbers X1 , X2 , . . . , Xn . Also, the channel contains some
noise. Specifically, if Y1 , Y2 , . . . , Yn is the sequence of values we receive, we have

Yi = Xi + Ni , Ni ∼ N (0, σ 2 )
N
The rate of transmission is the ratio (which is the ratio of the number of source bits to the
n
number of uses of the channel). We want to develop a scheme so that we can reliably reconstruct
Ui from the given Yi . One way, if we have no usage power constraint, is to make Xi a large positive
value if Ui = 1 and Xi a large negative value if Ui = 0. In this manner, the noise from Ni will be
trivial relative to the signal magnitude, and will not impact reconstruction too much. However,
suppose there is an additional constraint on the average power of the transmitted signal, such that
we require
n
1X
X 2 ≤ p,
n i=1 i
for a given value p. In fact, we will see that
 
1 p
Theorem 4. If the rate of transmission is < 2 log2 1 + σ2
, then ∃ family of schemes that com-
 
1 p
municate reliably. And if the rate of transmission is > 2 log2 1 + σ2
, then there is no family of
schemes which communicates reliably.
p
The ratio is referred to as the signal-to-noise ratio (SNR).
σ2

EE376A COURSE NOTES 4

1.3 Lossy Compression
Suppose we have a source that emits a sequence of values U1 , U2 , . . ., where the Ui is i.i.d random
variables. according to U ∼ N (0, σ 2 ). Suppose we want to encode the source using one bit per
value. Since we are representing continuous variables with discrete bits, we are employing lossy
compression. Can we come up with a scheme that reconstructs the original signal as accurately as
possible, based on the bits sent?
Let B1 , B2 , . . . be the bits sent. One natural scheme is to set
(
1 if Ui ≥ 0
Bi =
0 if Ui < 0.

After receiving the bits, let V1 , V2 , . . . be the reconstructed values. The distortion of the scheme
is defined as

D , E[(Ui − Vi )2 ] (1.5)
The optimal estimation rule for minimum mean squared error is the conditional expectation.
Therefore, to minimize distortion, we should reconstruct via Vi = E[Ui | Bi ]. This results in

D = E[(Ui − Vi )2 ]
= V ar(Ui | Bi )
= 0.5 × V ar(Ui | Bi = 1) + 0.5 × V ar(Ui | Bi = 0) (because U is symmetric)
= V ar(Ui | Bi = 1)
= E[Ui2 | Bi = 1] − (E[Ui | Bi = 1])2
2
 
= σ2 1 −
π
2
≈ 0.363σ .

We will see in fact that 0.363σ 2 can be improved considerably, as such:

Theorem 5. Consider a Gaussian memoryless source with mean µ and variance σ 2 . ∀ε > 0, ∃
family of schemes such that D ≤ σ 2 /4 + ε. Moreover, ∀ families of schemes, D ≥ σ 2 /4.

As we saw, the few examples signify the usefulness of information theory to the field of communi-
cations. In the next few chapters, we will try to build the mathematical foundations for the theory
of information theory, which will make it much more convenient for us to use them later on.

EE376A COURSE NOTES 5

Chapter 2

Entropy, Relative Entropy, and

Mutual Information

In this chapter ,we will introduce certain key measures of information, that play crucial roles in
theoretical and operational characterizations throughout the course. These include the entropy, the
mutual information, and the relative entropy. We will also exhibit some key properties exhibited
by these information measures.

Notation
A quick summary of the notation

1. Random Variables (objects): used more “loosely", i.e. X, Y, U, V

2. Alphabets: X , Y, U, V

3. Specific Values: x, y, u, v

For discrete random variable (object), U has p.m.f: PU (u) , P (U = u). Often, we’ll just write
p(u). Similarly: p(x, y) for PX,Y (x, y) and p(y|x) for PY |X (y|x), etc.

2.1 Entropy
Before we understand entropy, let us take a look at the "surprise function", which will give us more
intuition into the definition of entropy.

Definition 6. "Surprise Function":

1
s(u) , log
p(u)
The surprise function represents the amount of surprise or the amount of information a partic-
ular symbol u of a distribution holds. Intuitively the definition can be understood as follows: we
would be surprised if a rare symbol ( p(u) is small ) is observed. Thus, lower the p(u), higher the
surprise, which is what achieved by the above definition.

6
Definition 7. Entropy: Let U a discrete R.V. taking values in U. The entropy of U is defined
by:
X 1
H(U ) , p(u) log , E[s(u)] (2.1)
u∈U
p(u)

The Entropy represents the expected value of surprise a distribution holds. Intuitively, the
more the expected surprise or the entropy of the distribution, the harder it is to represent.
Note: The entropy H(U ) is not a random variable. In fact it is not a function of the object
(u)
U , but rather a functional (or property) of the underlying distribution PU , u ∈ U. An analogy is
E[U ], which is also a number (the mean) corresponding to the distribution.

Properties of Entropy
Although almost everyone would have encountered the Jensen’s Inequality in their calculus class,
we take a brief look at it in a form most useful for information theory. Jensen’s Inequality: Let
Q denote a convex function, and X denote any random variable. Jensen’s inequality states that

E[Q(X)] ≥ Q(E[X]). (2.2)

Further, if Q is strictly convex, equality holds iff X is deterministic. Example: Q(x) = ex is a

convex function. Therefore, for a random variable X, we have by Jensen’s inquality:

E[eX ] ≥ eE[X]

Conversely, if Q is a concave function, then

E[Q(X) ≤ Q(E[X]). (2.3)

Example: Q(x) = log x is a concave function. Therefore, for a random variable X ≥ 0,

E[log X] ≤ log E[X] (2.4)

W.L.O.G suppose U = {1,2,...,m}

1
1. H(U ) ≤ log m, with equality iff P (u) = m ∀u (i.e. uniform).
Proof:
1
H(U ) = E[log ] (2.5)
p(u)
1
≤ log E[ ] (Jensen’s inequality, since log is concave) (2.6)
p(u)
X 1
= log p(u) · (2.7)
u p(u)
= log m. (2.8)
1 1
Equality in Jensen, iff p(u) is deterministic, iff p(u) = m

EE376A COURSE NOTES 7

 2. H(U ) ≥ 0, with equality iff U is deterministic.
Proof:
1 1
H(U ) = E[log ] ≥ 0 since log ≥0 (2.9)
p(u) p(u)
1
The equality occurs iff log P (U ) = 0 with probability 1, iff P(U) = 1 w.p. 1 iff U is determin-
istic.

3. For a PMF q, defined on the same alphabet as p, define

X 1
Hq (U ) , p(u) log . (2.10)
u∈U
q(u)

Note that this is the expected surprise function, but instead of the surprise associated with
p, it is the surprise associated U , which is distributed according to PMF p, but incorrectly
assumed to be having the PMF of q. The following result stipulates, that we will (on average)
be more surprised if we had the wrong distribution in mind. This makes intuitive sense!
Mathematically,

H(U ) ≤ Hq (U ), (2.11)
with equality iff q = p.
Proof:
1 1
   
H(U ) − Hq (U ) = E log − E log (2.12)
p(u) q(u)
q(u)
 
H(U ) − Hq (U ) = E log (2.13)
p(u)
h i h i
q(u) q(u)
By Jensen’s, we know that E log p(u) ≤ log E p(u) , so

q(u)
 
H(U ) − Hq (U ) ≤ log E (2.14)
p(u)
X q(u)
= log p(u) (2.15)
u∈U
p(u)
X
= log q(u) (2.16)
u∈U
= log 1 (2.17)
=0 (2.18)

Therefore, we see that

H(U ) − Hq (U ) ≤ 0.
q(u)
Equality only holds when Jensen’s yields equality. That only happens when p(u) is determin-
istic, which only occurs when q = p, i.e. the distributions are identical.

EE376A COURSE NOTES 8

 Definition 8. Relative Entropy. An important measure of distance between probability
measures is relative entropy, or the Kullback–Leibler divergence:
p(u) p(u)
X  
D(p||q) , p(u) log = E log (2.19)
u∈U
q(u) q(u)

Note that property 3 is equivalent to saying that the relative entropy is always greater than
or equal to 0, with equality iff q = p (convince yourself).
4. If X1 , X2 , . . . , Xn are independent random variables, then
n
X
H(X1 , X2 , . . . , Xn ) = H(Xi ) (2.20)
i=1

Proof:
1
 
H(X1 , X2 , . . . , Xn ) = E log (2.21)
p(x1 , x2 , . . . , xn )
= E [− log p(x1 , x2 , . . . , xn )] (2.22)
= E [− log p(x1 )p(x2 ) . . . p(xn )] (2.23)
n
" #
X
=E − log p(xi ) (2.24)
i=1
n
X
= E [− log p(xi )] (2.25)
i=1
Xn
= H(Xi ). (2.26)
i=1

Therefore, the entropy of independent random variables is the sum of the individual entropies.
This is also intuitive, since the uncertainty (or surprise) associated with each random variable
is independent.

2.2 Conditional and Joint Entropy

We defined the entropy of a random variable U . We also saw that when U is a joint random variable
of independent variables, then H(U ) is the sum of the individual entropies. Can we say anything
more in general for a joint random variable?
Definition 9. Conditional Entropy of X given Y
 1 
H(X|Y ) , E log (2.27)
P (X|Y )
X 1
= Pr [x, y] (2.28)
x,y log P (x|y)
X X 1 
= P (y) P (x|y) (2.29)
y x log P (x|y)
X
= P (y)H(X|y). (2.30)
y

EE376A COURSE NOTES 9

 Note: The conditional entropy is a functional of the joint distribution of (X, Y ). Note that this
is also a number, and denotes the “average” surprise in X when we observe Y. Here, by definition,
we also average over the realizations of Y. Note that the conditional entropy is NOT a function of
the random variable Y . In this sense, it is very different from a familar object in probability, the
conditional expectation E[X|Y ] which is a random variable (and a function of Y ).

Definition 10. Joint Entropy of X and Y

 1 
H(X, Y ) , E log (2.31)
P (X, Y )
 1
= E log ] (2.32)
P (X)P (Y |X)

Properties of conditional and Joint entropy

1. H(X|Y ) ≤ H(X), equal iff X ⊥ Y
Proof:
 1   1 
H(X) − H(X|Y ) = E log − E log (2.33)
P (X) P (X|Y )
 P (X|Y ) P (Y )   P (X, Y )
= E log = E log ] (2.34)
P (X) P (Y ) P (X)P (Y )
X P (x, y)
= P (x, y) log (2.35)
x,y P (x)P (y)
= D(Px,y ||Px × Py ) (2.36)
≥0 equal iff X ⊥ Y. (2.37)

The last step follows from the non-negativity of relative entropy. Equality holds iff Px,y ≡
Px × Py , i.e. X and Y are independent.

2. Chain rule for entropy:

H(X, Y ) = H(X) + H(Y |X) (2.38)

= H(Y ) + H(X|Y ) (2.39)

3. Sub-additivity of entropy

H(X, Y ) ≤ H(X) + H(Y ), (2.40)

with equality iff X ⊥ Y (follows from the property that conditioning does not increase
entropy)

EE376A COURSE NOTES 10

2.3 Mutual Information
Definition 11. Mutual information between X and Y
We now define the mutual information between random variables X and Y distributed according
to the joint PMF P (x, y):
I(X, Y ) , H(X) + H(Y ) − H(X, Y ) (2.41)
= H(Y ) − H(Y |X) (2.42)
= H(X) − H(X|Y ) (2.43)
= D(Px,y ||Px × Py ) (2.44)
The mutual information is a canonical measure of the information conveyed by one random variable
about another. The definition tells us that it is the reduction in average surprise, upon observing a
correlated random variable. The mutual information is again a functional of the joint distribution
of the pair (X, Y ). It can also be viewed as the relative entropy between the joint distribution, and
the product of the marginals

1. I(X; Y ) ≥ 0, coming from the fact that H(Y ) ≥ H(Y |X).

2. I(X; Y ) ≤ min{H(X), H(Y )}, since the conditional entropies are non-negative. The equality
occurs iff there exists a deterministic function f s.t. Y = f (X) or X = f (Y ) (so that either
H(Y |X) or H(X|Y ), respectively, is zero).
3. Properties for Markov Chains:
We introduce the notation X − Y − Z to reflect that
X and Z are conditionally independent given Y
⇔(X, Y, Z) is a Markov triplet
⇔p(x, z|y) = p(x|y)p(z|y)
⇔p(x|y, z) = p(x|y)
⇔p(z|y, x) = p(z|y)
For example, let X, W1 , W2 be three independent Bernoulli random variables, with Y =
X ⊕ W1 and Z = Y ⊕ W2 . Then, X and Z are conditionally independent given Y , i.e.,
X − Y − Z. Intuitively, Y is a noisy measurement of X, and Z is a noisy measurement of Y .
Since the noise variables W1 and W2 are independent, we only need Y to infer X.
We can also show that if X − Y − Z, then
(a) H(X|Y ) = H(X|Y, Z)
(b) H(Z|Y ) = H(Z|X, Y )
(c) H(X|Y ) ≤ H(X|Z)
(d) I(X; Y ) ≥ I(X; Z), and I(Y ; Z) ≥ I(X; Z)
Intuitively, X − Y − Z indicates that X and Y are more closely related than X and Z.
Therefore I(X; Y ) (i.e., the dependency between X and Y ) is no smaller than I(X; Z), and
H(X|Y ) (the uncertainty in X given knowledge Y ) is no greater than

EE376A COURSE NOTES 11

Chapter 3

Asymptotic Equipartition Properties

In this chapter, we will try to understand how the distribution of n-length sequences generated
by memoryless sources behave as we increase n. We observe that a set of small fraction of all
the possible n-length sequences occurs with probability almost equal to 1. Thus, this makes the
compression of n-length sequences easier as we can then concentrate on this set.
We begin by introducting some important notation:
• For a set S, |S| denotes its cardinality (number of elements contained on the set). For
example, let U = {1, 2, . . . , M }, then |U| = M .

• un = (u1 , . . . , un ) is an n-tuple of u.

• U n = {un | ui ∈ U; i = 1, . . . , n}. It is easy to see that |U n | = |U|n .

• Ui generated by a memoryless source U " implies U1 , U2 , . . . i.i.d. according to U (or PU ).

That is,
n
Y
p(un ) = p(ui )
i=1

3.1 Asymptotic Equipartition Property (AEP)

Definition 12. The sequence un is -typical for a memoryless source U for  > 0, if
 
 1
− log p(un ) − H(U ) ≤ 

 n 

or equivalently,
2−n(H(U )+) ≤ p(un ) ≤ 2−n(H(U )−)
(n)
Let A denote the set of all -typical sequences, called the typical set.
So a length-n typical sequence would assume a probability approximately equal to 2−nH(U ) .
Note that this applies to memoryless sources, which will be the focus on this course1 .
 
(n)
Theorem 13 (AEP). ∀ > 0, P U n ∈ A → 1 as n → ∞.
1
For a different definition of typicality, see e.g. [1]. For treatment of non-memoryless sources, see e.g. [2], [3].

12
Proof This is a direct application of the Law of Large Numbers (LLN).

1
   
P U n ∈ A(n) = P − log p(U n ) − H(U ) ≤ 
 

n
n
  !
 1 Y 
= P − log p(Ui ) − H(U ) ≤ 
 
 n 
i=1
 " n #  !
1 X 
=P − log p(Ui ) − H(U ) ≤ 
 

n 
i=1
→ 1 as n → ∞

where the last step is due to the Law of Large Numbers (LLN), in which − log p(Ui )’s are i.i.d. and
hence their arithmetic average converges to their expectation H(U ).

This theorem tells us that with very high probability, we will generate a typical sequence. But
(n)
how large is the typical set A ?
Theorem 14. ∀ > 0 and sufficiently large n,
 
(1 − )2n(H(U )−) ≤ A(n)
 ≤2
n(H(U )+)
 

Proof The upper bound:

   
2−n(H(U )+) = A(n)  2−n(H(U )+) ,
X X
1 ≥ P U n ∈ A(n) = p(un ) ≥
 

(n) (n)
un ∈A un ∈A

which gives the upper bound. For the lower bound, by the AEP theorem, for any  > 0, there
exists sufficiently large n such that
   
2−n(H(U )−) = A(n) −n(H(U )−)
X X
1 −  ≤ P U n ∈ A(n) = p(un ) ≤  2 .
 

(n) (n)
un ∈A un ∈A

The intuition is that since all typical sequences assume a probability about 2−nH(U ) and their
nH(U ) . Although
total probability is almost 1, the size of the typical set has to be approximately 2
 (n) 
A  grows exponentially with n, notice that it is a relatively small set compared to U n . For some
 > 0, we have
 
 (n) 
A  2n(H(U )+)
≤ = 2−n(log |U |−H(U )−) → 0 as n → ∞
|U n | 2n log |U |
given that H(U ) < log |U| (with strict inequality!), i.e., the fraction that the typical set takes up
in the set of all sequences vanishes exponentially. Note that H(U ) = log |U| only if the source is
uniformly distributed, in which case all the possible sequences are typical.

EE376A COURSE NOTES 13

  
(n)
(n)
A P U n ∈ A ≈1

(n)
un ∈ A ⇔ p(un ) ≈ 2−nH(U )
The set U n of all sequences
 
 (n) 
A  ≈ 2nH(U )

Figure 3.1: Summary of AEP

In the context of lossless compression of the source U , the AEP tells us that we may only focus
on the typical set, and we would need about nH(U ) bits, or H(U ) bits per symbol, for a good
representation of the typical sequences.
(n)
We know that the set, A has probability 1 as n increases. However is it the smallest such
set? The next theorem gives a definitive answer to the question.
 
Theorem 15. For all δ > 0 and all sequences of sets B (n) ⊆ U n such that B (n)  ≤ 2n[H(U )−δ] ,
 

 
lim P U n ∈ B (n) = 0 (3.1)
n→∞

A visualization of Theorem 15 is shown in Figure 3.1.

Figure 3.2: Visualization of all source sequences and ε-typical sequences.

(n) (n)
We can justify the theorem in the following way: As n increases |B (n) ∩ A | ≈ 2−nδ |A |.
As every typical
 sequence has probability of ≈ 2−nH(U ) , and is the same for every sequence,
n
P U ∈B (n) =0

We will next look at a simple application of the AEP for the compression of symbols generated
by a discrete memoryless source.

EE376A COURSE NOTES 14

3.2 Fixed Length (Near) Lossless Compression

Figure 3.3: Block Diagram for Losless Compression

Suppose we have a source U1 , . . . , Un i.i.d. with distribution U . We wish to devise a compression

scheme, as shown in Figure 3.2. The compressor takes a block of n source symbols and converts
them into m binary bits. The decompressor does the inverse process. The rate of such a scheme
(compression and decompression) is defined to be m n bits/source symbol.
We relax our requirements slightly: rather than insisting on strictly lossless compression, we
will simply require the probability of error to be small. That is,
 
Pe = P Û n 6= U n  1 (3.2)

Definition 16 (Achievable rate). R is an achievable rate if for all ε > 0, there exists a scheme (n,
m, compressor, decompressor) whose rate mn ≤ R and whose probability of error Pe < ε.

We are interested in the question: What is the lowest achievable rate? Theorems 17 and 18 tell
us the answer.

Theorem 17 (Direct theorem). For all R > H(U ), R is achievable.

Proof Fix R > H(U ) and ε > 0. Set δ = R − H(U ) > 0 and note that for all n sufficiently large,
by Theorem 13,  
(n)
P Un ∈/ Aδ < ε, (3.3)
and by Theorem 14,  
 (n) 
Aδ  ≤ 2n[H(U )+δ] = 2nR . (3.4)
(n)
Consider a scheme that enumerates sequences in Aδ . That is, the compressor outputs a binary
(n)
representation of the index of U n if U n ∈ Aδ ; otherwise, it outputs (0, 0, . . . , 0). The decompressor
(n)
maps this binary representation back to the corresponding sequence in Aδ . For this scheme, the
probability of error is bounded by
 
(n)
Pe ≤ P U n ∈
/ Aδ <ε (3.5)

and the rate is equal to  

 (n) 
log Aδ  nR
≤ =R (3.6)
n n
Hence, R is an achievable rate.

Theorem 18 (Converse theorem). If R < H(U ), R is not achievable.

EE376A COURSE NOTES 15

Proof For a given scheme of rate r ≤R (and (n) denote the set of possible
 block length n), let B
reconstruction sequences Ûn . Note that B (n)  ≤ 2m = 2nr ≤ 2nR . So if R < H(U ), by Theorem 15,
 

 
Pe ≥ P U n ∈
/ B (n) → 1, as n → ∞ (3.7)

Hence, increasing n cannot make the probability of error arbitrarily small. Furthermore, there is
clearly a nonzero probability of error for any finite n, so R is not achievable. Conceptually, if the
rate is too small, it can’t represent a large enough set.

EE376A COURSE NOTES 16

Bibliography

[1] A. E. Gamal and Y.-H. Kim, Network Information Theory, Cambridge Univ. Press, UK, 2012.

[2] R. M. Gray, Entropy and Information Theory, Springer-Verlag, New York, 1990.

[3] T. S. Han, Information-Spectrum Methods in Information Theory, Springer, 2003.

17
Chapter 4

Lossless Compression

4.1 Uniquely decodable codes and prefix codes

Last lecture, we talked about how using the AEP, entropy emerges when you want to describe
source symbols in fixed length at nearly lossless compression. In fixed-length compression, you
map source sequences to representations 1:1. We also said that if you use variable length coding,
there is a way to achieve H bits/source symbol with perfect lossless compression, where H is the
entropy. How can we achieve such a code? The next few lectures will be devoted to that question.

Let us start with a simple code. Let l(u) represent the length of a binary codeword representing
u, u ∈ U. We can then write ¯l = El(u) = u∈U p(u)l(u) where ¯l is the expected length of a
P

codeword.

Example 19. Let U = {a, b, c, d} and let us try to come up with a simple code for this alphabet.

Figure 4.1: Code I

Note: here l(u) = − log p(u)

⇒ ¯l = E[l(u)] = E[− log p(u)] = H(u)

This code satisfies the prefix condition since no codeword is the prefix for another codeword. It
also looks like the expected code length is equal to the entropy. Is the entropy the limit for variable
length coding? Can we do better? Let us try a better code.

Here is a "better" code, where ¯l < H

However, the code in Figure 4.2 is not uniquely decodable. For instance, both ’abd’ and ’cbb’ can
be represented by the code 0111. These codes are not useful. This motivates the notion of uniquely
decodable schemes.

18
 Figure 4.2: Better code with regard to l(u)

Definition 20. A code is uniquely decodable(UD) if every sequence of source symbols is mapped to
a distinct binary codeword.

Definition 21.
Prefix Condition: When no codeword is the prefix of any other.
Prefix Code: A code satisfying the prefix condition.

Codes that satisfy the prefix condition are decodable on the fly. Codes that do not satisfy the
prefix condition can also be uniquely decodable, but they are less useful.

Exercise 22. Consider Code II in Figure 4.3

Figure 4.3: Code II

Prove that this code is UD.

Let us construct binary trees to represent codes. Here, the terminal nodes represent source symbols,
and the path from the root to each terminal node represents the codeword for that source symbol.
We can construct binary trees for all UD codes, as we will see later.

Here are Binary trees for Code I and Code II:

From here on, let us restrict our attention to prefix codes. In fact, we will see that for any
non-prefix code with a given expected code length, we can always find a prefix code with at least
as small of a code length.

EE376A COURSE NOTES 19

 Figure 4.4: Binary tree for Code I

Figure 4.5: Binary tree for Code II

4.2 Prefix code for dyadic distributions

We would like to systematically construct uniquely decodable prefix codes for any alphabet with
arbitrary probability mass functions. We will start with dyadic distributions.
Definition 23. A dyadic distribution has p(u) = 2−nu , ∀u ∈ U, where nu are integers. (∗)
Note: If we find a UD code with l(u) = nu = − log p(u), then ¯l = H.
We claim that we can always find a UD code for a dyadic distribution.
Lemma 24. Assume (∗) and nmax = maxu∈U nu , considering that we have a nontrivial distribution
(where p(u) is not 1 at one value and 0 everywhere else). The number of symbols u with nu = nmax
is even.
Proof
X
1= p(u)
u∈U

2−nu
X
=
u∈U
nX
max
= (# of symbols u with nu = n) · 2−n
n=1
nXmax
⇒ 2nmax = (# of symbols u with nu = n) · 2nmax −n
n=1
X−1
nmax
= (# of symbols u with nu = n) · 2nmax −n + (# of symbols u with nu = nmax )
n=1
(4.1)

EE376A COURSE NOTES 20

Since all terms except (# of symbols u with nu = nmax ) are even, the number of elements in the
alphabet with the smallest probability has to be even.

Now with this lemma in hand, we can prove our claim that we can find a UD code for a dyadic
distribution. Consider the following procedure:

• Choose 2 symbols with nu = nmax and merge them into one symbol with (twice the) proba-
bility 2−nmax +1

• The new source distribution is also dyadic.

• Repeat the procedure until left with one symbol.

Note: This procedure induces a binary tree.

E.g.:

Figure 4.6: Induced binary tree using the procedure

Also note that the symbol with p(u) = 2−nu has distance nu from root. This means that the
induced prefix code satisfies l(u) = nu = − log p(u)

4.3 Shannon codes

How can we get a good code for a non-dyadic distribution? We can attempt to use the above
principles. Let us look for a code with

l(u) = d− log p(u)e = n∗u ∀u ∈ U

Here, we take the ceiling of − log p(u) as the length of the codeword for the source symbol u. Be-
cause the ceiling of a number is always within 1 of the actual number, the expected code length
¯l = E[− log p(u)] is within 1 of H(u).

∗
Let us consider the "PMF" p∗ (u) = 2−nu . This "PMF" is a dyadic distribution because all
Pp∗ (u)
probabilities are a power of 2. We put PMF in quotes because for a non-dyadic source, u∈U is
less than 1, and so the PMF is not a true PMF. See the following:
∗
p∗ (u) = 2−nu = 2−d− log p(u)e
X X X

u∈U u∈U u∈U

(4.2)
−(− log p(u))
X
< 2 =1
u∈U

EE376A COURSE NOTES 21

To make it a true PMF, let us add fictitious source symbols so that U ∗ ⊇ U and u∈U ∗ p∗ (u) = 1.
P

We can thus construct a prefix code for U ∗ . Because this is a dyadic distribution, we can use the
binary tree principle above to construct a code for all u ∈ U ∗ , and we can consider only the source
symbols u in the original alphabet U . The lengths of each codeword will satisfy

l(u) = − log p∗ (u) = n∗u = d− log p(u)e ∀u ∈ U

The expected code length for a Shannon code can be expressed as the following:

¯l = P p(u)l(u) = P p(u)d− log p(u)e ≤ P p(u)(− log p(u) + 1) = H(U ) + 1

u u u

Therefore, the expected code length is always less or equal to the entropy plus 1. This result
could be good or bad depending on how large H(U ) is to start with. If the extra “1" is too much,
alternatively, we can construct a Shannon code for the multi-symbol un = (u1 , u2 , ...un ), where ui
is memoryless. Then,

¯ln ≤ H(U n ) + 1 or 1¯
≤ n1 H(U n ) + 1 1
n ln n = H(U ) + n

Now we can make it arbitrarily close to the entropy. In the end, there is a trade-off between ideal
code length and memory since the code map is essentially a lookup table. If n gets too large, the
exponential increase in lookup table size could be a problem.

4.4 Average codelength bound for uniquely decodable codes

We looked at a way of obtaining a prefix code for any given distribution. We will now try to
understand the bounds on the average codelength for any generic uniquely decodable code for a
distribution.

Theorem 25 (Kraft-McMillan Inequality). For all uniquely decodable (UD) codes,

2−`(u) ≤ 1
X
(4.3)
u∈U

Conversely, any integer-valued function satisfying (4.3) is the length function of some UD code.

To see the “conversely" statement, note that we know how to generate a UD code (in fact, a
prefix code) with length function satisfying (4.3), using Huffman Codes. Here, we prove the first
claim of the Kraft-McMillan Inequality.

EE376A COURSE NOTES 22

Proof Take any UD code and let `max = maxu `(u). Fix any integer k and note the following.
!k
−`(u)
2−`(u1 ) ) · ( 2−`(u2 ) ) · . . . · ( 2−`(uk ) )
X X X X
2 =(
u∈U u1 u2 uk
k
2−`(ui )
XX XY
= ...
u1 u2 uk i=1
Pk
2− `(ui )
X
= i=1

(u1 ,...,uk )
k)
2−`(u
X
=
uk
k·`max n o
 u `(uk ) = i  · 2−i
X  k 
=

i=1
k·` max
2i · 2−i
X
≤
i=1
= k · `max

Note that the inequality in the second to last line arises because we know that our code is one-to-
one so there can be at most 2i symbols whose codewords have length i. Finally, we can see the
theorem through the following inequality.

2−`(u) ≤ lim (k`max )1/k = 1

X
k→+∞
u

Now, we can prove the important theorem relating UD codes to the binary entropy.

Theorem 26. For all UD codes, ` ≥ H(U )

Proof

Consider, H(U ) − ` = H(U ) − E`(U )

1
 
= E log − `(U )
p(U )
" #
2−`(U )
= E log
p(U )
" #
2−`(U )
≤ log E
p(U )
2−`(u) ≤ log 1 = 0
X
= log
u

EE376A COURSE NOTES 23

Note that we used the earlier proved Kraft-McMillan Inequality for UD codes in the proof.
As an aside, suppose `(u) = − log q(u) for some pmf q. Then,
1
` − H = E[`(U ) − log ]
p(u)
X p(u)
= p(u) log
u q(u)
= D(pkq)

Thus, D(pkq) can be thought of as the “cost of mismatch", in designing a code for a distribution
q, when the actual distribution is p.

4.5 Huffman Coding

We earlier looked at Shannon code, which is a pretty good construction of a prefix code for a given
distribution. However, the best prefix code for a general source code distribution is the Huffman
Code.
The construction of the huffman code follows is very similar to that of the dyadic code. To find
the code c(u), we follow these steps:

1. Find 2 symbols with the smallest probability and then merge them to create a new “node"
and treat it as a new symbol.

2. Then merge the next 2 symbols with the smallest probability to create a new “node"

3. Repeat steps 1 and 2 until there is only 1 symbol left. At this point, we created a binary tree.
The paths traversed from the root to the leaves are the prefix codes.

We consider an example of huffman code construction:

Example 27. Prefix code for a senary source (six letters) is given below:

u p(u) c(u) l(u)

p=0.15 0 e e 0.1 1110 4
1
1
p=0.3 2
1 f f 0.05 1111 4
0
p=0.55
1
4
d d 0.15 110 3
0
1 a a 0.25 10 2
5
0 0 b b 0.2 00 2
3
p=0.45
1 c c 0.2 01 2

We will next try to understand why Huffman codes are the optimal prefix codes.

EE376A COURSE NOTES 24

4.6 Optimality of Huffman codes
Suppose we have a random object U ∼ P with alphabet U = {1, 2, . . . , r}. We let c(u), `(u) be
the codeword and length associated with some u ∈ U, respectively. Recall, the average length is
denoted as ` = E`(U ). Assume, WLOG, that the symbols in U are ordered in decreasing order of
probability according to p, i.e. p(1) ≥ p(2) ≥ · · · ≥ p(r). Let Ur−1 denote a random variable over
Ur−1 = {1, 2, . . . , r − 1} and
(
p(i) 1≤i≤r−2
p(Ur−1 = i) =
p(r − 1) + p(r) i=r−1

and cr−1 (ur−1 ), `r−1 (ur−1 ) are the codeword and length of ur−1 ∈ Ur−1 , respectively. Again,
`r−1 = E`r−1 (Ur−1 ).
"Splitting a prefix code cr−1 ": creating a prefix code for U by

c(i) = cr−1 (i)

 1≤i≤r−2
c(r − 1) = cr−1 (r − 1)0

r−1 (r − 1)1

c(r) = c

We will use the following lemma to justify the optimality of Huffman Codes. Intuitively, we
will show that if we start with an optimal code on r − 1 symbols, splitting gives us an optimal code
over r symbols. We can use an inductive argument, starting with a binary object to prove that
Huffman Codes are optimal for alphabets with any number of symbols.

Lemma 28. Let copt,r−1 be an optimal prefix code for Ur−1 . Let c be the code obtained from copt,r−1
by splitting. Then c is an optimal prefix code for U .

Proof Note there is an optimal prefix code for U satisfying:

1. `(1) ≤ `(2) ≤ . . . ≤ `(r) Otherwise, we could rearrange the codes to satisfy this property, and
the result would be at least as good due to the ordergin we have assumed on the probabilities.

2. `(r − 1) = `(r) = `max

Suppose in an optimal code the two longest codewords were not of the same length. Since
the prefix property holds, no codeword is a prefix of the longest codeword. The longest
codeword can be truncated, preserving the prefix property but achieving lower expected
codeword length. Since the code was optimal, this leads to a contradiction, so the two longest
codewords must be of the same length.

3. c(r − 1) and c(r) differ only in the last bit

Given any optimal code, rearranging the code can result in a code with this property. Suppose
there is a codeword of maximal length such that it does not have a ‘sibling’. The last bit of this
codeword can be deleted, preserving the prefix property but achieving lower expeted codeword
length. This leads to a contradiction, so every codeword of maximal length must have a
‘sibling’ in an optimal code. By rearranging the assignments of the maximal length codewords,
we can ensure that the two least likely symbols are assigned a pair of sibling codewords. This
rearrangement maintains the expected codeword length, and achieves property.

EE376A COURSE NOTES 25

Note, by the second and third properties, there is an optimal prefix code for U that is the result of
splitting a prefix code for Ur−1 (where the code for Ur−1 can be seen as our original code with the
the codeword c(r − 1) is truncated before the final bit). Let `r−1 be the length function of a prefix
code for Ur−1 and ` be the length function of a prefix code for U obtained by splitting Ur−1 . Then
r
X
`= p(i)l(i)
i=1
r−2
X
= p(i)l(i) + p(r − 1)l(r − 1) + p(r)l(r)
i=1
r−2
X
= p(Ur−1 = i)lr−1 (i) + (p(r − 1) + p(r))l(r − 1)
i=1
r−2
X
= p(Ur−1 = i)lr−1 (i) + (p(r − 1) + p(r))(lr−1 (r − 1) + 1)
i=1
r−2
X
= p(Ur−1 = i)lr−1 (i) + p(Ur−1 = r − 1)lr−1 (r − 1) + (p(r − 1) + p(r))
i=1
r−1
X
= p(Ur−1 = i)lr−1 (i) + (p(r − 1) + p(r))
i=1
= `r−1 + p(r − 1) + p(r)

` = `r−1 + p(r − 1) + p(r)

because the expectation sums differ only in the final two terms, where an additional bit is added
for symbols r − 1 and r. We can see, by this simple relationship, that if we want to optimize ` for
some fixed probability distribution, it suffices to optimize `r−1 . So if `r−1 is optimal, then so is `.
Thus, we have that an optimal prefix code for U is obtained by splitting an optimal prefix code for
Ur−1 .

EE376A COURSE NOTES 26

Bibliography

[1] D. J. C. MacKay, Information Theory, Inference, and Learning Algorithms, Cambridge Uni-
versity Press, UK, 2003.

[2] S. Geman and D. Geman, “Stochastic Relaxation, Gibbs Distributions, and the Bayesian
Restoration of Images,” IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 6,
pp. 721–741, 1984.

27
Chapter 5

Communication and Channel

Capacity

5.1 The Communication Problem

We model communication as the transmission of an input consisting of n symbols (denoted X n )
through a noisy channel. This channel will emit an output of n symbols (denoted Y n ). Think of
the channel as corrupting or adding noise to the input.
The behavior of the channel is characterized by the conditional probability distribution PY n |X n ,
which tells us the distribution over the outputs that it will emit for any given input.
Conceptually, we have the following picture:

noisy channel
X n −→ PY n |X n −→ Y n

Our goal, then, is to find an encoder e and a decoder d such that we can take an input of m
bits (called B m ), encode it using e to get an encoding of length n denoted X n , pass the encoding
through the channel to get Y n , and then decode Y n using d to recover an estimate of the original
string of bits, denoted B̂ m . Pictorially, that is:

noisy channel
Xn Yn
(B1 , B2 , . . . , Bm ) = B m −→ encoder (e) −→ PY n |X n −→ decoder (d) −→ B̂ m = (B̂1 , B̂2 , . . . , B̂m )

Ideally, n will be small relative to m (we will not have to send a lot of symbols through the noisy
channel), and the probability that the received message B̂ m does not match the original message
B m will be low. We will now make rigorous these intuitively good properties.
Definition 29. We define a scheme to be a pair of encoder and decoder, denoted (e, d).
Note that the definition of a scheme does not include the noisy channel itself. We must take the
channel as it is given to us: we cannot modify it, but we can choose what symbols X n to transmit
through it.

28
Definition 30. We define the rate of a scheme, denoted R, to be the number of bits communicated
per use of the channel. This is equal to m/n in the notation of the diagram above.
Definition 31. We define the probability of error for a scheme, denoted Pe , to be the probability
that the output of the decoder does not exactly match the input of the encoder. That is,
Pe = P (B m 6= B̂ m ).
Definition 32. For a given channel, we say that a rate R is achievable if there exists a sequence
of schemes (e1 , d1 ), (e2 , d2 ), . . ., such that:
1. For all n = 1, 2, . . ., scheme (en , dn ) has rate at least R, and
2.
lim Pe(n) = 0,
n→∞
(n)
where Pe denotes the probability of error of the nth scheme.

5.2 Channel Capacity

We want to know what the best possible performance is under a particular noisy channel. This is
essentially what channel capacity tells us.
Definition 33. For a given channel, the channel capacity (denoted C) is the theoretical limit on the
number of bits that can be reliably communicated (i.e., communicated with arbitrarily low probability
of error) in one channel use.
That is,
C := sup{R : R is achievable}
n
Y
We assume a "memoryless channel": PY n |X n (y n |xn ) = PY |X (yi |xi ) exhibiting a "single letter
i=1
channel" characteristic of output symbol given input symbol. Restated, the i-th channel output
only cares about the ith channel input.

X ∼ Px −→ PY |X −→ random Y

With this single letter channel, we now examine I(X; Y ). What distribution of X will maximize
I(X; Y ) over all possible channel inputs?

C (I) , max I(X; Y )

PX

Theorem 34. Channel Coding Theorem:

C = C (I) = max I(X; Y ) (sometimes written as max)

X PX

Proof:
Direct Theorem: If R < C (I) , then the rate R is achievable.
Converse Theorem: If R > C (I) , then R is not achievable.
The direct part and the converse part of the proof are given at the end of this chapter.

EE376A COURSE NOTES 29

5.2.1 Channel Capacity of various discrete channels
Example 35. Binary Symmetric Channel (BSC)

Let: X = Y = {0, 1}

and the crossover probability be

(
p y 6= x
PY |X (y|x) =
1−p y =x

0 1
⇐⇒ channel matrix: 0 1 − p p
1 p 1−p
1−p
0 0
p
p
1 1
1−p
⇐⇒ bipartite graph:

⇐⇒ Y = X ⊕2 Z ← ber(p)

To compute the capacity of the BSC, we first examine mutual information

I(X; Y ) = H(Y ) − H(Y |X) definition of mutual information

= H(Y ) − H(X ⊕2 Z|X) substitute Y with X ⊕2 Z
= H(Y ) − H(Z) given X, X ⊕2 Z is simply Z
= H(Y ) − h2 (p) ≤ 1 − h2 (p) h2 (p) is binary entropy

To achieve equality, H(Y ) = 1, i.e. Y is bernoulli 21 . Taking X ∼ Ber( 12 ) produces this desired Y
and therefore gives I(X; Y ) = 1 − h2 (p)

=⇒ C = 1 − h2 (p)

(1 − h2 (p))n bits of information can be communicated reliably.

Example 36. Binary Symmetric Channel (BEC)

Let: X = {0, 1}, Y = 0, 1, e

and the crossover probability be

(
α y=e
PY |X (y|x) =
1−α y=x

EE376A COURSE NOTES 30

 1−α 0
0 α
e
1 α

1−α 1

I(X; Y ) = H(X) − H(X|Y )

= H(X) − [H(X|Y = e)P (Y = e) + H(X|Y = 0)P (Y = 0) + H(X|Y = 1)P (Y = 1)]
= H(X) − [H(Xα + 0 + 0]
= (1 − α)H(x)

To achieve equality, H(X) = 1, i.e. X is bernoulli 21 .

I(X; Y ) = 1 − α

=⇒ C = 1 − α
(1 − α)n bits of information can be communicated reliably.

5.2.2 Recap
J ⇐⇒
B1 , B2 , ..., Bm Jˆ ⇐⇒
i.i.d. ∼Ber(1/2) Noisy B̂1 , B̂2 , ..., B̂m
Encoder/ Xn Yn Decoder/
Memoryless
Transmitter Receiver
Channel

m bits
• Rate = n channel use

• Pe = P(Jˆ 6= J)
m
• R is achievable if ∀  > 0, ∃ a scheme (m, n, encoder, decoder) with n ≥ R and Pe < .

• Capacity: C = sup{R: R is achievable}.

• Channel Coding Theorem: C = maxX I(X; Y ).

Note: The Channel Coding Theorem is equally valid for analog signals, e.g., the AWGN chan-
nel. However, we must extend our definition of the various information measures such as entropy,
mutual information, etc.

Next we extend the information measures to continue random variables, and analyze the AWGN
channel.

EE376A COURSE NOTES 31

5.3 Information Measures for Continuous Random Variables
In this chapter we extend the information measures to continues random variables.
Definition 37. Relative Entropy for two PDFs f and g, is defined as
f (x)
Z
D(f ||g) = f (x) log dx, x ∈ R.
g(x)
Can similarly define for x ∈ Rn .
Note: When integral on the right hand side is not well defined, D(f ||g) = ∞
Definition 38. Mutual Information between two continuous r.v. X, Y with joint pdf fX,Y (i.e.,
X, Y ∼ fX,Y ) is

I(X; Y ) = D(fX,Y ||fX · fY )

fX,Y (x, y)
Z Z
= fX,Y (x, y) log dxdy,
fX (x)fY (y)
where fX × fY is the product of the marginal distributions.
Definition 39. Differential entropy of
1. X ∼ fX :
h(x) , E[− log fX (x)]

2. X, Y ∼ fX,Y :

h(X, Y ) , E[− log fX,Y (X, Y )] (“Joint Differential Entropy”)

h(X|Y ) , E[− log fX|Y (X|Y )] (“Conditional Differential Entropy”)

Each of the above definitions is totally analogous to the discrete case.

In the homework we will show the following result:

I(X; Y ) = h(X) + h(Y ) − h(X, Y )

= h(X) − h(X|Y )
= h(Y ) − h(Y |X)

This is the main/only interest in differential entropy.

Note: Unlike discrete entropy H(X), differential entropy can be positive or negative. This is not
the only way in which they differ.

h(X + c) = h(X), for constant c

h(X · c) = h(X) + log |c|, c 6= 0

EE376A COURSE NOTES 32

5.3.1 Examples
Example 40.
X ∼ U [a, b]
(
1
b−a a≤x≤b
fX (x) =
0 otherwise
h(x) = log(b − a)
Example 41.
1 −x2
X ∼ N (0, σ 2 ), f (x) = √ e 2σ2
2πσ 2
" #
x2 /2σ 2
1
h(x) = E − log √ +
2πσ 2 ln 2
1 σ 2 /2σ 2
= log 2πσ 2 +
2 ln 2
1 2 1
= log 2πσ +
2 2 ln 2 
1 log e
= log 2πσ 2 +
2 log 2
1h 2
i
= log 2πσ + log e
2
1
= log(2πσ 2 e)
2

Note: differential entropies can be either positive or negative. The more correlated the random
variable the more negative

Example 42. Significance of Differential entropy

Many Information theorists would argue none whatsoever. However, some others offer a different
perspective.
If you discretize X ∼ f into X∆ with time period ∆,
Z i∆+∆/2
P (x∆ = i) = f (x)dx ≈ f (i∆) · ∆
i∆−∆/2

X 1
H(X∆ ) = Pi log
i
P1
X 1
≈ f (i∆) · ∆ · log
i
∆f (i∆)
1 X 1
 
= log + f (i∆) log ∆
∆ i
f (i∆)
1 1 1
Z
X ∆→0
H(X∆ ) − log = f (i∆) log ∆ −−−→ f (x) log dx = h(x)
∆ i
f (i∆) f (x)

EE376A COURSE NOTES 33

 1
=⇒ H(X∆ ) ≈ log + h(x)
∆
1
Majority of the entropy for discretized system is accounted for with log ∆ . The rest of it is h(x)
the differential entropy

5.3.2 Gaussian Distribution

Claim: The Gaussian distribution has maximal differential entropy, i.e.,:
If X ∼ fX with E[X 2 ] ≤ σ 2 (second moment), and G ∼ N (0, σ 2 ),
Then h(X) ≤ h(G), with equality iff X ∼ N (0, σ 2 ).

Note: If E[X 2 ] ≤ σ 2 and V ar(X) = σ 2 , then necessarily E[X] = 0.

Proof of Claim:

1 1 2 √ 1
2X
2
fG (X) = √ e− 2σ2 X , − log fG (X) = log 2πσ 2 + 2σ
2πσ 2 ln 2

fX (X)
 
0 ≤ D(fX ||fG ) = E log
fG (X)
1
 
= −h(X) + E log
fG (X)
√ 1
" #
2σ 2
X2
= −h(X) + E log 2πσ 2 +
ln 2
√ 1
" #
2σ 2
G2
≤ −h(X) + E log 2πσ 2 +
ln 2
1
 
= −h(X) + E log
fG (G)
= −h(X) + h(G)

∴ h(X) ≤ h(G), with equality iff D(fX ||fG ) = 0, i.e., X ∼ G

5.4 Channel Capacity of the AWGN Channel (Additive White

Gaussian Noise)
Wi

Xi + Yi

EE376A COURSE NOTES 34

 Note: The AWGN channel is memoryless.

• Transmission is restricted to power P :

n
1X
X 2 ≤ P, ∀n ∈ N
n i=1 i

• R is achievable with power P if: ∀  > 0, ∃ scheme restricted to power P and with rate
m
n ≥ R and probability of error Pe < .

• Channel Capacity: C(P ) = sup{R: R is achievable with power P}

5.4.1 Channel Coding Theorem for this Setting

C(P ) = max I(X; Y )
E[X 2 ]≤P

Note: We could instead have considered the restriction that E[ n1 ni=1 Xi2 ] ≤ P . This constitutes
P

a relaxed constraint. However, it turns out that even with the relaxation, you cannot perform any
better in terms of the fundamental limit.

5.4.2 An Aside: Cost Constraint

More generally, we can consider an arbitrary cost function constraint on X, rather than the
above power constraint. We can denote this cost function by φ(Xi ). The cost constraint is then
1 Pn
n i=1 φ(Xi ) ≤ α. This means that the average cost cannot exceed the core parameter α, so we
consider C(α). In this case, the coding theorem becomes C(α) = maxE[φ(X)]≤α I(X; Y ).

5.4.3 The Example

W ∼ N (0, σ 2 )

X
+ Y
E[X 2 ] ≤P

EE376A COURSE NOTES 35

I(X; Y ) = h(Y ) − h(Y |X)
(given X, X is a constant, so we can use invariance
= h(Y ) − h(Y − X|X)
of differential entropy to constant shifts)
= h(Y ) − h(W |X)
= h(Y ) − h(W ) (since W and X are independent)
2 2
≤ h(N (0, P + σ )) − h(N (0, σ )) (V ar(Y ) = V ar(X + W ) = V ar(X) + V ar(W ) ≤ P + σ 2 )
1 1
= log 2πe(P + σ 2 ) − 2πeσ 2
2 2
1 P +σ 2
= log 2
2  σ
1 P

= log 1 + 2
2 σ
So in conclusion,
1 P
 
I(X; Y ) ≤ log 1 + 2
2 σ
with equality
⇐⇒ Y ∼ N (0, P + σ 2 )
⇐⇒ X ∼ N (0, P )
Therefore, equality is achievable. So,
1 P
 
C(P ) = log 1 + 2
2 σ

(i.e., the capacity of the AWGN channel.)

5.4.4 Rough Geometric Interpretation (Picture)

qP
n 2 √
• Transmission Power Constraint: i=1 Xi ≤ np
qP
n
√
• Noise: 2 ≈ nσ 2
i=1 Wi

• Channel Output Signal:

" n # " n #
X X
E Yi2 = E (Xi + Wi )2
i=1 i=1
" n n
#
X X
=E Xi2 + Wi2
i=1 i=1
(independence ⇒ cross-terms have zero expectation)
≤ nP + nσ 2
= n(P + σ 2 )

EE376A COURSE NOTES 36

 Figure 5.1: Geometrically, consider the input and output sequences as points in Rn .

See Figure 7.2 for the geometric interpretation of this problem. We want the high probability
output balls to not intersect. This way, we can uniquely distinguish the input sequences associated
with any given output sequence.
p
Vol(n-dim ball of radius n(P + σ 2 ))
# messages ≤ √
Vol(n-dim ball of radius nσ 2 )
This inequality is due to inefficiencies in the packing ratio. Equality corresponds to perfect
packing, i.e. no dead-zones. So,
n/2
Kn ( n(P + σ 2 ))n
p
P

# of bits = √ = 1+ 2
Kn ( nσ 2 )n σ
log # of messages 1 P
 
⇒ rate = ≤ log 1 +
n 2 Q
The achievability of the equality indicates that in high dimension, can pack the balls very
effectively.

EE376A COURSE NOTES 37

5.5 Joint Asymptotic Equipartition Property (AEP)
Let X, Y be jointly random variables with alphabets X , Y, respectively. Let the source be memo-
ryless so that (Xi , Yi ) are i.i.d. ∼ PX,Y . That is,
n
Y n
Y n
Y
P (xn ) = PX (xi ), P (y n ) = PY (yi ), P (X n , Y n ) = PX,Y (xi , yi ) (5.1)
i=1 i=1 i=1

5.5.1 Set of Jointly Typical Sequences

(n)
Let A (X, Y ) denote the set of jointly typical sequences. That is,
n n 1
A(n)
 (X, Y ) = {(X , Y ) : | − log P (xn ) − H(X)| < , (5.2)
n
1
|− log P (y n ) − H(Y )| < , (5.3)
n
1
| − log P (xn , y n ) − H(X, Y )| < } (5.4)
n
Theorem 43. If (X n , Y n ) are formed by i.i.d. (Xi , Yi ) ∼ PX,Y , then

1.
lim P ((X n , Y n ) ∈ A(n)
 (X, Y )) = 1 (5.5)
n→∞
By the AEP, we have that X n is typical, Y n is typical, and (X n , Y n ) is typical too.
2. ∀ > 0, ∃n0 ∈ N such that ∀n > n0
(1 − )2n(H(X,Y )−) ≤ |A(n)
 (X, Y )| ≤ 2
n(H(X,Y )+)
(5.6)

Theorem 44. If (X̃ n , Ỹ n ) are formed by i.i.d. (X̃i , Ỹi ) ∼ (X̃, Ỹ ), where PX̃,Ỹ = PX · PY , then
∀ > 0, ∃n0 ∈ N such that ∀n > n0
(1 − )2−n(I(X;Y )+3) ≤ P ((X̃ n , Ỹ n ) ∈ A(n) (X, Y )) ≤ 2−n(I(X;Y )−3) (5.7)
Intuition:
nH(X̃,Ỹ )
|A(n)
 (X̃, Ỹ )| ≈ 2 (5.8)
n(H(X)+H(Y ))
=2 (5.9)
= 2nH(X) · 2nH(Y ) (5.10)
≈ |A(n)
 (X)| · |A(n)
 (Y )| (5.11)
(n) (n)
Note that (X̃, Ỹ ) are distributed uniformly within a set of size |A (X)| · |A (Y )|
(n)
|A (X, Y )|
⇒ P ((X̃ n , Ỹ n ) ∈ A(n)
 (X, Y )) = (n) (n)
(5.12)
|A (X)| · |A (Y )|
2nH(X,Y )
≈ (5.13)
2nH(X) · 2nH(Y )
= 2−nI(X;Y ) (5.14)

EE376A COURSE NOTES 38

Proof :
X
P ((X̃ n , Ỹ n ) ∈ A(n)
 (X, Y )) = P (x̃n ) · P (ỹn ) (5.15)
(n)
(x̃n ,ỹ n )∈A (X,Y )

2−n(H(X)−) · 2−n(H(Y )−)

X
≤ (5.16)
(n)
(x̃n ,ỹ n )∈A (X,Y )

= |A(n) (X, Y )| · 2−n(H(X)+H(Y )−2) (5.17)

n(H(X,Y )−) −n(H(X)+H(Y )−2)
≤2 ·2 (5.18)
n(H(X,Y )−H(X)−H(Y )−3)
≤2 (5.19)
= 2−nI(X;Y )−3 (5.20)

5.6 Direct Theorem

Recall the setting under consideration:

J ⇐⇒ B1 , B2 , . . . , BM Jˆ ⇐⇒ B̂1 , B̂2 , . . . , B̂M

i.i.d. ∼Bern(1/2) n n
Encoder/ X Memoryless Channel Y Decoder/
Transmitter P (Y |X) Receiver

J is uniformly distributed on {1, 2, . . . , M }. We define our scheme as follows:

Encoder (also known as “codebook”): {1, 2, . . . , M } → X n .

That is, codebook cn = {X n (1), X n (2), . . . , X n (M )}

ˆ : Y n → {1, 2, . . . , M }
Decoder: J(·)

Rate: Bits per channel use = log(M )/n = log(|cn |)/n

Theorem 45. (Direct Theorem) If R < maxPX I(X; Y ), then R is achievable. Equivalently, if
∃ PX s.t. R < I(X; Y ), then R is achievable.

Proof
Fix PX and a rate R < I(X; Y ). Choose  = (I(X; Y )−R)/4. This means that R < I(X; Y )−3.
Generate codebook Cn of size M = d2nR e.
X n (k) are i.i.d. with distribution PX , ∀ k = 1, 2, . . . , M . Then

(
ˆ n) = j if (X n (j), Y n ) ∈ An (X, Y ) and (X n (k), Y n ) ∈
/ An (X, Y ), ∀j 6= k
J(Y (5.21)
error otherwise

EE376A COURSE NOTES 39

Denote probability of error using a codebook cn as Pe (cn ). Thus, Pe (cn ) = P (Jˆ 6= J|Cn = cn )

E[Pe (Cn )] = P (Jˆ 6= J) (5.22)

M
P (Jˆ 6= J|J = i)P (J = i)
X
= (5.23)
i=1
= P (Jˆ = J|J = 1) (5.24)

This follows because, by symmetry, P (Jˆ 6= J|J = i) = P (Jˆ 6= J|J = j), ∀i, j, and P (J = i) =
1/M, ∀i

By union bound, it follows that

P (Jˆ 6= J|J = 1)
(n) M (n)
≤ P ((X n (1), Y n ) ∈ P ((X n (k), Y n ) ∈ A (X, Y ))
P
/ A (X, Y )) +
k=2
The first term on the right tends to zero as n tends to infinity. Therefore,
M
P (Jˆ 6= J|J = 1) ≤
X
P ((X n (k), Y n ) ∈ A(n)
 (X, Y )) (5.25)
k=2
XM
≤ P ((X̃ n , Ỹ n ) ∈ A(n) (X, Y )) (5.26)
k=2
≤ (M − 1) · 2−n(I(X;Y )−3) (5.27)
≤ 2nR · 2−n(I(X;Y )−3) (5.28)
−n(I(X;Y )−3−R)
≤2 (5.29)

Since R < I(X; Y ) − 3, the expression tends to zero as n tends to infinity.

This means that,

∃cn s.t. |cn | ≥ 2nR and Pe (cn ) ≤ E[Pe (Cn )]

⇒ ∃cn s.t. |cn | ≥ 2nR and limn→∞ Pe (cn ) = 0
⇒ R is achievable.

5.7 Fano’s Inequality

Theorem 46 (Fano’s Inequality). Let X be a discrete random variable and X̂ = X̂(Y ) be a guess
of X based on Y . Let Pe := P (X̂ 6= X). Then H(X|Y ) ≤ h(Pe ) + Pe log(|X | − 1).

EE376A COURSE NOTES 40

Proof Let V = 1{X̂6=X } .

H(X|Y ) ≤ H(X, V |Y ) (Data Processing Inequality) (5.30)

= H(V |Y ) + H(X|V, Y ) (Chain Rule) (5.31)
≤ H(V ) + H(X|V = 0, Y = y)P (V = 0, Y = y)
+H(X|V = 1, Y = y)P (V = 1, Y = y) (5.32)
We can simplify terms in (5.32). First, H(V ) = h(Pe ), the entropy of a binary random variable
with success probability Pe . Furthermore, X is deterministic given V = 0 and Y = y, so H(X|V =
0, Y = y) = 0. Finally, if V = 1 and Y = y, then X̂ is known and X can take up to |X | − 1 values.
Thus H(X|V = 1, Y = y) ≤ log(|X | − 1). Putting these facts together, we arrive at:
H(X|Y ) ≤ h(Pe ) + log(|X | − 1)P (V = 1) (5.33)
= h(Pe ) + Pe log(|X | − 1) (5.34)

A weaker version of Fano’s Inequality uses the facts that h(Pe ) ≤ 1 and log(|X | − 1) ≤ log(|X |):
H(X|Y ) ≤ 1 + Pe log(|X |) (5.35)
or equivalently,
H(X|Y ) − 1
Pe ≥ (5.36)
log(X )

5.8 Converse Theorem

Theorem 47 (Converse Theorem). If R > C (I) , then rate R is not achievable.
Proof
log M − H(J|Y n ) = H(J) − H(J|Y n ) (5.37)
n
= I(J; Y ) (5.38)
n n
= H(Y ) − H(Y |J) (5.39)
X X
i−1 i−1
= H(Yi |Y )− H(Yi |Y , J) (5.40)
i i
X X
≤ H(Yi ) − H(Yi |Y i−1 , J, X n ) (conditioning reduces entropy) (5.41)
i i
X X
= H(Yi ) − H(Yi |Xi ) (memorylessness) (5.42)
i i
X
= I(Xi ; Yi ) (5.43)
i
≤ nC (I) (5.44)
log M
Thus, for schemes with rate (= n ) ≥ R, we have

H(J|Y n ) − 1 log M − nC (I) − 1 C (I) 1 n→∞ C (I)

Pe ≥ ≥ ≥1− − −−−→ 1 − (5.45)
log M log M R nR R

EE376A COURSE NOTES 41

If R > C (I) , then Pe is bounded from below by a positive constant, so it does not approach 0.
Therefore, R > C (I) is not achievable.

5.9 Some Notes on the Direct and Converse Theorems

5.9.1 Communication with Feedback: Xi (J, Y i−1 )
Even if the encoder gets feedback of what has been received on the other side of the channel, one
can verify that the proof of converse carries over verbatim; C = C (I) with or without feedback!
But, feedback can help improve simplicity and reliability of schemes to achieve the best rate. Here
is an example:

Example 48. Communicating Through Erasure Channel

Recall that the capacity of the erasure channel (Fig. 7.2) is C = 1 − α bits/channel use. If
feedback exists, the transmitter can repeat each information bit until it goes through unerased.
On average, one needs 1/(1 − α) channel uses per information bit. This means that the rate
achieved by this scheme is 1 − α bits/channel use. This simple scheme is completely reliable since
the probability of error is equal to zero (every bit will eventually be error-free).

1-α
0
0 α

α
e
1
1-α
1

Figure 5.2: An Erasure Channel

5.9.2 Practical Schemes

In the proof of the direct part, we showed mere existence of Cn (a codebook achieving the rate
equivalent to the channel capacity) with a size |Cn | ≥ 2nR , and small Pe . Even if such Cn is given,
encoding and decoding using this codebook for large n is not practical. For practical schemes, see:

1. LDPC Codes: "Low Density Parity Check Codes", Gallager 1963 Thesis [3].

2. Polar Codes: "Channel Polarization", Arikan 2009 [4].

3. Or, take EE388 – Modern Coding Theory.

5.9.3 Pe vs. Pmax

In our discussion so far, our notion of reliability has been the (average) probability of error, which
is defined as:

EE376A COURSE NOTES 42

 M
1 X
Pe = P (Jˆ 6= J) = P (Jˆ 6= J|J = j) (5.46)
M j=1

A more stringent notion of reliability is the maximal probability of error Pmax , which is defined as:

Pmax = max P (Jˆ 6= j|J = j) (5.47)

1≤j≤M

It turns out that our results, i.e., direct and converse theorems, are still valid for this more stringent
notion of reliability. The converse theorem is clear. If arbitrarily small Pe cannot be achieved,
arbitrarily small Pmax cannot be achieved either, therefore the converse theorem holds for Pmax .
We now show that the result of the direct proof holds for vanishing Pmax . Note that with application
of the Markov inequality, we have:

M
|{1 ≤ j ≤ M : P (Jˆ 6= j|J = j) ≤ 2Pe }| ≥ (5.48)
2
Given Cn with |Cn | = M and Pe , there exists Cn0 with |Cn0 | = M/2 and Pmax ≤ 2Pe . By extracting
a better half of Cn , one can construct Cn0 . The rate of Cn0 is:

log(M/2) log M 1
Rate of Cn0 ≥ = − (5.49)
n n n
This implies that if there exists schemes of rate ≥ R with Pe → 0, then for any  > 0, there exists
schemes of rate ≥ R −  with Pmax → 0

EE376A COURSE NOTES 43

Bibliography

[1] D. J. C. MacKay, Information Theory, Inference, and Learning Algorithms, Cambridge Uni-
versity Press, UK, 2003.

[2] S. Geman and D. Geman, “Stochastic Relaxation, Gibbs Distributions, and the Bayesian
Restoration of Images,” IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 6,
pp. 721–741, 1984.

[3] R. G. Gallager, Low-Density Parity-Check Codes, Cambridge, MA: MIT Press, 1963.

[4] E. Arikan, “Channel polarization: A method for constructing capacity achieving codes for
symmetric binary-input memoryless channels,” IEEE Trans. Inf. Theory, vol. 55, no. 7, pp.
3051–3073, July 2009.

44
Chapter 6

Method of Types

For additional material on the Method of Types, we refer the reader to [1] (Section 11.1).

6.1 Method of Types

Denote xn = (x1 , . . . , xn ), with xi ∈ X = {1, 2, . . . , r}.

Definition 49. The empirical distribution or type of xn is the vector (Pxn (1), Pxn (2), . . . Pxn (r))
N (a|xn )
, where N (a|xn ) = ni=1 1{xi =a} .
P
of relative frequencies Pxn (a) =
n
Definition 50. Pn denotes the collection of all empirical distributions of sequences of length n.

Example 51. X = {0, 1}

1 n−1 2 n−2
     
Pn = (0, 1) , , , , , . . . , (1, 0) .
n n n n

Definition 52. If P ∈ Pn (probabilities are integer multiples of 1/n), the type class or type of P
is T (P ) = {xn : Pxn = P }. The type class of xn is Txn = T (Pxn ) = {x̃ : Px̃n = Pxn }.

Example 53. X = {a, b, c}, n = 5, xn = (aacba)

Then Pxn = (3/5, 1/5, 1/5)

Txn = {aaabc, aaacb, . . . , cbaaa}

5

5!
|Txn | = 311 = = 20.
3!1!1!

45
Theorem 54. |Pn | ≤ (n + 1)r−1

Proof Type of xn is determined by (N (1|xn ) , N (2|xn ) , . . . , N (r|xn )). Each component can
assume no more than n + 1 values (0 ≤ N (i|xn ) ≤ n) (and the last component is dictated by the
others).

Example 55. For X = {0, 1}, |Pn | = n + 1 = (n + 1)r−1 .

Notation:

- Q = {Q(x)}x∈X is a PMF, write H(Q) for H(X) when X ∼ Q.

Qn
- Qn (xn ) = S ⊆ X n , Qn (S) = Qn (xn ).
P
i=1 Q(xi ), xn ∈S

Theorem 56. ∀xn : Qn (xn ) = 2−n[H(Pxn )+D(Pxn ||Q)] .

Proof
n
Y
Qn (xn ) = Q(xi )
i=1
Pn
log Q(xi )
=2 i=1
P
N (a|xn ) log Q(a)
=2 a∈X
P N (a|xn )
= 2n a∈X n
log Q(a)
P N (a|xn ) 1
−n log
=2 a∈X n Q(a)
P 1

−n Pxn (a) log
=2 a∈X Q(a)
hP i
1
P Pxn (a)
−n a∈X
Pxn (a) log Pxn (a)
+ a∈X
Pxn (a) log Q(a)
=2
= 2−n[H(Pxn )+D(Pxn ||Q)] .

Theorem 57. ∀P ∈ Pn ,
1
2nH(P ) ≤ |T (P )| ≤ 2nH(P ) .
(n + 1)r−1
n
n!
Note: |T (P )| = nP (1) nP (2) ... nP (r) =Q .
a∈X (nP (a))!

EE376A COURSE NOTES 46

Proof UPPER BOUND:
X
1 ≥ P n (T (P )) = P n (xn )
xn ∈T (P )

= |T (P )|2−n[H(P )+D(P ||P )]

= |T (P )|2−nH(P ) .

For the lower bound we will use a Lemma.

Lemma: ∀P, Q ∈ Pn : P n (T (P )) ≥ P n (T (Q)).

Proof
n
P n (T (P )) |T (P )| a∈X P (a)[nP (a)]
Q

nP (1) ... nP (r) Y
= = n P (a)n[P (a)−Q(a)]
P n (T (Q)) |T (Q)| a∈X P (a)[nQ(a)]
Q

nQ(1) ... nQ(r) a∈X
Y (nQ(a))!
= P (a)n[P (a)−Q(a)]
a∈X
(nP (a))!

m!
Note: ≥ nm−n
n!
m!
If m > n, then = m(m − 1) . . . (n + 1) ≥ nm−n .
n!
 n−m
m! 1 1
If n > m, then = ≥ = nm−n .
n! n(n − 1) . . . (m + 1) n
Therefore,
Y (nQ(a))!
(nP (a))n[Q(a)−P (a)] P (a)n[P (a)−Q(a)]
Y
P (a)n[P (a)−Q(a)] ≥
a∈X
(nP (a))! a∈X
Y
= nn[P (a)−Q(a)]
a∈X
P
= nn a∈X
[P (a)−Q(a)]

= 1.

Proof PROOF OF LOWER BOUND:

X
1= P n (T (Q)) ≤ |Pn | max P n (T (Q))
Q
Q∈Pn

= |Pn |P n (T (P ))
= |Pn ||T (P )|2−n[H(P )+D(P ||P )]
≤ (n + 1)r−1 |T (P )|2−nH(P ) .

EE376A COURSE NOTES 47

Theorem 58. ∀P ∈ Pn , Q,
1
2−nD(P ||Q) ≤ Qn (T (P )) ≤ 2−nD(P ||Q) .
(n + 1)r

Proof Qn (T (P )) = |T (P )|2−n[H(P )+D(P ||Q)] .

Now bound |T (P )| as in previous theorem.

.
Note: We will write αn = βn : “equality to first order in the exponent"
1 αn n→∞
⇐⇒ log −−−→ 0
n βn
1 1 n→∞
⇐⇒ | log αn − log βn | −−−→ 0
n n
. n→∞
E.g. : αn = 2nJ ⇐⇒ αn = 2n(J+n ) where n −−−→ 0.

6.1.1 Recap on Types

Consider the sequence xn ∈ X n , where X is a finite alphabet. Let Pxn be the empirical distribution
and Pn the set of all empirical distributions over sequences of length n. Then we define the type
to be:

T (P ) = {xn : Pxn = P },

for P ∈ Pn . We have shown that:

1. |Pn | ≤ (n + 1)|x|

2. Qn (xn ) = 2−n[H(Pxn )+D(Pxn ||Q)]

.
3. 1
(n+1)|x|
2nH(p) ≤ T (P ) ≤ 2nH(p) , or equivalently, |T (P )| = 2nH(P ) , P ∈ Pn .
.
4. 1
(n+1)|x|
2−nD(Pxn ||Q) ≤ Q(T (P )) ≤ 2−nD(Pxn ||Q) , or equivalently, Qn (T (P )) = 2−nD(P ||Q) .

6.2 A Version of Sanov’s Theorem

Next we prove a version of Sanov’s Theorem, which bounds the probability that a function’s
empirical mean exceeds some value α.

EE376A COURSE NOTES 48

Theorem 59. Sanov’s Theorem. For sufficiently large n, we have

n
!
1 1X
2−n min D(P ||Q) ≤ f (Xi )) ≥ α ≤ (n + 1)|x| 2−n min D(P ||Q) ,
(n + 1)|x| n i=1
where the min is over the set {P : P ∈ Pn , hPxn , f i ≥ α}.
Proof First observe that we can write

n
1X 1 X
f (x) = N (a|xn )f (a)
n i=0 n a∈X
X
= Pxn (a)f (a)
a∈X
= hPxn , f i
Pn
where we have used the Euclidean inner product, defined as ha, bi := i=1 ai bi for a, b ∈ R. Then
by the Law of Large numbers,

n
1X
f (Xi ) ≈ EX∼Q f (X)
n i=1
X
= Q(a)f (a)
a∈X
= hQ, f i
We can proceed to find the desired upper bound.
n
!
1X
P f (Xi ) ≥ α = P (hPxn , f i)
n i=1
 
[
= Qn  T (P )
P ∈Pn ,hPxn ,f i≥α
X
= Qn (T (P ))
P ∈Pn ,hPxn ,f i≥α

≤ |Pn | max Qn (T (P ))
≤ (n + 1)|x| max 2−nD(P ||Q)
= (n + 1)|x| 2−n min D(P ||Q)
Now we solve for the lower bound:
n
!
1X
P f (Xi ) ≥ α ≥ max Qn (T (P ))
n i=1
1
≥ max |x|
2−nD(P ||Q)
(n + 1)
1
= 2−n min D(P ||Q)
(n + 1)|x|

EE376A COURSE NOTES 49

 Note that we are taking the min,max over the set {P : P ∈ ∗Pn , hPxn , f i ≥ α}. Therefore, we
1 Pn .
have, up to a polynomial in n, that P n i=1 f (Xi ) ≥ α = 2−nD (α) where D∗ (α) = min D(P ||Q).
This is exactly what we were looking for.

 
1
Example 60. Take Xi ∼ Ber 2 . Then:

n
!
1X . ∗
P (fraction of ones in X1 X2 · · · Xn ≥ α) = P Xi ≥ α = 2−nD (α)
n i=1

where D∗ (α) = min D(Ber(α)||Ber(1/2)). This gives:


0

 0 ≤ α ≤ 21
D∗ (α) = D(Ber(α)||Ber(1/2)) 12 < α ≤ 1

∞

α>1

and since
α 1−α
D(Ber(p)||Ber(1/2)) = α log + (1 − α) log
1/2 1/2
= 1 − h(α)

where h(·) is the binary entropy function. Thus we can write


0

 0 ≤ α ≤ 21
∗ 1
D (α) = 1 − h(α) 2 <α≤1

∞

α>1

Interestingly, this function explodes at α = 1, which makes sense because the probability that the
mean of random variables which take values up to 1 is greater than 1 is impossible. Furthermore,
we have a region where the cost of mismatch is zero, since we are guaranteed that one of the
probabilities is always going to be ≥ 1/2, so we would expect our mean to be so as well.

EE376A COURSE NOTES 50

Bibliography

[1] Cover, Thomas M., and Joy A. Thomas. Elements of information theory. John Wiley & Sons,
2012.

51
Chapter 7

Conditional and Joint Typicality

Notation:
We always write a sequence of symbols by small letter. For example xn is an individual sequence
without any probability distribution assigned to it. We use capital letter for random variables, e.g.
X n i.i.d. according to some distribution. Throughout, X will denote the set of possible values a
symbol can take.

Before going forward, we define empirical distribution.

Definition 61. For any sequence xn , empirical distribution is the probability distribution derived
for letters of the alphabet based on frequency of appearance of that specific letter in the sequence.
More precisely:
1X
pxn (a) = 1(xi = a), ∀a ∈ X (7.1)
n

7.1 Typical Set (again)

A rough intuition for typical sets is that if one picks a sequence from an i.i.d distribution, p ∼ X n ,
then the typical set T (X) is a set of length n sequences with the following properties:
1. A sequence chosen at random will be in the typical set with probability almost one.
2. All the elements of the typical set have (almost) equal probabilities.
More precisely, if T (X) is the typical set, |T (X)| ≈ 2nH(X) and probability of each sequence inside
the typical set is ∼ 2−nH(X) . So a random sequence chosen from the set looks like one chosen
uniformly from the typical set.

7.2 δ–strongly typical set

Definition 62. A sequence xn is said to be strongly δ typical with respect to the pmf P if,

∀a ∈ X : |Pxn (a) − P (a)| ≤ δP (a)

Where X is the support of the distribution.

52
 All(sequences(of(
length(n(

Typical(Set(

Figure 7.1: Space of all sequences and typical set inside.

Definition 63. The strongly δ-typical set, Tδ (P ), is the set of all strongly δ typical sequences. That
is

Tδ (P ) = {xn : |Pxn (a) − P (a)| ≤ δP (a)}

For reference, recall that the weakly -typical set is:

 
 1
 
n n

A (P ) = x : − log P (x ) − H(P ) ≤ 
 
n
1
Example 64. Consider the following extreme example where P (a) = |X | is uniform with a ∈ X .
So for all xn ,
1
P (xn ) =
|X |n
= 2−n log |X |
= 2−nH(p)
(n)
Therefore, for all  > 0, A (P ) = X n ! This makes sense because this is a uniform distribution so
you would always expect the typical sequence to include all possibilities regardless of n.

Note that this "strongly typical" set is different from the other "(weakly) typical" set defined
in previous chapters. This new notion is stronger, but as we will see it retains all the desirable
properties of typicality, and more. The following example illustrates difference of this strong notion
and weak notion defined earlier:

Example 65. Suppose that alphabet is X = {a, b, c} with the probabilities p(a) = 0.1, p(b) = 0.8
and p(c) = 0.1. Now consider two strings of length 1000:

x1000
strong = (100a, 800b, 100c)

x1000
weak = (200a, 800b)

In this example, these two sequences have same probability, so they are both identical in the weak
()
notion of typical set A1000 (for some ). But it is not hard to see that x1000
strong is a δ- strong typical

EE376A COURSE NOTES 53

set while the other is not (for sufficiently small δ). The strong notion is sensitive to frequency of
different letters and not only the total probability of sequence.

We will show in the homework that that we preserve important properties of typical sets in the
new strong notion. Specifically, we will show the following results:

1. ∀ δ > 0, there exists  = δH(p) such that Tδ (P ) ⊆ A (P ) (i.e., strong typical sets are inside
weak typical sets).

2. Empirical probability pX n is almost equal to the probability distribution p. Therefore p(xn ) ≈

2−nH(X) for xn in the strong typical set.

3. There exists (δ) such that for all n sufficiently large:

2n[H(P )−(δ)] ≤ |T (P )| ≤ 2n[H(P )+(δ)] ,

where (δ) → 0 as δ → 0. Thus |Tδ (X)| ≈ 2nH .

4. P (xn ∈ Tδ (X)) → 1 as n → ∞

7.3 δ–jointly typical set

In this section we extend the notion of δ - typical sets to pair of sequences xn = (x1 , ..., xn ) and
y n = (y1 , ..., yn ) from alphabets X and Y.

Definition 66. For xn = (x1 , x2 , . . . xn ) ∈ X n and y n = (y1 , y2 , . . . yn ) ∈ Y n , the joint empirical

distribution is defined as
1
Pxn ,yn (x, y) = |{i ∈ {1, . . . , n} : xi = x, yi = y}|
n
1
= N (x, y | xn , y n )
n
where we have defined N (x, y | xn , y n ) := |{i ∈ {1, . . . , } : xi = x, yi = y}|

Definition 67. A pair of sequences (xn , y n ) is said to be δ - jointly typical with respect to a pmf
PXY on X × Y if:
|Pxn ,yn (x, y) − P (x, y)| ≤ δP (x, y),
where Pxn ,yn (x, y) is the empirical distribution.

Then, we can make the following definition.

Definition 68. For (X, Y ) ∼ P , the jointly δ typical set is given by

Tδ (P ) = {(X n , Y n ) : |Pxn ,yn (x, y) − P (x, y)| < δP (x, y), ∀(x, y) ∈ X × Y}

If we look carefully, nothing is really very new. We just require that empirical distribution of
pair of sequences be δ - close to the pmf PXY .

EE376A COURSE NOTES 54

 (n) Qn
• Conditional Typicality Lemma: Let xn 2 T✏0 (X) and Y n ⇠ i=1 pY |X (yi |xi ).
Then for every ✏ > ✏0,
P{(xn, Y n) 2 T✏(n)(X, Y )} ! 1 as n ! 1
This follows by the LLN. Note that the condition ✏ > ✏0 is crucial to apply the
LLN (why?)
(n)
The conditional typicality lemma implies that for all xn 2 T✏0 (X)
|T✏(n)(Y |xn )| (1 ✏)2n(H(Y |X) (✏))
for n sufficiently large
Often as is the case with H(X) vs H(P ), we will (n) Tδ (X) for Tδ (P ) when P ∼ X and
write
• In fact, a stronger statement holds: For every xn 2 T✏ (X) and n sufficiently
δ (P ) when P ∼ (X, Y ).
Tδ (X, Y ) for Tlarge,
0
|T✏(n)(Y |xn )| 2n(H(Y |X) (✏))
,
The notionfor
of some
strong0(✏)
vs!weak
0 as typicality
✏!0 is important. For example, consider the random variable
1 Pn n
Gn = n i=1 g(x ). If X is strongly typical, then G is close to (the
Thisi can be proved by counting jointly typical ny n sequences
E [g(x)] for large n. On the other
method of
hand, this would not necessarily have been the case if X n were only weakly typical.
types [12]) as shown in the Appendix

It is easy to see that size of typical set is:

|Tδ (X, Y )| ≈ 2nH(X,Y )

and LNIT: Information Measures and Typical Sequences (2010-06-22 08:45) Page 2 – 19
P ((xn , y n ) ∈ Tδ (X, Y )) ≈ 2−nH(X,Y )
In Figure 7.2 we have depicted the sequences of length n from alphabet X on the x axis and
sequences from alphabet Y on the y axis.
Useful Picture
⇣ ⌘
.
T✏(n)(X) | · | = 2nH(X)

xn
yn

(n)
⇣T✏ (X, Y ) ⌘
.
T✏(n)(Y ) ⌘ | · | = 2nH(X,Y )
⇣
. nH(Y )
|·|=2

(n) n (n) n
⇣ T✏ (Y |x ) ⌘ ⇣ T✏ (X|y ) ⌘
. nH(Y |X) . nH(X|Y )
|·|=2 |·|=2

Figure 7.2: A useful diagram depicting typical sets, from El Gamal and Kim (Chapter 2)
LNIT: Information Measures and Typical Sequences (2010-06-22 08:45) Page 2 – 20

Then we can look inside the table and any point corresponds to a pair of sequences. We have
marked the jointly typical sets with dots. It is easy to see that if a set is jointly typical then both
of the sequences in the set are typical as well. Also we will see in the next section that number
of dots in the column corresponding to a typical xn sequence is approximately 2nH(Y |X) (a similar
statement is also correct for rows). We can quickly check that this is consistent by a counting
argument: we know that number of typical xn sequences is 2nH(X) and each column there are
2nH(Y |X) jointly typical pairs. So in total there are 2nH(X) · 2nH(Y |X) number of typical pairs. But
2nH(X) · 2nH(Y |X) = 2n(H(X)+H(Y |X)) = 2nH(X,Y ) which is consistent to the fact that total number
of jointly typical pairs is equal to 2nH(X,Y ) .

EE376A COURSE NOTES 55

7.4 δ–conditional typicality
In many applications of typicality, we know one sequence (say, the input to a channel) and want to
know typical values for some other sequence. In this case, a useful concept is conditional typicality.
Definition 69. Fix δ > 0, xn ∈ X n . Then, the conditional typical set Tδ (Y |xn ) is defined by
Tδ (Y |xn ) , {y n ∈ Y n : (xn , y n ) ∈ Tδ (X, Y )}.
As usual, it is useful to have bounds on the size of this set. For all δ 0 < δ (and n large enough,
.
as usual), xn ∈ Tδ0 (X) ⇒ |Tδ (Y |xn )| = 2nH(Y |X) (below, we will be a bit more careful, and we
will see that this exponent should depend on a function (δ) which vanishes as δ → 0). Note that
this asymptotic behavior is does not depend on the specific sequence xn , as long as xn is typical!
This is all in accordance with the intuition we have developed: all typical sequences behave roughly
similarly. If xn 6∈ Tδ (X), then |Tδ (Y |xn )| = 0, as (xn , y n ) cannot be jointly typical if xn is not
typical.
To illustrate the methods used to describe the asymptotic behavior of |Tδ (Y |xn )|, we find an
upper bound on this value. If xn ∈ Tδ0 (X) ⊆ Tδ (X) and y n ∈ Tδ (Y |xn ), then by definition of the
conditional typical set, (xn , y n ) ∈ Tδ (X, Y ). Since strong typicality implies weak typicality,
1
(1 − δ)H(X, Y ) ≤ − log p(xn , y n ) ≤ (1 + δ)H(X, Y ), (7.2)
n
1
(1 − δ)H(X) ≤ − log p(xn ) ≤ (1 + δ)H(X). (7.3)
n
n
So, fix some x ∈ Tδ0 (X). Then,
X X p(xn , y n )
1≥ p(y n |xn ) =
y n ∈Tδ (Y |xn ) y n ∈Tδ (Y |xn )
p(xn )

≥ |Tδ (Y |xn )| · 2−n(H(X,Y )−H(X)+(δ)) = |Tδ (Y |xn )| · 2−n(H(Y |X)+(δ))

⇒ |Tδ (Y |xn )| ≤ 2n(H(Y |X)+(δ)) .

7.5 Encoding – Decoding Schemes for Sending Messages

We now explain the idea of how these concepts relate to sending messages over channels. Consider
a discrete memoryless channel (DMC), described by conditional probability distribution p(y|x),
where x is the input to the channel and y is the output. Then,
n
Y
p(y n |xn ) = p(yi |xi ).
i=1

Say we want to send one of M messages over a channel. We encode each m ∈ {1, . . . , M } into a
codeword X n (m). We then send the codeword over the channel, obtaining Y n . Finally, we use a
decoding rule M̂ (Y n ) which yields M̂ = m with high probability.
The conditional typicality lemma (proved in the homework) characterizes the behavior of this
channel for large n: if we choose a typical input, then the output is essentially chosen uniformly at
random from Tδ (Y |xn ). More precisely, for all δ 0 < δ, xn ∈ Tδ0 (X) implies that
P (y n ∈ Tδ (Y |xn )) = P ((xn , Y n ) ∈ Tδ (X, Y )) → 1,

EE376A COURSE NOTES 56

as n → ∞; furthermore, the probability of obtaining each element of Tδ (Y |xn ) is essentially
1
≈ 2−nH(Y |X) .
|Tδ (Y |xn )|
This lemma limits the values of M for which there is an encoding – decoding scheme that can
succeed with high probability. As illustrated in Figure 7.3, what we are doing is choosing a typical
codeword xn ∈ Tδ (X), and receiving some element Y n ∈ Tδ (Y |xn ). We can think of this latter set
as a “noise ball": it is the set of outputs y n that we could typically expect to receive, given that
our input is xn . If these noise balls corresponding to different inputs overlap significantly, then we
have no hope for being able to obtain m from Y n with high probability, as multiple inputs give
indistinguishable outputs. Since, for any input, the output will (with high probability) be typical
– that is, Y n ∈ Tδ (Y ), the number of messages we can send is limited by the number of noise balls
we can fit inside of Tδ (Y ). Since the number of elements of Tδ (Y ) is (approximately) 2nH(Y ) and
the number of elements of Tδ (Y |xn ) is (approximately) 2nH(Y |X) , it follows that the number of
messages we can send over this channel is at most

2nH(Y )
= 2n(H(Y )−H(Y |X)) = 2nI(X;Y ) ≤ 2nC ,
2nH(Y |X)
where C is the channel capacity. Note that this argument does not give a construction that lets
us attain this upper bound on the communication rate. The magic of the direct part of Shannon’s
channel coding theorem is that random coding lets us attain this upper bound.

All(sequences(of(
length(n(
All(sequences(of(
length(n( Typical(Set(

Typical(Set(
Tδ(Y|xn(2))(
xn(2)(

xn(1)( Tδ(Y|xn(1))(

Figure 7.3: The typical sets Tδ (X), Tδ (Y ), and Tδ (Y |xn ).

EE376A COURSE NOTES 57

7.6 Joint Typicality Lemma
In this final section, we discuss the joint typicality lemma, which tells us how well we can guess the
output without knowing the input. Intuitively, if X and Y are strongly correlated, then we might
expect that not knowing the input could strongly impair our ability to guess the output, and if X
and Y are independent then not knowing the input should not at all impair our ability to guess
the output. So, say that the actual input is xn . We will look for a bound on the probability that
an output will be in the conditional typical set Tδ (Y |xn ) – that is, the probability that we’ll guess
that xn was the input – in terms of the mutual information I(X; Y ).
Fix any δ > 0 and δ 0 < δ, and fix xn ∈ Tδ0 (X). Choose Ỹ n ∈ Y n by choosing each Ỹi i.i.d.
according to the marginal distribution p(y) (so, intuitively we’ve forgotten what we sent as input
to the channel, and are simulating the output). Then, noting that

y n ∈ Tδ (Y |xn ) ⇒ y n ∈ Tδ (Y ) ⇒ p(y n ) ≤ 2−n(H(Y )−(δ)) ,

where (δ) is a function that approaches 0 as δ → 0, we have

P (Ỹ n ∈ Tδ (Y |xn )) = P ((xn , Ỹ n ) ∈ Tδ (X, Y ))

X
= p(y n )
y n ∈T δ (Y |xn )

≤ |Tδ (Y |xn )| · 2−n(H(Y )−(δ))

≤ 2nH(Y |X)+(δ) · 2−n(H(Y )−(δ))
= 2−n(H(Y )−H(Y |X)−˜(δ))
= 2−n(I(X;Y )−˜(δ)) ,

where ˜(δ) → 0 as δ → 0.

Intuitive argument for joint typicality lemma The joint typicality lemma asserts that
the probability of observing two random xn and y n sequences is roughly 2−nI(X;Y ) . Observe that
there are roughly 2nH(X) typical xn sequences, and 2nH(Y ) typical y n sequences. The total number
of jointly typical sequences is 2nH(X,Y ) . Thus, what is the probability that two randomly chosen
sequences are jointly typical?

2nH(X,Y )
≈ nH(X) nH(Y )
= 2−nI(X;Y ) (7.4)
2 ×2

EE376A COURSE NOTES 58

Chapter 8

Lossy Compression & Rate Distortion

Theory

8.1 Definitions and main result

U N = U1 , . . . , U N J ∈ {1, 2, . . . , M } V N = V1 , . . . , V N
Encoder Decoder

where Ui ∼ U , i.i.d.

A scheme is characterized by:

• N, M

• An encoder, i.e., a mapping from U N to J ∈ {1, 2, ..., M } (log M bits used to encode a symbol
sequence, where a symbol sequence is U N and a symbol is Ui )

• A decoder, i.e., a mapping from J ∈ {1, 2, ..., M } to V N

In working with lossy compression, we examine two things:

log(M ) bits
1. Rate R = N source symbol

N
2. Expected distortion (figure of merit) = d(U N , V N ) = E[ N1
P
d(Ui , Vi )] (we always specify
i=1
distortion on a per-symbol basis, and then average the distortions to arrive at d(U N , V N ))
distortion
There’s a trade-off between rate and symbol . Distortion theory deals with this trade-off.
log M
Definition 70. (R,D) is achievable if ∀ > 0 ∃ scheme (N ,M ,encoder,decoder) such that N ≤
R +  and E[d(U N , V N )] ≤ D + 
Definition 71. R(D) , inf{R0 : (R0 , D) is achievable}
Definition 72. R(D)(I) , min I(U ; V )
E[d(U,V )]≤D

59
Theorem 73. R(D) = R(I) (D).

Proof
(
Direct Part : R(D) ≤ R(I) D
⇔
Converse Part : R(D) ≥ R(I) D
The proof of the direct part and the converse part are given below.

Note that R(D) is something we can’t solve for (solution space is too large!), but R(I) (D) is
something we can solve for (solution space is reasonable).

Theorem 74. R(D) is convex, i.e.,

∀0 < α < 1 , D0 , D1 : R(αD0 + (1 − α)D1 ) ≤ αR(D0 ) + (1 − α)R(D1 )

Sketch of proof: We consider a “time-sharing” scheme for encoding N bits. We encode the
first αN bits using a “good" scheme for distortion D = D0 and encode the last (1 − α)N bits
using a “good” scheme for D = D1 . Overall, the number of bits in the compressed message is
N αR(D0 ) + N (1 − α)R(D1 ), so that the rate is αR(D0 ) + (1 − α)R(D1 ). Further, the expected
distortion is the average, weighted by α between the distortions between the two different schemes,
i.e. αD0 + (1 − α)D1 . We therefore have constructed a scheme which achieves distortion αD0 +
(1 − α)D1 with rate αR(D0 ) + (1 − α)R(D1 ), and the optimal scheme can only do better. That is

R(αD0 + (1 − α)D1 ) ≤ αR(D0 ) + (1 − α)R(D1 ),

as desired.

8.2 Examples
Example 75.
1
Consider U ∼ Ber(p), p ≤ 2 and Hamming distortion. That is
(
0 for u = v
d(u, v) =
1 for u 6= v

Claim: (
h2 (p) − h2 (D) 0≤p≤D
R(D) =
0 D>p
Proof: We will not be overly pedantic by worrying about small  factors in the proof.
Note we can achieve distortion p without sending any information by setting V = 0. Therefore,
for D > p, R(D) = 0, as claimed. For the remainder of the proof, therefore, we assume D ≤ p ≤ 21 .

EE376A COURSE NOTES 60

 Consider U, V such that U ∼ Ber(p) and Ed(U, V ) = P (U 6= V ) ≤ D. We can show that R(D)
is lower-bounded by h2 (p) − h2 (D) by noting

I(U ; V ) = H(U ) − H(U |V )

= H(U ) − H(U 2 V |V )
≥ H(U ) − H(U 2 V )
= h2 (p) − h2 (P (U 6= V ))
≥ h2 (p) − h2 (D)

In the second line we have used the fact that H(U |V ) = H(U 2 V |V ) because there is a one to one
mapping (U, V ) ↔ (U 2 V, V ). In the third line, we have used that conditioning reduces entropy,
so H(U 2 V |V ) ≤ H(U 2 V ). Finally, in the last line we have used that h2 is increasing on [0, 12 ]
and that P (U 6= V ) ≤ D ≤ p ≤ 12 . This establishes that R(D) ≥ h2 (p) − h2 (D).
Now we must show equality can be achieved. The first and second inequalities above demon-
strate that we get equality if and only if

1. U 2 V is independent of V .

2. U 2 V ∼ Ber(D).

Z ∼ Ber(D)

V ∼ Ber(q) + U ∼ Ber(p)

Denoting U 2 V , Z, this is equivalent to finding q such that if V ∼ Ber(q) and Z ∼ Ber(D)

is independent of V , U = V ⊕2 Z ∼ Ber(p). Because V ⊕2 Z is binary, it is Bernoulli, with

p = P (U = 1)
= P (V = 1)P (Z = 0) + P (V = 0)P (Z = 1)
= q(1 − D) + (1 − q)D

Solving for q gives

p−D
q=
1 − 2D
Because D ≤ p ≤ 12 , both the numerator and denominator are positive. Further, because p ≤ 21 ,
we have q ≤ 1/2−D 1
1−2D = 2 , which shows that q is a valid probability. This completes the proof.

Example 76. Consider U ∼ N(0, σ 2 ) and distortion given by: d(u, v) = (U − V )2

Claim: (
1
log((σ 2 )/D) 0 ≤ D ≤ p
R(D) = 2
0 D > σ2

EE376A COURSE NOTES 61

 Proof: First note we may achieve distortion σ 2 without transmitting any information by setting
V = 0 with certainty. Therefore, R(D) = 0 for D > σ 2 . For the remainder of the proof, therefore,
we assume that D ≤ σ 2 .
For any U ,V such that U ∼ N(0, σ 2 ) and E(U − V )2 ≤ D, we assume D ≤ σ 2 .
We can find the lower-bound by noting

I(U ; V ) = h(U ) − h(U |V )

= h(U ) − h(U − V |V )
≥ h(U ) − h(U − V )
≥ h(U ) − h(N (0, D))
1 1
= log 2πEσ 2 − log 2πeD
2 2
1 σ2
= log
2 D
For the first inequality we have used that conditioning reduces even the differential entropy, and
in the second inequality we have used the result, proved earlier in the course that the maximum
differential entropy of a distribution constrained by Var(U − V ) ≤ D is achieved when U − V ∼
2
N(0, D). This establishes that R(D) ≥ 21 log σD .
Now we must show that equality can be achieved. The first and second inequalities above
demonstrate that we get equality if and only if
1. U − V is independent of V .

2. U − V ∼ N(0, D).

Z ∼ N (0, D)

V + U ∼ N (0, σ 2 )

Denoting U − V , Z, we want to find a distribution for V such that Z independent of Z and

distributed N(0, D) makes V + Z ∼ N(0, σ 2 ). We see that this is possible for V ∼ N(0, σ 2 − D),
2
which is a valid distribution because D ≤ σ 2 . This completes the proof, and R(D) = 21 log σD .

8.3 Proof of Direct Part R(D) ≤ R(I) (D)

8.3.1 An Equivalent Statement
First, we are going to show the equivalence of the following statements

R(D) ≤ R(I) (D) ⇐⇒ R(D) ≤ min {I(U ; V ) : U, V s.t. E d(U, V ) ≤ D}

⇐⇒ If U, V s.t. E d(U, V ) ≤ D, then R(D) ≤ I(U ; V )
⇐⇒ If U, V s.t. E d(U, V ) ≤ D, then (R, D) is achievable for any R > I(U ; V ).

EE376A COURSE NOTES 62

Proof The first and second lines follow the definition of R(I) (D). For the last line, it only suffices
to show
R(D) ≤ I(U ; V ) ⇐⇒ (R, D) is achievable for any R > I(U ; V ).

• For the ⇒ part, consider any R > I(U ; V ),

R > I(U ; V ) ≥ R(D) = inf{R0 : (R0 , D) is achievable}

thus (R, D) is achievable.

• For the ⇐ part, consider some R00 = I(U ; V ) + . By the assumption (R, D) is achievable for
any R > I(U ; V ), implying that (R00 , D) is achievable, and thereafter

R(D) = inf{R0 : (R0 , D) is achievable} ≤ R00 = I(U ; V ) + .

Since  can be arbitrarily small, we must have R(D) ≤ I(U ; V ).

Hence we can prove the equivalent statement instead of R(D) ≤ R(I) (D). That’s to show

(R, D) is the achievable for fixed U, V s.t. E[d(U, V )] ≤ D and fixed R > I(U ; V ).

8.3.2 Two Useful Lemmas

The proof of the equivalent statement uses two lemmas appearing in the homeworks.Let’s recall
them in advance.
i.i.d.
Lemma 77. (Joint Typicality Lemma) Suppose un ∈ Tδ0 (U ), 0 < δ 0 < δ and Vi ’s ∼ V ,

2−n(I(U ;V )+(δ)) ≤ P ((un , V n ) ∈ Tδ (U, V ))

for sufficiently large n and some (δ) > 0 where limδ→0 (δ) = 0.

Lemma 78. (Typical Average Lemma)

n
1X
(un , v n ) ∈ Tδ (U, V ) =⇒ d(un , v n ) , d(ui , vi ) ≤ (1 + δ) E d(U, V )
n i=1

8.3.3 Proof of the Equivalent Statement

For fixed U, V s.t. E d(U, V ) ≤ D and R > I(U ; V ), we are going to show (R, D) is achievable.

Proof Take M = b2nR c. Denote by Cn = {V n (1), V n (2), ..., V n (M )} the random codebook
i.i.d.
which is generated by Vi ’s ∼ V and independent of U . Let d(un , Cn ) = minV n ∈Cn d(un , V n ).

EE376A COURSE NOTES 63

For sufficient small 0 < δ 0 < δ which appear in Lemma 77, the assumption R > I(U, V ) implies
R > I(U ; V ) + (δ). For any un ∈ Tδ0 (U ) and sufficiently large n,

P (d(un , Cn ) > D(1 + δ)) = P (d(un , Vn (i)) > D(1 + δ) for i = 1, 2, ..., M )
(Definition of d(un , Cn ))
i.i.d.
= P (d(un , Vn (1)) > D(1 + δ))M (Vi ∼ V )
≤ P (d(un , Vn (1)) > E d(U, V )(1 + δ))M (Assumption of E d(U, V ) ≤ D)
≤ P ((un , Vn (1)) 6∈ Tδ (U, V ))M (Inverse-negative of Lemma 78)
= [1 − P ((un , Vn (1)) ∈ Tδ (U, V ))]M
h iM
≤ 1 − 2−n(I(U ;V )+(δ)) (Lemma 77 with un ∈ Tδ0 (U ) and large n)
 
≤ exp −M · 2−n(I(U ;V )+(δ)) (1 − x ≤ e−x )

So far, we have an upper bound of P (d(un , Cn ) > D(1 + δ)) for any un ∈ Tδ0 (U ) and sufficiently
large n.  
P (d(un , Cn ) > D(1 + δ)) ≤ exp −M · 2−n(I(U ;V )+(δ)) (8.1)
i.i.d.
Then for Ui ∼ U ,
X
P (d(U n , Cn ) > D(1 + δ)) = P (d(un , Cn ) > D(1 + δ), U n = un )
un ∈Tδ0 (U )
X
+ P (d(un , Cn ) > D(1 + δ), U n = un )
un 6∈Tδ0 (U )
X
≤ P (d(un , Cn ) > D(1 + δ)) P (U n = un )
un ∈Tδ0 (U )
(U n independent of Cn )
+ P (U n 6∈ Tδ0 (U ))
 
≤ exp −M · 2−n(I(U ;V )+(δ)) + P (U n 6∈ Tδ0 (U ))
(Upper bound in Eq. 8.1)

where the first term goes to 0 as n → ∞ because

M = b2nR c, R > I(U ; V ) + (δ),

i.i.d.
and the second term goes to 0 as n → ∞ because of AEP. Thus for Ui ∼ U ,

P (d(U n , Cn ) > D(1 + δ)) → 0 as n → ∞ (8.2)

Further, let d(Cn ) = E (d(U n , Cn )|Cn ) be the average distortion by random codebook Cn , and thus
d(cn ) = E (d(U n , cn )|Cn = cn ) = E (d(U n , cn )) (Cn is independent of U n ) is the average distortion

EE376A COURSE NOTES 64

by a realization cn of Cn .

E d(Cn ) = E [E (d(U n , Cn )|Cn )]

= E (d(U n , Cn )) (tower property)
≤ P (d(U n , C n ) > D(1 + δ)) Dmax (Dmax , maxu∈U ,v∈V d(u, v))
n n
+ P (d(U , C ) ≤ D(1 + δ)) D(1 + δ)
→ D(1 + δ) as n → ∞ (Limiting result in Eq. 8.2)

It implies that
E d(Cn ) < D + 2δDmax for sufficiently large n,
which further implies existence of cn , a realization of Cn , satisfying

d(cn ) ≤ E d(Cn ) < D + 2δDmax for sufficiently large n.

Taking arbitrarily small δ and sufficiently large n, we can get the average distortion d(cn ) arbitrarily
close to D. And the size of codeword lists

|cn | = M = b2nR c ≤ 2nR .

(R, D) is achieved by the codebook cn .

8.4 Proof of the converse

Proof
Fix a scheme satisfying E[d(U N , V N )] ≤ D, then H(V N ) ≤ log M for V N taking M different

EE376A COURSE NOTES 65

values.

log M ≥ H(V N )
≥ H(V N ) − H(V N | U N )
= I(U N ; V N )
= H(U N ) − H(U N | V N )
N
X
= H(Ui ) − H(Ui | U i−1 , V N ) (by chain rule)
i=1
N
X
≥ H(Ui ) − H(Ui | U i−1 , Vi ) (conditioning reduces entropy)
i=1
N
X
= I(Ui ; Vi )
i=1
XN
≥ R(I) (E[d(Ui , Vi )]) (by definition ofR(I) (D))
i=1
N
X 1
=N R(I) (E[d(Ui , Vi )]) (average ofR(I) (D)over all i)
i=1
N
N
1 X
≥ N R(I) ( E[d(Ui , Vi )]) (By the convexity of R(I) (D))
N i=1
≥ N R(I) (D) (R(I) (D) is nonincreasing)
log M
rate = ≥ R(I) (D)
N

8.5 Geometric Interpretation

I(U ; V ) is the expected distortion if both U, V are in jointly typical set, as we just proved. The
following figures will give a geometric interpretation to the results.

How large does a codebook has to be so that every source sequence in the typical set has a
reconstruction, which it is jointly typical? Let T (U | V N (i)) be the set of source sequences jointly
typical with the reconstruction sequence V N (i). Therefore, to cover every source sequence, we need
a codebook of at least the size of typical set of the input divided by the number of source sequences
one reconstruction can cover.
|T (U )| 2N H(U )
The size of the codebook = ≈ = 2N I(U ;V ) . This is showed in Fig. 8.1 on
|T (U | V N (i))| 2N H(U |V )
the distortion function.

EE376A COURSE NOTES 66

 Figure 8.1: Distortion Function

Achievability: generate V N (i) i.i.d. ∼ V, P ((U N , V N (i)) ∈ T (U, V )) ≈ 2−nI(U ;V ) . Therefore, in

order for all typical sequences to be described, P ((U N , V N (i)) ∈ T (U, V ), i = 1, 2, · · · , M ) ≈ 1 if
the codebook is sufficiently large, i.e., if R > I(U ; V ), as the codebook is of size b2N R c

Figure 8.2: Communication Channel

The communication problem has a similar setup. In order to achieve reliable communication,
|T (Y )| 2nH(Y )
the number of messages ≤ = |X)
= 2−nI(X;Y ) . This communication channel
|T (Y | X n (i))| 2nH(Y

is shown in Fig. 8.2, as the distortion occurs on V N .

Achievability: ∀i s.t. P (Y n ∈ T (Y n | X n (i)|i 6= j) ≈ 2−nI(X;Y ) . Therefore, because the

number of messages is b2N R c, in order to guarantee that P (Y n ∈ T (Y n | X n (i))for any i 6= j) ≈ 0,
R < I(X; Y ).

EE376A COURSE NOTES 67

Chapter 9

Joint Source Channel Coding

9.1 Joint Source Channel Coding

Now that we understand lossy compression as well as the communication problem, we can combine
them into a joint source-channel coding theorem. A schematic of this setup is shown below:

Ui iid ∼ U
U N = (U1 , . . . , UN ) X n Memoryless Channel Yn V N = (V1 , . . . , VN )
Transmitter PY |X Receiver

With this channel description, the goal is to communicate the U N = (U1 , U2 , . . . ,hUN ) through
i
the memoryless channel given by PY |X with small expected distortion, measured by E d(U N , V N ) .
In other words, the goal is to find the best possible distortion given some rate and some noise during
transmission. Note that the Ui are not necessarily bits.

The rate of communication is then

N source symbols
rate =
n channel use
h i
We also allow an expected distortion E d(U N , V N ) , with

N
1 X
d(U N , V N ) = d (Ui , Vi ) .
N i=1

N
Definition 79. A rate-distortion pair (ρ, D) is achievable if ∀ > 0, ∃ a scheme with ≥ ρ−
n
and E[d(U N , V N )] ≤ D + .

68
 Note: under any scheme, E[d(U N , V N )] ≤ D, and U N → X n → Y n → V N forms a Markov
chain. Therefore,
nC ≥ I(X n ; Y n ) (proven in channel coding converse theorem)
I(X n ; Y n ) ≥ I(U N ; V N )) (Data processing inequality)
I(U N ; V N ) ≥ N R(D) (proven in converse of rate distortion theorem)
N R(D)
= Rate · R(D) ≤ C.
n
Thus, if (ρ, D) is achievable ⇒ ρR(D) ≤ C.

Consider the following “Separation Scheme"

UN N ·R(D) bits Xn
−−→ Good Distortion Compressor −−−−−−−→ Reliable Channel Encoder −−→ Memoryless Channel

YN N ·R(D) bits VN
−−→ Reliable Channel Decoder −−−−−−−→ Good Distortion Compressor −−→

All these pieces work correctly to ensure that distortion and channel noise are handled properly.

N
It is guaranteed that E[d(U N , V N )] ≈ D provided that n · C ≥ N R(D) · C ≥ · R(D) =
n
rate · R(D). Thus, if R(D) ≤ C, then (ρ, D) is achievable.

9.2 Source – Channel Separation Theorem

Theorem 80. (ρ, D) is achievable if and only if ρ · R(D) ≤ C.
Essentially, we have separated the problem of compression from the problem of transmission
and have proven that a separated solution is optimal. There is no need nor advantage to address
both problems simultaneously.
ρ

not achievable

C
R(D)

C
R(D0 )

achievable

D0 Dmax D
We can achieve points on the curve above by first thinking about representing the data as bits in
an efficient manner (compression) with rate R(D) and then transmitting these bits losslessly across
the channel with rate C. Note that the distortion without sending anything over the channel is
Dmax .

EE376A COURSE NOTES 69

9.3 Examples
Example 81. Binary Source and Binary Channel
Source: U ∼ Ber(p), 0 ≤ p ≤ 1/2
Channel: BSC(q), 0 ≤ q ≤ 1/2
Distortion: Hamming
Recall that for U ∼ Ber(p), the rate distortion function is R(D) = h2 (p) − h2 (D) and that a
binary symmetric channel with crossover probability q has capacity C = 1 − h2 (q)
So, we see that if we want distortion ≤ D, then (for D ≤ p) the maximum achievable rate is:

1 − h2 (q)
ρ=
h2 (p) − h2 (D)

1−h2 (q)
h2 (p)−h2 (D)

not achievable

1−h2 (q)
h2 (p)
achievable

p D
Note that the communication problem corresponds to D = 0.
In particular, if p = 1/2, then if we want distortion ≤ D, the maximum rate we can transmit
at is:
1 − h2 (q)
ρ=
1 − h2 (D)
ρ

1−h2 (q)
1−h2 (D)
not achievable

1
1 − h2 (q)
achievable

D=q 1 D
2

Consider the following scheme for rate=1:

Channel input: Xi = Ui
reconstruction: Vi = Yi

EE376A COURSE NOTES 70

 The expected distortion is then P (Ui 6= Vi ) = P (Xi 6= Yi ) = q
−→ this scheme is optimal, since ρ = (1 − h2 (q))/(1 − h2 (D = q)) = 1.
In this particular case, it is possible to achieve the optimal rate using a scheme that individually
encodes and transmits each symbol.

Example 82. Gaussian Source and Gaussian Channel

Source: U ∼ N (1, σ 2 )
Channel: AWGN (Additive White Gaussian Noise Channel) with power constraint P

Z ∼ (0, 1)

X + Y

distortion: squared error

2
Recall that for U ∼ N (1, σ 2 ), the rate distortion function is R(D) = 12 log( σD ) (for 0 ≤ D ≤ σ 2 )
and that the AWGN channel with power constraint P has capacity C = 12 log(1 + P )
ρ

log(1+P )
log(σ 2 /D)

not achievable

1 achievable
σ2 σ2 D
P +1

Then, for a given distortion D ≤ σ 2 , the maximum achievable rate is

log(1 + P )
ρ=
log(σ 2 /D)
Consider the following scheme at rate=1:
q
P
transmit: Xi = U
σ2 i q
receive: Yi = Xi + Zi = σP2 Ui + Zi
reconstruction: Vi = E[Ui |Vi ]

The distortion is squared error, so we know that reconstruction using the expected value is
optimal. Thus, we take Vi = E[Ui |Vi ].
The expected distortion is then:

EE376A COURSE NOTES 71

 E[Ui |Vi ] = Var(Ui |Vi )
 s 
 σ2

= Var Ui  Yi 
 P
  s 
2

 σ
= Var Ui Ui + Zi 
 P
(a)σ 2 (σ 2 /P )
=
σ 2 + σ 2 /P
σ2
=
P +1

where (a) follows from the fact that for X ∼ N (0, σ12 ) independent from Y ∼ N (0, σ22 ):

σ12 σ22
Var(X|X + Y ) =
σ12 + σ22

Now, at rate = 1:
The optimal D satisfies

log(1 + P )
=1
log(σ 2 /D)
σ2
→1+P =
D
So, in the specific case of rate = 1 we see that the simple scheme above is optimal, just as the
simple scheme for the Binary Source and Channel was also optimal when rate = 1.

EE376A COURSE NOTES 72