C. A.

Bouman: Digital Image Processing - April 17, 2013

Types of Coding
Source Coding - Code data to more efficiently represent
the information
Reduces size of data
Analog - Encode analog source data into a binary format
Digital - Reduce the size of digital source data
Channel Coding - Code data for transmition over a noisy
communication channel
Increases size of data
Digital - add redundancy to identify and correct errors
Analog - represent digital values by analog signals
Complete Information Theory was developed by Claude
Shannon

C. A. Bouman: Digital Image Processing - April 17, 2013

Digital Image Coding

Images from a 6 MPixel digital cammera are 18 MBytes
each
Input and output images are digital

Output image must be smaller (i.e. 500 kBytes)

This is a digital source coding problem

C. A. Bouman: Digital Image Processing - April 17, 2013

Two Types of Source (Image) Coding

Lossless coding (entropy coding)
Data can be decoded to form exactly the same bits
Used in zip
Can only achieve moderate compression (e.g. 2:1 3:1) for natural images
Can be important in certain applications such as medical imaging
Lossly source coding
Decompressed image is visually similar, but has been
changed
Used in JPEG and MPEG
Can achieve much greater compression (e.g. 20:1 40:1) for natural images
Uses entropy coding

C. A. Bouman: Digital Image Processing - April 17, 2013

Entropy

Let X be a random variables taking values in the set {0, , M

1} such that
pi = P {X = i}
Then we define the entropy of X as
H(X) =

M
1
X

pi log2 pi

i=0

= E [log2 pX ]
H(X) has units of bits

C. A. Bouman: Digital Image Processing - April 17, 2013

Conditional Entropy and Mutual Information

Let (X, Y ) be a random variables taking values in the set
{0, , M 1}2 such that
p(i, j) = P {X = i, Y = j}
p(i, j)

p(i|j) = PM 1
k=0

p(k, j)

Then we define the conditional entropy of X given Y as

H(X|Y ) =

1
M
1 M
X
X

p(i, j) log2 p(i|j)

i=0 j=0

= E [log2 p(X|Y )]
The mutual information between X and Y is given by
I(X; Y ) = H(X) H(X|Y )
The mutual information is the reduction in uncertainty of
X given Y .

C. A. Bouman: Digital Image Processing - April 17, 2013

Entropy (Lossless) Coding of a Sequence

Let Xn be an i.i.d. sequence of random variables taking
values in the set {0, , M 1} such that
P {Xn = m} = pm
Xn for each n is known as a symbol
How do we represent Xn with a minimum number of bits
per symbol?

C. A. Bouman: Digital Image Processing - April 17, 2013

A Code
Definition: A code is a mapping from the discrete set of
symbols {0, , M 1} to finite binary sequences
For each symbol, m their is a corresponding finite binary sequence m
|m| is the length of the binary sequence

Expected number of bits per symbol (bit rate)

n
= E [|Xn |]
=

M
1
X
m=0

|m| pm

Example for M = 4
m
0
1
2
3

m
|m|
01
2
10
2
0
1
100100 6

Encoded bit stream

(0, 2, 1, 3, 2) (01|0|10|100100|0)

C. A. Bouman: Digital Image Processing - April 17, 2013

Fixed versus Variable Length Codes

Fixed Length Code - |m| is constant for all m

Variable Length Code - |m| varies with m

Problem

Variable length codes may not be uniquely decodable

Example: Using code from previous page
(6) (100100)
(1, 0, 2, 2) (10|01|0|0)
Different symbol sequences can yield the same code
Definition: A code is Uniquely Decodable if there exists
only a single unique decoding of each coded sequence.
Definition: A Prefix Code is a specific type of uniquely
decodable code in which no code is a prefix of another
code.

C. A. Bouman: Digital Image Processing - April 17, 2013

Lower Bound on Bit Rate

Theorem: Let C be a uniquely decodable code for the
i.i.d. symbol sequence Xn. Then the mean code length is
greater than H(Xn).

n
= E [|Xn |]
=

M
1
X
m=0

|m| pm

H(Xn)
Question: Can we achieve this bound?
Answer: Yes! Constructive proof using Huffman codes

C. A. Bouman: Digital Image Processing - April 17, 2013

Huffman Codes
Variable length prefix code Uniquely decodable
Basic idea:

Low probability symbols Long codes

High probability symbols short codes

Basic algorithm:

Low probability symbols Long codes

High probability symbols short codes

10

C. A. Bouman: Digital Image Processing - April 17, 2013

11

Huffman Coding Algorithm

1. Initialize list of probabilities with the probability of each
symbol
2. Search list of probabilities for two smallest probabilities,
pk and pl.
3. Add two smallest probabilities to form a new probability,
pm = pk + pl.
4. Remove pk and pl from the list.
5. Add pm to the list.
6. Go to step 2 until the list only contains 1 entry

C. A. Bouman: Digital Image Processing - April 17, 2013

12

Recursive Merging for Huffman Code

Example for M = 8 code
p0

p1

p2

p3

p4

p5

p6

p7

0.4

0.08

0.08

0.2

0.12

0.07

0.04

0.01

0.4

0.08

0.08

0.2

0.12

0.07

0.4

0.08

0.08

0.2

0.12

0.12

0.12

0.12

0.4

0.16

0.2

0.4

0.16

0.2

0.4

0.24

0.36

0.4

0.24

0.6

1.0

0.05

C. A. Bouman: Digital Image Processing - April 17, 2013

13

Resulting Huffman Code

Binary codes given by path through tree
p0

p1

p2

p3

p4

p5

p6

0111

0110

010

001

0001

p7

00001 00000
1

0
root

C. A. Bouman: Digital Image Processing - April 17, 2013

Upper Bound on Bit Rate of Huffman Code

Theorem: For a Huffman code, n
has the property that
H(Xn) n
< H(Xn) + 1
A Huffman code is within 1 bit of optimal efficiency
Can we do better?

14

C. A. Bouman: Digital Image Processing - April 17, 2013

15

Coding in Blocks
We can code blocks of symbols to achieve a bit rate that
approaches the entropy of the source symbols.
,X
, Xm1}, X
| m, {z, X2m1},
| 0, {z
Y0

Y1

So we have that

Yn = Xnm, , X(n+1)m1
where

Yn {0, , M m 1}

C. A. Bouman: Digital Image Processing - April 17, 2013

16

Bit Rate Bounds for Coding in Blocks

It is easily shown that H(Yn) = mH(Xn) and the number
n

y is the
of bits per symbol Xn is given by n
x = my where n
number of bits per symbol for a Huffman code of Yn.
Then we have that
H(Yn) n
y < H(Yn) + 1
1
H(Yn)
m

1
1
H(Yn) +
m
m

n
y
m

<

n
y
m

< H(Xn) +

1
m

H(Xn) n
x < H(Xn) +

1
m

H(Xn)

As the block size grows, we have

1
m m

lim H(Xn) lim n

x H(Xn) + lim

H(Xn) lim n
x H(Xn)
m

So we see that for a Huffman code of blocks with length

m
lim n
x = H(Xn)
m

C. A. Bouman: Digital Image Processing - April 17, 2013

17

Comments on Entropy Coding

As the block size goes to infinity the bit rate approaches
the entropy of the source
x = H(Xn)
lim n

A Huffman coder can achieve this performance, but it requires a large block size.
As m becomes large M m becomes very large large
blocks are not practical.
This assumes that Xn are i.i.d., but a similar result holds
for stationary and ergodic sources.
Arithmetic coders can be used to achieve this bitrate in
practical situations.

C. A. Bouman: Digital Image Processing - April 17, 2013

18

Run Length Coding

In some cases, long runs of symbols may occur. In this
case, run length coding can be effective as a preprocessor
to an entropy coder.
Typical run length coder uses
, (value, # of repetitions), (value, # of repetitions+1),
where 2b is the maximum number of repetitions

Example: Let Xn {0, 1, 2}

| 0000000
111 | 222222
| {z } | |{z}
| {z } |
07

13

26

If more than 2b repetitions occur, then the repetition is broken into segments
111 |
00 | |{z}
| 00000000
| {z } | |{z}
08

02

Many other variations are possible.

13

C. A. Bouman: Digital Image Processing - April 17, 2013

19

Predictive Entropy Coder for Binary Images

Uses in transmission of Fax images (CCITT G4 standard)
Framework

Let Xs be a binary image on a rectangular lattice s =

(s1, s2) S
Let W be a causal window in raster order
Determine a model for p(xs|xs+r r W )

Algorithm

1. For each pixel in raster order

(a) Predict

s = 1 if p(1|Xs+r r W ) > p(0|Xs+r r W )
X
0 otherwise
s send 0; otherwise send 1
(b) If Xs = X
2. Run length code the result
3. Entropy code the result

C. A. Bouman: Digital Image Processing - April 17, 2013

20

Predictive Entropy Coder Flow Diagram

Encoder
Xs

XOR

Run Length
Encoding

Huffman
Coding

Causal
Predictor

Decoder
Huffman
Decoding

Run Length
Decoding

XOR

Xs
Causal
Predictor

C. A. Bouman: Digital Image Processing - April 17, 2013

21

How to Choose p(xs|xs+r r W )?

Non-adaptive method
Select typical set of training images
Design predictor based on training images
Adaptive method
Allow predictor to adapt to images being coded
Design decoder so it adapts in same manner as encoder

C. A. Bouman: Digital Image Processing - April 17, 2013

22

Non-Adaptive Predictive Coder

Method for estimating predictor
1. Select typical set of training images
2. For each pixel in each image, form zs = (xs+r0 , , xs+rp1 )
where {r0, , rp1} W .
3. Index the values of zs from j = 0 to j = 2p 1
4. For each pixel in each image, compute
hs(i, j) = (xs = i)(zs = j)
and the histogram
h(i, j) =

hs(i, j)

sS

5. Estimate p(xs|xs+r r W ) = p(xs|zs) as

h(i, j)
p(xs = i|zs = j) = P1
k=0 h(k, j)

C. A. Bouman: Digital Image Processing - April 17, 2013

23

Adaptive Predictive Coder

Adapt predictor at each pixel

Update value of h(i, j) at each pixel using equations

h(i, j) h(i, j) + (xs = i)(zs = j)
N (j) N (j) + 1
Use updated values of h(i, j) to compute new predictor at
each pixel
p(i|j)

h(i, j)
N (j)

Design decoder to track encoder

C. A. Bouman: Digital Image Processing - April 17, 2013

24

Adaptive Predictive Entropy Coder Flow

Diagram
Encoder
Xs

XOR

Run Length
Encoding

Huffman
Coding

Causal
Predictor
Causal
Histogram Estimation

Decoder
Huffman
Decoding

Run Length
Decoding

XOR

Xs
Causal
Predictor
Causal
Histogram Estimation

C. A. Bouman: Digital Image Processing - April 17, 2013

25

Lossy Source Coding

Method for representing discrete-space signals with minimum distortion and bit-rate
Outline
Rate-distortion theory
Karhunen-Loeve decorrelating Transform
Practical coder structures

C. A. Bouman: Digital Image Processing - April 17, 2013

26

Distortion
Let X and Z be random vectors in IRM . Intuitively, X is
the original image/data and Z is the decoded image/data.
Assume we use the squared error distortion measure given
by
d(X, Y ) = ||X Z||2
Then the distortion is given by
h

D = E [d(X, Y )] = E ||X Z||

This actually applies to any quadratic norm error distortion measure since we can define
= AX and
X
So

Z = AZ

i
h
h
 2 i
2
= E X
Z  = E ||X Z||
D
B

where B = AtA.

C. A. Bouman: Digital Image Processing - April 17, 2013

27

Lossy Source Coding: Theoretical Framework

Notation for source coding

Xn IRM for 0 n < N - a sequence of i.i.d. random

vectors
Y {0, 1}K - a K bit random binary vector.
Zn IRM for 0 n < N - the decoded sequence of
random vectors.

X (N ) = (X0, , XN 1)
Z (N ) = (Z0, , ZN 1)
Encoder function: Y = Q(X0, , XN 1)
Decoder function: (Z0, , ZN 1) = f (Y )
Resulting quantities
Bit-rate =

K
N

N 1
i
1 X h
2
E ||Xn Zn||
Distortion =
N n=0

How do we choose Q() to minimize the bit-rate and distortion?

C. A. Bouman: Digital Image Processing - April 17, 2013

28

Differential Entropy
Notice that the information contained in a Gaussian random variable is infinite, so the conventional entropy H(X)
is not defined.
Let X be a random vector taking values in IRM with density function p(x). Then we define the differential entropy
of X as
Z
p(x) log2 p(x)dx
h(X) =
xIRM

= E [log2 p(X)]
h(X) has units of bits

C. A. Bouman: Digital Image Processing - April 17, 2013

29

Conditional Entropy and Mutual Information

Let X and Y be a random vectors taking values in IRM
with density function p(x, y) and conditional density p(x|y).
Then we define the differential conditional entropy of X
given Y as
Z
Z
h(X|Y ) =
p(x, y) log2 p(x|y)
xIRM

xIRM

= E [log2 p(X|Y )]
The mutual information between X and Y is given by
I(X; Y ) = h(X) h(X|Y ) = I(Y ; X)
Important: The mutual information is well defined for
both continuous and discrete random variables, and it represents the reduction in uncertainty of X given Y .

C. A. Bouman: Digital Image Processing - April 17, 2013

30

The Rate-Distortion Function

Define/Remember:
Let X0 be the first element of the i.i.d. sequence.
Let D 0 be the allowed distortion.

For a specific distortion, D, the rate is given by





2
R(D) = inf I(X0; Z) : E ||X0 Z|| D
Z

where the infimum (i.e. minimum) over Z is taken over

all random variables Z.

Later, we will show that for a given distortion we can find

a code that gets arbitrarily close to this optimum bit-rate.

C. A. Bouman: Digital Image Processing - April 17, 2013

Properties of the Rate-Distortion Function

Properties of R(D)
R(D) is a monotone decreasing function of D.
h
i
2
If D E ||X0|| , then R(D) = 0
R(D) is a convex function of D

31

C. A. Bouman: Digital Image Processing - April 17, 2013

32

Shannons Source-Coding Theorem

Shannons Source-Coding Theorem:
For any R > R(D) and D > D there exists a sufficiently
large N such that there is an encoder
Y = Q(X0, , XN 1)
which achieves
Rate =

K
R
N

and
N 1
i
1 X h
Distortion =
E ||Xn Zn||2 D
N n=0

Comments:
One can achieve a bit rate arbitrarily close to R(D) at
a distortion D.
Proof is constructive (but not practical), and uses codes
that are randomly distributed in the space IRM N of source
symbols.

C. A. Bouman: Digital Image Processing - April 17, 2013

33

Example 1: Coding of Gaussian Random

Variables


Let X N (0, ) with distortion function E |X Z| ,

then it can be shown that the rate-distortion function has
the form

 2 
1

R() = max
log
,0
2

 2
D() = min ,
Intuition:

is a parameter which represents the quantization step

 2
1
2 log represents the number of bits required to encode the quantized scalar value.
Minimum number of bits must be 0.
Maximum distortion must be 2.
Distortion versus Rate Curve

Distortion

Rate

C. A. Bouman: Digital Image Processing - April 17, 2013

34

Example 2: Coding of N Independent Gaussian

Random Variables
Let X = [X1, , XN 1]t with independent components
such that Xn N (0, n2 ), and define the distortion function to be


2
Distortion = E ||X Z||
Then it can be shown that the rate-distortion function has
the form

 2 
N
1
X
1
n
max
R() =
log
,0
2

n=0
D() =

N
1
X
n=0

Intuition:

min

n2 ,

In an optimal coder, the quantization step should be

approximately equal for each random variable being
quantized.
The bit rate and distortion both add for each component.
It can be proved that this solution is optimal.

C. A. Bouman: Digital Image Processing - April 17, 2013

35

Example 3: Coding of Gaussian Random

Vector
Let X N (0, R) be a N dimensional Gaussian random
vector, and define the distortion function to be


2
Distortion = E ||X Z||

Analysis: We know that we can always represent the covariance in the form
R = T tT
where the columns of T are the eigenvectors of R, and
2
= diag{02, , N
1 } is a diagonal matrix of eigenvalues. We can then decorrelate the Gaussian random vector with the following transformation.
= T tX
X
has the covariance matrix
From this we can see that X
given by

 t


t
t

E X X = E T XX T


t
t
= T E XX T = T tRT =
meets the conditions of Example 2. Also,
So therefore, X
we see that




2
2

E ||X Z|| = E ||X Z||

= T tX and Z = T tZ because T is an orthonorwhere X
mal transform.

C. A. Bouman: Digital Image Processing - April 17, 2013

36

Example 3: Coding of Gaussian Random

Vector (Result)
Let X N (0, R) be a N dimensional Gaussian random
vector, and define the distortion function to be


2
Distortion = E ||X Z||
Then it can be shown that the rate-distortion function has
the form
 2 

N
1
X
n
1
log
,0
max
R() =
2

n=0
D() =

N
1
X
n=0

min

n2 ,

2
where 02, , N
1 are the eigenvalues of R.

Intuition:
An optimal code requires that the components of a vector be decorrelated before source coding.

C. A. Bouman: Digital Image Processing - April 17, 2013

37

Example 4: Coding of Stationary Gaussian

Random Process
Let Xn be a stationary Gaussian random process with power
spectrum Sx(), and define the distortion function to be


2
Distortion = E |Xn Z|
Then it can be shown that the rate-distortion function has
the form

 

Z
1
Sx()
1
R() =
log
, 0 d
max
2 Z
2

1
D() =
min {Sx(), } d
2

Intuition:

The Fourier transform decorrelates a stationary Gaussian random process.

Frequencies with amplitude below are clipped to zero.
Distortion versus Rate Curve

Distortion

Rate
Distortion

C. A. Bouman: Digital Image Processing - April 17, 2013

38

The Discrete Cosine Transform (DCT)

DCT (There is more than one version)


N
1
X
(2n + 1)k
1
f (n) c(k) cos
F (k) =
2N
N n=0
where

c(k) =

Inverse DCT (IDCT)

1
f (n) =
N

1 k = 0
2 k = 1, , N 1

N
1
X

F (k) c(k) cos

k=0

(2n + 1)k
2N

Comments:
In this form, the DCT is an orthonormal transform. So
if we define the matrix F such that


(2n + 1)k
Fn,k = c(k) cos
,
2N
then F 1 = F H where
F

= F


t

= FH

Takes and N -point real valued signal to an N -point

real valued signal.

C. A. Bouman: Digital Image Processing - April 17, 2013

39

Relationship Between DCT and DFT

Let us define the padded version of f (n) as

f (n) 0 n N 1
fp(n) =
0
N n 2N 1
and its 2N -point DFT denoted by Fp(k). Then the DCT
can be written as


N
1
X
(2n + 1)k
c(k)

f (n) cos
F (k) =
2N
N n=0
N 1
n 2(2n+1)k
o
k
c(k) X
j
j
2N
f (n)Re e
e 2N
=
N n=0
(
)
2N
1
2(2n+1)k
k X
c(k)
j 2N
j
2N
= Re e
fp(n)e
N
n=0
o
c(k) n j k
= Re e 2N Fp(k)
N

k
k
c(k) 
j 2N
+j 2N
Fp(k)e
+ Fp(k)e
=
N
k 

 
j 2N
2k
c(k)e

Fp(k) + Fp(k)ej 2N
=
N

C. A. Bouman: Digital Image Processing - April 17, 2013

40

Interpretation of DFT Terms

Consider the inverse DCT for each of the two terms
Fp(k) fp(n)
DCT {fp(n)} Fp(k)

DCT {fp(n + N 1)} Fp(k)e

Simple example for N = 4

j 2k
2N

f (n)
= [f (0), f (1), f (2), f (3)]
fp(n)
= [f (0), f (1), f (2), f (3), 0, 0, 0, 0]
fp(n+N 1)
= [0, 0, 0, 0, f (3), f (2), f (1), f (0)]
f (n) + fp(n+N 1)
= [f (0), f (1), f (2), f (3), f (3), f (2), f (1), f (0)]

C. A. Bouman: Digital Image Processing - April 17, 2013

41

Relationship Between DCT and DFT

(Continued)
So the DCT is formed by 2N -point DFT of f (n)+f (n+
N 1).
F (k) =

k
j 2N

c(k)e

DF T2N {f (n) + f (n + N 1)}