ECE/CS 778 Multimedia Communication

Lec 03: Entropy Coding II -

Huffman & Golomb Coding

Zhu Li
Dept of CSEE, UMKC
Office: FH560E, Email: [email protected], Ph: x 2346.
http://l.web.umkc.edu/lizhu

slides created with WPS Office Linux and EqualX LaTex equation editor

Z. Li: ECE/CS 5578 Multimedia Communication p.1

 Outline
 Lecture 02 ReCap
 Huffman Coding
 Golomb Coding and JPEG 2000 Lossless Coding

Z. Li: ECE/CS 5578 Multimedia Communication p.2

 Info Theory on Entropy
 Self Info of an event
= =− log Pr = =− log
Main application: Context
 Entropy of a source
= log
1
Modeling

 Conditional Entropy, Mutual Information

=
abcbcab
1 2 1, 2 − 2

1, = + − 1,
2 1 2 2
cbabcba
Total area: H(X, Y)
Relative Entropy/Cross Entropy

= log
H(X | Y) I(X; Y) H(Y | X)

H(X) H(Y)

Z. Li: ECE/CS 5578 Multimedia Communication p.3

 GetEntropy()
 Implementation
%function [e]=getEntropy(pmf)
function [e]=getEntropy(pmf)
dbg=0;
if dbg
pmf=[0.002205 0.002573 0.008453 0.016538 0.058802 0.712973
0.155825 0.026828 0.011393 0.004410];
end

pmf = pmf(find(pmf>0));
e = -1*sum(pmf.*log(pmf));

return;

Z. Li: ECE/CS 5578 Multimedia Communication p.4

 Image Pixel Value PMF
 Joint Distribution of 2 horizontally adjacent pixels

Credit: T. Wiegand, TU Berlin

Z. Li: ECE/CS 5578 Multimedia Communication p.5

 Context Reduces Entropy Example
lenna.png
Condition reduces entropy:
H(x5) > H(x5|x4,x3, x2,x1)

H(x5) > H(x5|f(x4,x3, x2,x1))

Context :

x1 x2 x3

x4 x5
f(x4,x3, x2,x1)==100

getEntropy.m, lossless_coding.m f(x4,x3, x2,x1)<100

Context function:
f(x4,x3, x2,x1)= sum(x4,x3, x2,x1)
Z. Li: ECE/CS 5578 Multimedia Communication p.6
 Lossless Coding
 Prefix Coding
Root node
 Codes on leaves
 No code is prefix of other codes
0 1 Internal node
 Simple encoding/decoding
0 0 1
10
0 1

leaf node 110 111

Kraft- McMillan Inequality:

 For a coding scheme with code length: l1, l2, …ln,
2− ≤1

 Given a set of integer length {l1, l2, …ln} that satisfy above inequality,
we can always find a prefix code with code length l1, l2, …ln

Z. Li: ECE/CS 5578 Multimedia Communication p.7

 Outline
 Lecture 02 ReCap
 Hoffman Coding
 Golomb Coding and JPEG 2000 Lossless

Morse Code is a Huffman Code

Z. Li: ECE/CS 5578 Multimedia Communication p.8

 Huffman Coding

A procedure to construct optimal prefix code

Result of David Huffman’s term paper in 1952 when he was a
PhD student at MIT
 Shannon  Fano  Huffman (1925-1999)

Z. Li: ECE/CS 5578 Multimedia Communication p.9

 Hoffman Code Design
 Requirement:
 The source probability distribution.
(But not available in many cases)
 Procedure:
1. Sort the probability of all source symbols in a descending order.
2. Merge the last two into a new symbol, add their probabilities.
3. Repeat Step 1, 2 until only one symbol (the root) is left.
4. Code assignment:
Traverse the tree from the root to each leaf node,
assign 0 to the top branch and 1 to the bottom branch.

Z. Li: ECE/CS 5578 Multimedia Communication p.10

 Example

Source alphabet A = {a1, a2, a3, a4, a5}

Probability distribution: {0.2, 0.4, 0.2, 0.1, 0.1}
Sort merge Sort merge Sort merge Sort merge

1 0.4 1 0.4 1 0.6 0

1 a2 (0.4) 1
01 0.2 01 0.4 00
01 a1(0.2) 0.6 0.4 1
000 0.2 000 0.2
000 a3(0.2) 0.4
0010 01
0010 a4(0.1) 0.2 0.2
001
0011 a5(0.1)
0011 Assign code

Z. Li: ECE/CS 5578 Multimedia Communication p.11

 Huffman code is prefix-free

1
1 (a2)
01 01 (a1)

000
000 (a3)
0010

0011 0010 (a4) 0011(a5)

All codewords are leaf nodes

 No code is a prefix of any other code.
(Prefix free)

Z. Li: ECE/CS 5578 Multimedia Communication p.12

 Average Codeword Length vs Entropy

 Source alphabet A = {a, b, c, d, e}

 Probability distribution: {0.2, 0.4, 0.2, 0.1, 0.1}
 Code: {01, 1, 000, 0010, 0011}

 Entropy:
H(S) = - (0.2*log2(0.2)*2 + 0.4*log2(0.4)+0.1*log2(0.1)*2)
= 2.122 bits / symbol

 Average Huffman codeword length:

L = 0.2*2+0.4*1+0.2*3+0.1*4+0.1*4 = 2.2 bits / symbol
 This verifies H(S) ≤ L < H(S) + 1.

Z. Li: ECE/CS 5578 Multimedia Communication p.13

 Huffman Code is not unique

 Two choices for each split: 0, 1 or 1, 0

0.4 1 0.6 0 0.4 0 0.6 1

0.4 00 0.4 10
0.6 0.4 1 0.6 0.4 0
0.2 0.2
01 11

 Multiple ordering choices for tied probabilities

0 0.6 1 0 0.6 1
a 0.4 b 0.4

b 0.4 10 0.4 0 a 0.4 10 0.4 0

0.6 0.6
c 0.2 c 0.2
11 11

Z. Li: ECE/CS 5578 Multimedia Communication p.14

 Huffman Coding is Optimal

Assume the probabilities are ordered:

 p1 ≥ p2 ≥ …… pm.
Lemma: For any distribution, there exists an optimal
prefix code that satisfies:
 If pj ≥ pk, then lj ≤ lk: otherwise can swap codewords to reduce
the average length.

The two least probable letters have the same length:

otherwise we can truncate the longer one without violating
prefix-free condition.

The two longest codewords differ only in the last bit and
correspond to the two least probable symbols. Otherwise we
can rearrange to achieve this.
 Proof skipped.
Z. Li: ECE/CS 5578 Multimedia Communication p.15
 Canonical Huffman Code
 Huffman algorithm is needed only to compute the optimal
codeword lengths
 The optimal codewords for a given data set are not unique
 Canonical Huffman code is well structured
 Given the codeword lengths, can find a canonical Huffman
code
 Also known as slice code, alphabetic code.

Z. Li: ECE/CS 5578 Multimedia Communication p.16

 Canonical Huffman Code
 Rules:
 Assign 0 to left branch and 1 to right branch
 Build the tree from left to right in increasing order of depth
 Each leaf is placed at the first available position
Example:
N
 Codeword lengths: 2, 2, 3, 3, 3, 4, 4
 Verify that it satisfies Kraft-McMillan inequality
å £1
2 - li

i =1

A non-canonical example The Canonical Tree

10

01 00 01
11 111
100 101 110
000 001 100

1010 1011 1110 1111

Z. Li: ECE/CS 5578 Multimedia Communication p.17

 Canonical Huffman
Properties:
 The first code is a series of 0
 Codes of same length
are consecutive: 100, 101, 110 00 01
 If we pad zeros to the right side such that all 100 101 110
codewords have the same length, shorter codes would
have lower value than longer codes: 1110 1111
0000 < 0100 < 1000 < 1010 < 1100 < 1110 < 1111

 Coding from length level n to level n+1:

 C(n+1, 1) = 2 ( C(n, last) + 1): append a 0 to the next available level-n code

First code Last code of

of length n+1 length n
0
 If from length n to n + 2 directly:
e.g., 1, 3, 3, 3, 4, 4 100 101 110
C(n+2, 1) = 4( C(n, last) + 1)
1110 1111
Z. Li: ECE/CS 5578 Multimedia Communication p.18
 Advantages of Canonical Huffman
1. Reducing memory requirement
 Non-canonical tree needs:
All codewords
01 Lengths of all codewords
11
100
 Need a lot of space for large table
000 001
1010 1011
 Canonical tree only needs:
 Min: shortest codeword length
 Max: longest codeword length
 Distribution:
00 01
 Number of codewords in each level
100 101 110
 Min=2, Max=4,
1110 1111
# in each level: 2, 3, 2
Z. Li: ECE/CS 5578 Multimedia Communication p.19
 Outline
 Lecture 02 ReCap
 Hoffman Coding
 Golomb Coding

Z. Li: ECE/CS 5578 Multimedia Communication p.20

 Unary Code (Comma Code)
Encode a nonnegative integer n by n 1’s and a 0
(or n 0’s and an 1).

 No need to store codeword table, very simple

 Is this code prefix-free?
n Codeword
0 0 0 1
1 10 0 0 1
2 110 10
0 1
3 1110 110
4 11110
1110
5 111110 ……
……
 When is this code optimal?
 When probabilities are: 1/2, 1/4, 1/8, 1/16, 1/32 …  D-adic
 Huffman code becomes unary code in this case.
Z. Li: ECE/CS 5578 Multimedia Communication p.21
 Implementation – Very Efficient
Encoding:
UnaryEncode(n) {
while (n > 0) {
WriteBit(1);
n--;
}
WriteBit(0);
}

 Decoding:
UnaryDecode() {
n = 0;
while (ReadBit(1) == 1) {
n++;
}
return n;
}

Z. Li: ECE/CS 5578 Multimedia Communication p.22

 Golomb Code [Golomb, 1966]
A multi-resolutional approach:
 Divide all numbers into groups of equal size m
o Denote as Golomb(m) or Golomb-m
 Groups with smaller symbol values have shorter codes
 Symbols in the same group has codewords of similar lengths
o The codeword length grows much slower than in unary code

m m m m

0 max

 Codeword :
 (Unary, fixed-length)

Group ID: Index within each group:

Unary code
Z. Li: ECE/CS 5578 Multimedia Communication p.23
 Golomb Code

ênú
n = qm + r = ê ú m + r
ëmû
 q: Quotient,  r: remainder, “fixed-length” code
used unary code
 K bits if m = 2^k
q Codeword
 m=8: 000, 001, ……, 111
0 0
1 10  If m ≠ 2^k: (not desired)
2 110
3 1110 ëlog 2 mû bits for smaller r
4 11110 bits for larger r
5 111110
élog 2 mù
m = 5: 00, 01, 10, 110, 111
6 1111110
……
Z. Li: ECE/CS 5578 Multimedia Communication p.24
 Golomb Code with m = 5 (Golomb-5)
n q r code n q r code n q r code
0 0 0 000 5 1 0 1000 10 2 0 11000
1 0 1 001 6 1 1 1001 11 2 1 11001
2 0 2 010 7 1 2 1010 12 2 2 11010
3 0 3 0110 8 1 3 10110 13 2 3 110110
4 0 4 0111 9 1 4 10111 14 2 4 110111

Z. Li: ECE/CS 5578 Multimedia Communication p.25

 Golomb vs Canonical Huffman

 Codewords: 000, 001, 010, 0110, 0111,

1000, 1001, 1010, 10110, 10111
 Canonical form:
From left to right
From short to long
Take first valid spot

Golomb code is a canonical Huffman

 With more properties

Z. Li: ECE/CS 5578 Multimedia Communication p.26

 Golobm-Rice Code

 A special Golomb code with m= 2^k

 The remainder r is the fixed k LSB bits of n

m=8 n q r code n q r code

0 0 0 0000 8 1 0 10000
1 0 1 0001 9 1 1 10001
2 0 2 0010 10 1 2 10010
3 0 3 0011 11 1 3 10011
4 0 4 0100 12 1 4 10100
5 0 5 0101 13 1 5 10101
6 0 6 0110 14 1 6 10110
7 0 7 0111 15 1 7 10111

Z. Li: ECE/CS 5578 Multimedia Communication p.27

 Implementation

Encoding: n q r code
Remainder bits:
RBits = 3 for m = 8 0 0 0 0000
1 0 1 0001
GolombEncode(n, RBits) {
q = n >> RBits; 2 0 2 0010
UnaryCode(q); 3 0 3 0011
WriteBits(n, RBits); 4 0 4 0100
} Output the lower (RBits) bits of n. 5 0 5 0101
 Decoding: 6 0 6 0110
7 0 7 0111
GolombDecode(RBits) {
q = UnaryDecode();
n = (q << RBits) + ReadBits(RBits);
return n;
}

Z. Li: ECE/CS 5578 Multimedia Communication p.28

 Exponential Golomb Code (Exp-Golomb)
n code
Golomb code divides the alphabet into
0 0
groups of equal size 1 100
m m m m 2 101
3 11000

0 4 11001
max
 In Exp-Golomb code, the group size 5 11010
6 11011
increases exponentially 7 1110000

 Codes still contain two parts: 8 1110001

9 1110010
 Unary code followed by fixed-length code 10 1110011
1 2 4 8 16 11 1110100
12 1110101
0 max 13 1110110
14 1110111
 Proposed by Teuhola in 1978 15 111100000

Z. Li: ECE/CS 5578 Multimedia Communication p.29

 Implementation
Decoding
n code Group
ID

ExpGolombDecode() { 0 0 0
GroupID = UnaryDecode(); 1 100 1
if (GroupID == 0) { 2 101
return 0; 3 11000 2
} else { 4 11001
Base = (1 << GroupID) - 1; 5 11010
Index = ReadBits(GroupID); 6 11011
return (Base + Index); 7 1110000 3
} 8 1110001
} 9 1110010
} 10 1110011
11 1110100
12 1110101
13 1110110
14 1110111

Z. Li: ECE/CS 5578 Multimedia Communication p.30

 Outline
Golomb Code Family:
 Unary Code
 Golomb Code
 Golomb-Rice Code
 Exponential Golomb Code
Why Golomb code?

Z. Li: ECE/CS 5578 Multimedia Communication p.31

 Geometric Distribution (GD)

 Geometric distribution with parameter ρ:

 P(x) = ρx (1 - ρ), x ≥ 0, integer.
 Prob of the number of failures before the first success in a series of
independent Yes/No experiments (Bernoulli trials).
 Unary code is the optimal prefix code for geometric distribution
with ρ ≤ 1/2:
 ρ = 1/4: P(x): 0.75, 0.19, 0.05, 0.01, 0.003, …
 Huffman coding never needs to re-order  equivalent to unary code.
 Unary code is the optimal prefix code, but not efficient
( avg length >> entropy)
 ρ = 3/4: P(x): 0.25, 0.19, 0.14, 0.11, 0.08, …
 Reordering is needed for Huffman code, unary code not optimal prefix code.
 ρ = 1/2: Expected length = entropy.
 Unary code is not only the optimal prefix code, but also optimal among all
entropy coding (including arithmetic coding).
Z. Li: ECE/CS 5578 Multimedia Communication p.32
 Geometric Distribution (GD)
Geometric distribution is very useful for image/video compression
Example 1: run-length coding
 Binary sequence with i.i.d. distribution
 P(0) = ρ ≈ 1:
 Example: 0000010000000010000110000001
 Entropy << 1, prefix code has poor performance.

 Run-length coding is efficient to compress the data:

o r:Number of consecutive 0’s between two 1’s
o  run-length representation of the sequence: 5, 8, 4, 0, 6

 Probability distribution of the run-length r:

o P(r = n) = ρn (1- ρ): n 0’s followed by an 1.

P(r)
o  The run has one-sided geometric distribution
with parameter ρ.

Z. Li: ECE/CS 5578 Multimedia Communication p.33

 Geometric Distribution
GD is the discrete analogy of the Exponential distribution

f(x)
-l x
f ( x) = l e
x

Two-sided geometric distribution is the discrete

analogy of the Laplacian distribution (also called
double exponential distribution) f(x)

-l x
f ( x) = l e
1
2
x

Z. Li: ECE/CS 5578 Multimedia Communication p.34

 Why Golomb Code?
Significance of Golomb code:
 For any geometric distribution (GD), Golomb code is optimal prefix
code and is as close to the entropy as possible (among all prefix
codes)
 How to determine the Golomb parameter?
 How to apply it into practical codec?

closest to diadic:
1/2, 1/4, 1/8,...

Z. Li: ECE/CS 5578 Multimedia Communication p.35

 Geometric Distribution
Example 2: GD is a also good model for Prediction error
e(n) = x(n) – pred(x(1), …, x(n-1)). 1 - r |n|
p ( n) = r , 0 < r < 1.
Most e(n)’s have smaller values around 0: 1+ r
 can be modeled by geometric distribution.
p(n)

x1 x2 x3 0.2 0.3 0.2

0.2 0 0
p=0.7? x4 x5
0 0 0

lossless_coding.m
Z. Li: ECE/CS 5578 Multimedia Communication p.36
 Optimal Code for Geometric Distribution
 Find m s.t. group wise Geometric distribution with p ~ 1/2
 Geometric distribution with parameter ρ: P(x)
 P(X=n) = ρn (1 - ρ)
 Unary code is optimal prefix code when ρ ≤ 1/2.
 Also optimal among all entropy coding for ρ = 1/2. x
 How to design the optimal code when ρ > 1/2 ?
P(x)
 Transform into GD with ρ ≤ 1/2 (as close as possible)
How? By grouping m events together!
Each x can be written as x = xq m + xr x
m -1 m -1
1 - r m
PX q ( q) = å PX ( qm + r ) = å (1 - r )r qm + r =(1 - r )r qm = r mq (1 - r m )
r =0 r =0 1- r
 xq has geometric dist with parameter ρm.
Unary code is optimal for xq if ρm ≤ 1/2 
1 é 1 ù
m³- m = ê- ú is the minimal possible integer.
log 2 r ê log 2 r ú
Z. Li: ECE/CS 5578 Multimedia Communication p.37
 Golomb Parameter Estimation (J2K book: pp. 55)

• Goal of adaptive Golomb code:

• For the given data, find the best m such that ρm ≤1/2.
• How to find ρ from the statistics of past data?
P( x) = (1 - r ) r x
¥
r r
E ( x) = å (
x=0
1 - r ) x r x
= (1 - r ) =
(1 - r ) 2 1 - r
E ( x)
r= .
1 + E ( x)
m
æ E ( x) ö 1 æ 1 + E ( x) ö
r m = çç ÷÷ £ m ³ 1 / logçç ÷÷
è 1 + E ( x) ø 2 è E ( x) ø
æ æ 1 + E ( x) ö ö
Let m=2k ç
k ³ log 2 ç1 / log 2 çç ÷÷ ÷÷. Too costly to compute
è è E ( x) ø ø

Z. Li: ECE/CS 5578 Multimedia Communication p.38

 Golomb Parameter Estimation (J2K book: pp. 55)

r
E ( x) =
1- r

A faster method: Assume ρ ≈ 1, 1 – ρ ≈ 0.

1- r m
r = (1 - (1 - r )) » 1 - m(1 - r ) » 1 - m
m m
= 1-
r E ( x)

1
ρm ≤1/2 k
m = 2 ³ E ( x)
2

ì é æ1 öù ü
k = max í 0 , ê log 2 ç 2 E ( x ) ÷ ú ý.
î ê è øú þ

Z. Li: ECE/CS 5578 Multimedia Communication p.39

 Summary
 Hoffman Coding
 A prefix code that is optimal in code length (average)
 Canonical form to reduce variation of the code length
 Widely used
 Golomb Coding
 Suitable for coding prediction errors in image
 Optimal for Geometrical Distribution of p=0.5
 Simple to encode and decode
 Many practical applications, e.g., JPEG-2000 lossless.

Z. Li: ECE/CS 5578 Multimedia Communication p.40