Digital communication system

Representation of the source signal by a stream of (binary) symbols Adaptation to the properties of the transmission channel

information source

source coder

channel coder

modulation

binary symbols

channel noise

information sink

source decoder

channel decoder

demodulation

We will be looking at this part

Bernd Girod: EE368b Image and Video Compression

digital channel

Lossless Compression no. 1

Shannons separation principle

Assume: 1. Point-to-point communication 2. Ergodic channel 3. Delay

Source

Source Source Adapt to source statistics coder coder (and distortion measure)

Source Source decoder decoder

Channel Channel Adapt to channel statistics coder coder

Channel Channel decoder decoder

Channel
Bernd Girod: EE368b Image and Video Compression Lossless Compression no. 2

How does compression work?

n

Exploit redundancy.
l l l

Take advantage of patterns in the signal. Describe frequently occuring events efficiently. Lossless coding: completely reversible Remove information that the humans cannot perceive. Match the signal resolution (in space, time, amplitude) to the application Lossy coding: irreversible distortion of the signal

Introduce acceptable deviations.

l l

Bernd Girod: EE368b Image and Video Compression

Lossless Compression no. 3

Lossless compression in lossy compression systems

n

Almost every lossy compression system contains a lossless compression system Lossy compression system
Quantizer Encoder Lossless Encoder Lossless Decoder Quantizer Decoder

Lossless compression system

We will discuss the basics of lossless compression first, then move on to lossy compression
Bernd Girod: EE368b Image and Video Compression Lossless Compression no. 4

Topics in lossless compression

n n n n n n n

Binary decision trees and variable length coding Entropy and bit-rate Huffman codes Statistical dependencies in image signals Sources with memory Arithmetic coding Redundancy reduction by prediction

Bernd Girod: EE368b Image and Video Compression

Lossless Compression no. 5

Example: 20 Questions
n

Alice thinks of an outcome (from a finite set), but does not disclose his selection. Bob asks a series of yes-no questions to uniquely determine the outcome chosen. The goal of the game is to ask as few questions as possible on average. Our goal: Design the best strategy for Bob.

Bernd Girod: EE368b Image and Video Compression

Lossless Compression no. 6

Example: 20 Questions (cont.)

n

Observation: The collection of questions and answers yield a binary code for each outcome.

0 0 0

1 1 0 1 1 0 0 1

0 1 0

1 1 1

B C
0

C D 0 E

D
n

Which strategy (=code) is better?

Bernd Girod: EE368b Image and Video Compression Lossless Compression no. 7

Fixed length codes

0 0 0 0 1 1 0 1 0 0 1 1

A
n n n

B C

DE

F G

Average description length for K outcomes lav = log2 K Optimum for equally likely outcomes Verify by modifying tree

Bernd Girod: EE368b Image and Video Compression

Lossless Compression no. 8

Variable length codes

n

If outcomes are NOT equally probable:

l l

Use shorter descriptions for likely outcomes Use longer descriptions for less likely outcomes Optimum balanced code trees, i.e., with equally likely outcomes, can be pruned to yield unbalanced trees with unequal probabilities. The unbalanced code trees such obtained are also optimum. Hence, an outcome of probability p should require about

Intuition:
l

l l

1 log 2 bits p

Bernd Girod: EE368b Image and Video Compression

Lossless Compression no. 9

Entropy of a memoryless source

n

Let a memoryless source be characterized by an ensemble U0 with: Alphabet {a0 , a1 , a2 ,..., aK 1} Probabilities {P(a0 ), P(a1 ), P(a2 ),..., P(aK 1 )}

Shannon: information conveyed by message ak : I(ak ) = log(P(ak )) Entropy of the source is the average information contents: H (U0 ) = E{I ( ak )} = P( ak ) log( P( ak )) = P(u0 ) log( P(u0 ))
k =0 u0 K 1

For log = log 2 the unit is bits/symbol

Bernd Girod: EE368b Image and Video Compression Lossless Compression no. 10

Entropy and bit-rate

n

Properties of entropy:

H(U 0 ) 0 max{H(U0 )} = log(K) with P(a j ) = P(a k )j, k

n

The entropy H(U 0 ) is a lower bound for the average word length l av of a decodable variable-length code for the symbols u 0 . Conversely, the average wordlength l av can approach H(U 0 ), if sufficiently large blocks of symbols are encoded jointly. Redundancy of a code:

R = lav H(U 0 )
Bernd Girod: EE368b Image and Video Compression Lossless Compression no. 11

Encoding with variable word length

n

A code without redundancy, i.e.

Example

lav = H(U 0 )
is achieved, if all individual code word length

ai P(a i ) a0 a1 a2 a3
0.500 0.250 0.125 0.125

redundant optimum code code 00 01 10 11 0 10 110 111

lcw (ak ) = log(P(ak ))

n

For binary code words, all probabilities would have to be binary fractions:

H(U 0 ) = 1.75 bits / symbol lav = 1.75 bits / symbol R=0

Lossless Compression no. 12

P(a k ) = 2 l

cw (a k

Bernd Girod: EE368b Image and Video Compression

Huffman-Code
n

Design algorithm for variable length codes proposed by Huffman (1952) always finds a code with minimum redundancy. Obtain code tree as follows: 1 Pick the two symbols with lowest probabilities and merge them into a new auxiliary symbol. 2 Calculate the probability of the auxiliary symbol. 3 If more than one symbol remains, repeat steps 1 and 2 for the new auxiliary alphabet. 4 Convert the code tree into a prefix code.
Bernd Girod: EE368b Image and Video Compression Lossless Compression no. 13

Huffman-Code - Example

Fixed length coding: l =3.00 Huffman code: lav =2.71 Entropy H(U0 )=2.68 Redundancy of the Huffman code: R=0.03

bits/symbol bits/symbol bits/symbol bits/symbol

Bernd Girod: EE368b Image and Video Compression

Lossless Compression no. 14

Probability density function of the luminance signal Y

Images: 3 EBU test slides, 3 SMPTE test slides, uniform quantization with 256 levels (8 bits/pixel)

H(U Y ) = 7.34 bits / pixel

Image with lowest entropy:

HL (UY ) = 6.97 bits / pixel

Image with highest entropy: HH (UY ) = 7.35 bits / p i x e l
Bernd Girod: EE368b Image and Video Compression Lossless Compression no. 15

Probability density function of the color difference signals R-Y and B-Y

H(U RY ) = 5.57 bits/pixel

HL (U R Y ) = 4.65 bits/pixel HH (U R Y ) = 5.72 bits / pixel
Bernd Girod: EE368b Image and Video Compression

H(U B Y ) = 5.24 bits/pixel

HL (U B Y ) = 4.00 bits / p i x e l HH (U B Y ) = 5.34 bits / p i x e l
Lossless Compression no. 16

Joint sources
n

Joint sources generate N symbols simultaneously. A coding gain can be achieved by encoding those symbols jointly. The lower bound for the average code word length is the joint entropy:

H(U1 ,U 2 ,...,U N ) = ... P(u1 ,u2 ,...,u N )log(P(u1 ,u2 ,...,u N ))

u1 u 2 uN

It generally holds that

H(U1 ,U 2 ,...,U N ) H(U1 ) + H(U2 ) + ... + H(U N )

with equality, if U1 ,U2 ,...,U N are statistically independent.
Bernd Girod: EE368b Image and Video Compression Lossless Compression no. 17

Statistical dependencies between video signal components Y, R-Y, B-Y

n

Data: 3 EBU-, 3 SMPTE test slides, each component Y, R-Y, B-Y uniformly quantized to 64 levels

H 0 = 3x6 bits / sample

18

bits/sample

H(U Y ,U R Y ,U BY ) = 9.044 bits/sample H(U Y ) + H(U R Y ) + H(U BY ) = 11.218 bits/sample H

n n

= 2.174 bits/sample

Statistical dependency between R, G, B is much stronger. If joint source Y, R-Y, B-Y is treated as a source with memory, the possible gain by joint coding is much smaller.

Bernd Girod: EE368b Image and Video Compression

Lossless Compression no. 18

Markov process
n

Neighboring samples of the video signal are not statistically independent: source with memory

P(uT ) P(uT | u T 1 ,u T 2 ,...,uT N )

n

A source with memory can be modeled by a Markov random process. Conditional probabilities of the source symbols u T of a Markov source of order N:

P(uT , ZT ) = P(uT | uT 1 ,uT 2 ,...,u T N )

state of the Markov source at time T
Bernd Girod: EE368b Image and Video Compression Lossless Compression no. 19

Entropy of source with memory

n

Markov source of order N: conditional entropy

H(U T | ZT ) = H(UT | UT 1 ,UT 2 ,..., U T N ) = E{ log( p(UT | U T 1 ,UT 2 ,...,UT N ))} = u ...u
T T N

p(uT , uT 1 , uT 2 ,..., uT N ) log( p(uT | uT 1 , uT 2 ,...,u T N ))

H(U T | ZT ) H(UT )
n

(equality for memoryless sources)

Average code word length can approach H(U T | ZT ) e.g. with a switched Huffman code. Number of states for an 8-bit video signal:

N=1 N=2 N=3

256 states 65536 states 16777216 states

Lossless Compression no. 20

Bernd Girod: EE368b Image and Video Compression

10