EIE579 Advanced Telecommunication Systems

Lecture 3: Source Coding

LIU Liang

Assistant Professor
Department of Electronic and Information Engineering
The Hong Kong Polytechnic University

0
 Lecture 2 Review

q Sampling and De-Sampling (Nyquist Sampling Theory)

q Quantization and de-quantization

1
 An Example for A/D and D/A Conversion

q Sampling rate: 5 Hz
q Number of quantization bits per sample: 3
2
 Sampling at Transmitter

q 5 samples per second

q > Nyquist sampling rate (why not 1 billion samples per second)
3
 Quantization at Transmitter

111 (7)
110 (6)

101 (5)
100 (4)

011 (3)

010 (2)
001 (1)
000 (0)

q Quantization bits for 10 samples

110 111 100 000 001 110 111 100 000 001
4
 De-Quantization at Receiver

111
110

101
100

011

010
001
000

q Receiver recovers 10 samples based on quantization bits

110 111 100 000 001 110 111 100 000 001
5
 De-Sampling at Receiver

111
110

101
100

011

010
001
000

q Receiver recovers signal based on 10 samples

6
 Lecture 3: Source Coding

q How to encode a sample at transmitter

q How to decode the encoded bits to recover the sample at receiver

7
 Source Coding
q Source coding is also called data compression
q Wikipedia: “data compression, or source coding, is the process of encoding information
using fewer bits (or other information-bearing units) than an unencoded representation
would use through use of specific encoding schemes.”
q Applications
Ø General data compression: .zip, .gz …
Ø Image over network: telephone/internet/wireless/etc
Ø Slow device: 1xCD-ROM 150KB/s, bluetooth v1.2 up to ~0.25MB/s
Ø Large multimedia databases

8
 Digital Images

q Images include digitized

pictures (from scanners or digital
cameras) and computer-
generated graphics.
q A digital image is represented
in form of a two dimensional
matrix (table) with individual
picture elements called pixels.

9
 Aspect Ratio

q The aspect ratio is the ratio between the number of pixels on the horizontal axis
(W) and the number of pixels on the vertical axis (H).

l Aspect ratio of TV: 4:3

l 1024:768 = 4:3

l Aspect ratio of HDTV: 16:9

H Aspect Ratio = W/H
l 1920:1080 = 16:9

10
Color Depth

11
Image Size

12
 Digital Video

q A sequence of digitized pictures.

q Frame rates (frame per second – fps)

Ø The number of pictures (frames) shows per second.

q Typical frame rates.

Ø full-motion video: 24-30 frames/s
60 frames/s for HDTV
Ø animation: 15-19 frames/s Animations extracted from
http://en.wikipedia.org/wiki/F
Ø video telephony: 5-10 frames/s ile:Newtons_cradle_animatio
n_book_2.gif

13
 Why We Need Compression
q Image: 6.0 million pixel camera, 3000x2000
Ø RGB (color depth 24 bits or 3 bytes)
§ What is the size of each image (MB)? 18 MB
§ Suppose the storage size is 1 GB. How many images can be stored? 56 images
q Video: DVD Disc 4.7 GB
Ø Video 720x480, RGB, 30 frames/sec
§ The maximum duration of the video per DVD disc 31.1MB in 1 sec 2.5 minutes
q Watch online video using cellphone:
Ø 352x240, RGB, 15 frames/sec
§ The minimum network speed: 30.4 Mbps
§ Can you afford it? (How about 4K video)
q How to compress data with fewer bits?
14
 Lossless Compression and Lossy Compression
q Lossless compression: original data is perfectly reconstructed from the compressed data
q Lossy compression: only permit reconstruction of an approximation of the original data,
but with fewer encoded bits to store compressed data

15
Part I: Lossless Compression

16
 Examples of Codes
q For any sample in a sample set
Ø Codeword assigned to :
Ø Length of codeword assigned to :
Ø Expected length of all codewords:
q Example 1: encode 1, 2, 3, 4

q Example 2: Morse Code

17
 Morse Code

q Length of codeword for A, B, …: Morse code: a b c

18
 Non-Singular Code
q A code is said to be non-singular if every sample is mapped to a different codeword

q Is Morse Code non-singular?

19
 Uniquely Decodable Code
q A code is said to be uniquely decodable if every sequence of samples is
mapped to a different sequence of codewords

q Is Morse Code uniquely decodable if no space is added

between two letters?
Ø EAH ··—···· · ·— ····
Ø IDI ··—···· ·· —·· ··
q A non-singular code may be not uniquely decodable
q A uniquely decodable code must be non-singular (n=1)

20
 Uniquely Decodable Code
q Consider the following code
Ø Assigning codes to letters a, b, and c

Ø Is this code uniquely decodable?

Ø Need to read the whole codewords to decode

21
 Prefix Code
q A code is said to be a prefix code or instantaneous code if no codeword is prefix of any
other codeword
q Consider the following code
Ø Assigning codes to letters a, b, and c

Ø Is this a prefix code?

Ø Is this a uniquely decodable code?

22
Relation between Various Codes

How to design prefix code?

23
 Bound for Prefix Code
q Encode samples
Ø PDF:
q Theorem: The expected length L of any prefix code to encode is greater than
or equal to the entropy of , i.e.

Ø Equality holds if and only if the length of codeword is

fewer bits to encode samples with

higher frequency

24
 Bound for Prefix Code
q Example:
Ø Encode
Ø PDF:
Ø How to design prefix code with the minimum expected length?

25
 Bound for Prefix Code
q Core idea: using fewer bits to encode samples with higher frequency

q Q: can we reduce the length for coding the samples?

q A: no, for prefix code

26
 Practical Issue for Prefix Code
q Recall that to achieve the minimum expected code length, the length of each codeword is

q What if is not an integer?

q Example:
Ø Encode
Ø PDF:

27
 Practical Issue for Prefix Code
q Solution: the length of each codeword is

q Example:
Ø Encode
Ø PDF:

28
 Practical Issue for Prefix Code
q Lower bound on expected code length

q Upper bound on expected code length

q If , the expected code length satisfies

29
Revisit of Morse Code

30
 Huffman Code
q If n is small, we can easily construct prefix code based on (e.g., n=3)
q What if n is very large?
q Example:
Ø Encode:
Ø PDF:

Ø How to construct prefix code?

31
 Encoding
q Example:

Root
Edge 1

a:0.42 0.58
Leaf
Huffman Tree 0.23 0.35

e:0.11 f:0.12 0.15 b:0.2

Leaf Leaf Leaf

c:0.05 d:0.1
Leaf Leaf
32
 Encoding
q Encoding rule: from root to each leaf; on each edge
Ø Turn left: 0
Ø Turn right: 1 Root
0 1 1

a:0.42 0
0.58 1
Leaf
Huffman Tree 0.23 0.35
0 1 0 1

e:0.11 f:0.12 0.15 b:0.2

0 1
Leaf Leaf Leaf

c:0.05 d:0.1
Leaf Leaf
33
 Encoding
q Encoding rule: combining all bits on edges from a root to a leaf

Root
0 1 1

a:0.42 0
0.58 1
Leaf
Huffman Tree 0.23 0.35
0 1 0 1

e:0.11 f:0.12 0.15 b:0.2

0 1
Leaf Leaf Leaf

Fewer bits to samples with higher frequency

c:0.05 d:0.1
Leaf Leaf
How to show Huffman code is prefix code? 34
 Decoding
q Decoding rule: moving from root until to a leaf

Root
0 1 1

a:0.42 0
0.58 1
Leaf
Huffman Tree 0.23 0.35
0 1 0 1

e:0.11 f:0.12 0.15 b:0.2

0 1
Leaf Leaf Leaf

c:0.05 d:0.1
Leaf Leaf
35
 Summary of Huffman Code
q Greedy algorithm to assign fewer bits to encode samples with higher frequency

36
 Encoding and Decoding
q Example:

q Use 5 minutes
Ø Construct Huffman Tree and encode each letter by yourself
Ø How to decode the letters based on codeword 01111100

a:0.42

e:0.11 f:0.12 b:0.2

c:0.05 d:0.1

37
 Huffman Code
q Example:

q Expected code length:

q Entropy of x:

38
Morse Code v.s. Huffman Code

39
Morse Code v.s. Huffman Code

Verify this

40
Part II: Lossy Compression

41
 Lossy Compression
q Lossy compression: only permit reconstruction of an approximation of the original data,
but with fewer encoded bits to store compressed data

42
 Why We Need Compression
q Image: 6.0 million pixel camera, 3000x2000
Ø RGB (color depth 24 bits or 3 bytes)
§ What is the size of each image (MB)? 18 MB
§ Suppose the storage size is 1 GB. How many images can be stored? 56 images
q Video: DVD Disc 4.7 GB
Ø Video 720x480, RGB, 30 frames/sec
§ The maximum duration of the video per DVD disc 31.1MB in 1 sec 2.5 minutes
q Watch online video using cellphone:
Ø 352x240, RGB, 15 frames/sec
§ The minimum network speed: 30.4 Mbps
§ Can you afford it? (How about 4K video)
q How to compress data with fewer bits?
43
 What Can We Compress
q Redundancy: exceeding what is necessary or normal
Ø Spatial redundancy: adjacent pixels are highly correlated

44
 What Can We Compress
q Irrelevance or perceptual redundancy
Ø Not all visual information is perceived by eye/brain, so throw away those that are not

Red

Magenta Yellow

White
Blue Green

Cyan

45
 What Can We Compress

q Redundancy in digital video

Ø Temporal redundancy: adjacent frames are highly
correlated

Animations extracted from

http://en.wikipedia.org/wiki/F
ile:Newtons_cradle_animatio
n_book_2.gif

46
 How Much Can We Compress

q The same resolution and color depth,

but different file size
q Different images have different
redundant and irrelevant contents

47