ENTROPY CODING TECHNIQUES
RITHIKA
ROLL NO.12
S2, M.Tech, ece
ENTROPY CODING TECHNIQUES
Huffman coding
Shannon Fano Coding
Arithmetic coding
Run Length Coding
Huffman Coding
• It is a type of variable length coding
• Here coding redundancy can be eliminated by choosing a better way of assigning
the code.
• After compression, a file can be restored to its original form.
• Uses : text, sound & image etc.
The Huffman Coding algorithm is given as :
1. List the symbols and sort probabilities per symbol
2. Combine the lowest two probabilities of symbols and label the new code with it
3. Newly created item is given priority and placed at the highest position in the
sorted list
4. Repeat Step2 until only one node remain
5. Assign code 0 to higher up symbol and 1 to the lower down symbol
6. Now trace the code symbols going backwards
Example: Calculate the Huffman Codes for the set of five symbols A, B, C, D & E with probabilities
0.15, 0.19, 0.4, 0.16 and 0.15. Determine code efficiency and redundancy.
Symbols Stage 1 Stage 2 Stage 3 Stage 4
C 0.4 0.4 0.4 0.65
B 0.19 0.3 0.35 0.4
D 0.16 0.19 0.3
A 0.15 0.16
E 0.15
Symbols Stage 1 Stage 2 Stage 3 Stage 4
C 0.4 0.4 0.4 0.65 0
B 0.19 0.3 0.35 0 0.4 1
D 0.16 0.19 0 0.3
1
0 Symbols Prob. Tracing Codeword
A 0.15 0.16 Digit
1
C 0.4 1 1
E 0.15 B 0.19 000 000
1 D 0.16 100 001
A 0.15 010 010
E 0.15 110 011
• Average length(L )
= 1×0.4 + 3×0.19 + 3×0.16 + 3×0.15 + 3×0.15
= 2.35
• Entropy H(s)
• Efficiency (Ƞ) = H(s) / L
= 2.224 / 2.35
= 0.94
= 94 %
Arithmetic Coding
• In this coding technique, a string of characters like the words “ welcome home” is
represented using a fixed number of bits per character, as in the ASCII code.
• When a string is converted to arithmetic encoding, frequently used characters are
stored with fewer bits and not so frequently occurring characters are stored with
more bits, resulting in relatively fewer bits used in total.
The Arithmetic Coding algorithm is given as :
1. Fix the range as low = 0.0 ; high = 1.0
2. Allocate a range for given probabilities ( cumulatively).
3. Encoder:
Upper limit(UL) = Low Limit(LL) + (UL – LL) × probability distribution of the
symbol
Or
UL = LL + (UL – LL) × P
Example : Code the string INDIA using arithmetic coding
Symbol Probability CDF
I 2/5 = 0.4 0.4
N 1/5 = 0.2 0.6
D 1/5 = 0.2 0.8
A 1/5 = 0.2 1
I N D I A
1 0.4 0.24 0.224 0.2144 0.2144
A
( I ) UL = LL + D×P
0.8 0.32 = 0 + (0.4) ×0.4 =0.16
0.2208 I 0.2128
0.224
= 0 + (0.4) ×0.6 = 0.24 N 0.21408
= 0 + (0.4) × 0.8 =0.32 D
D 0.2116
0.208 0.2176
0.6 0.24 0.21376
( N ) UL = LL + D×P
N = 0.16 + (0.08) × 0.16 = 0.192
0.16 0.2144 = 0.16 + (0.08) ×0.2104
0.24 = 0.208
0.4 0.192
= 0.16 + (0.08) × 0.32 = 0.224 0.21344
I
0 0.208 0.208 0.2128
0 0.16
So the transmitted code for INDIA may be 0.2144 or 0.2128
………………. THANK YOU …………..
