ELEC1010 Electronic & Information Technology

Chapter 4 – Coding of Digital

Information
Source Coding and Channel Coding for
Compression and Error Protection

P.1
ELEC1010 Electronic & Information Technology

Lecture 13 – Introduction to Source

Coding
• Coding is the conversion of digital information
into different forms for the purpose of storage,
transmission, or other types of processing

• Why do we need coding? Reasons include:

- Compression
- Error Detection
- Error Correction
- Secrecy
P.2
 Coding for Digital Communications
Bits 011011010……
x(t) Storage
x̂t 
Quantization Reconstruction
Sampling
e.g. of Signal  xt 
100MB/s
(800Mb/s)
Analog to Digital Conversion
(Chapter 3 Digitization)

Source Coding

Source Coding Storage Source Decoding

(compression) e.g. 10MB/s (decompression)
(80Mb/s)

Communication
Channel Coding Channel e.g. Channel Decoding
20MB/s (160Mb/s)

Channel Coding
P.3
 Coding for Digital Communications
• In Chapter 3, we discussed digitization, where the analog signal
x(t) is sampled and quantized to become a digital signal

• In this chapter we discuss source coding and channel coding

• Source coding ensures that bits are used efficiently

Compresses, or reduces the number of bits needed to represent
information

• Channel coding helps the receiver to detect and possibly

correct bit errors. It adds redundant bits to achieve this purpose

P.4
Source Coding – Compression & Decompression
In source coding, data is compressed into a smaller number
of bits for storage or transmission. Decompression is the act
of reconstructing the original signal, exactly (lossless) or
approximately (lossy), from the compressed version.

P.5
 Lossless Compression
• It is a class of compression techniques for which nothing about
the original set of numbers or symbols is lost in the compression
process.
• Lossless compression is reversible and by the reverse
decompression process, the exact set of the original numbers
can be recovered.
• Lossless compression is important for signals and information
that we count on for accuracy; e.g. text documents, computer
programs, financial and medical records, scientific data

WinZIP and WinRAR are lossless

compression software

P.6
 Lossy Compression
• It is a class of compression techniques that selectively throw
away information about the sets of numbers or symbols being
compressed. The idea is to lose those features that would not
be noticed or that are not considered important in the
application.
• Lossy compression is not reversible – you cannot get back the
exact original set of numbers from the compressed version.
• Lossy compression are most often used for signals and
information that we perceive, such as sounds and images: MP3
music, DVD movies, etc. The quality of what we see and hear
is closely tied to the limitations of our eyes and ears. By
exploiting these limits and throwing away “unnecessary” bits,
we can achieve a higher compression ratio

P.7
 Lossless Compression Example – Text File

• Consider a text file.

• It can be compressed by sophisticated algorithms.
• It can also be compressed by simple mapping.
• For a word file “Huffman”. It contains 11,864
characters, the size is 18.5 KB.
• We can use winRAR to compress it, the size
becomes 15.8 KB

P.8
Lossy Compression Example: JPEG Images

• A 1024 x 1024 bitmap picture means that the picture is

stored as a array of 1024 x 1024 pixels. Each pixel is a
picture element which is a colored dot. A full-color (RGB)
pixel is usually represented by 24 bits. So a 1024 x 1024
bitmap picture will have a file size of 1K x 1K x 3 Bytes = 3
Mbytes.

• A JPEG (Joint Photographic Experts Group) picture

compresses by encoding the frequency components in the
picture. It uses fewer bits to encode frequency
components that are not as important to the human eyes.
Therefore, there are some losses in information and the
original picture cannot be 100% recovered from the
encoded picture.
P.9
• Original Bitmap image (.bmp file), file size = 549KB

P.10
• Compressed JPEG image (.jpg file), file size = 28 KB

P.11
• Compressed JPEG image (.jpg file), file size = 13.4 KB

P.12
Essence of Source Coding: Symbol-to-bit assignment
• We can think of the information that we want to encode as a
sequence of symbols. The set of possible symbols is a set
{x} such as:
– The 26 alphabets in a book {a, b, c, d,….z} (*ignoring
capital letters)
– Boy or Girl for a group of ELEC1010 students {B, G}
– The black dot and the white dot in a fax page {B, W}
• The essence of source coding is to assign different
sequences of bits (called codewords) to represent different
source symbols.
• We have some liberty to define the set of symbols to make
our coding more efficient. For example, we may consider
students in pairs, resulting in {BB, BG, GB, GG}.
P.13
Fixed-Length Code
• A simplest type of code is fixed-length code – where every
codeword has the same code length (number of bits).

• If there are N symbols, we need a minimum of M bits, such

that 2M ≥ N

e.g. for 5 symbols we need a minimum of 3 bits; for 10

symbols we need a minimum of 4 bits

Examples:
# of Symbols Symbol Set Possible Codewords
2: {B, W} 0, 1
4: {a, e, q, z} 00, 01, 10, 11

P.14
Variable-Length Code
• A fixed-length code has the advantage that it is simple.
The number of bits is the same for all symbols.
• But, a fixed-length code can be inefficient because we do
not take into consideration of frequency of occurrence of
the symbols

• Variable-length code assigns variable length codeword to

different symbols
Examples:
# of Symbols Symbol Set Possible Codewords
4: {a, e, q, z} 0, 1, 11, 10

• Received sequence 01110 can be decode as aeeea,

aeqa, aqea, aqz

P.15
Uniquely Decodable Code – Prefix Code
• What characteristics must a variable code have so that we
can always unambiguously decode?

• A “prefix code” is one where no codeword is a prefix of

another codeword. It is uniquely decodable.
For example a = 0, e = 10, q = 110, z = 111
– “000111” can only be “aaaz”

P.16
Prefix Code – A Code Tree
• A way to think of a prefix code is to think of a tree. Each branch of
the tree corresponds to a “0” bit or a “1” bit. For a prefix code all
codewords must be at the leaves. To decode, we start from the top
of the tree and follow the branches according to the bits received
until we hit a leaf. Then we start over from the top of the tree. (But if
we have one error, everything following will be messed up! )

0 1
0 1
1 0 1
e a c
0 1
0 1 0 1

0 1 0 1
00 01 11
000 001 010 011 e b d
a b c d Non-Prefix Code ->
Prefix Code -> Not Uniquely Decodable
Uniquely Decodable
P.17
How efficient is Variable-Length Code?
• Four symbols with frequency of occurrence:
• a: 0.2 e: 0.5 q: 0.15 z: 0.15
1 e
1
1
01 a
0
001 q 1 0

000 z 0

• average bits per symbol:

= 2×0.2+1×0.5+3×0.15+3×0.15 = 1.8 bit/symbol
• For fixed-length code average bit per symbol = 2 bit/symbol
• For variable-length code the data size of such 100 symbols
is 180 bits not 200 bits, and save 20 bits.
P.18
 Summary – Lecture 13
• Explained the meaning of source coding and channel coding.

• Explained the difference between lossless and lossy source

coding and considered the TEXT document and JPEG example

• Explain source coding as assignment of codewords to

represent symbols

• Discussed fixed-length code and variable-length code

• To be uniquely decodable, a variable-length code must be a

prefix code

P.19
ELEC1010 Electronic & Information Technology

Lecture 14 – Huffman Code and MPEG

P.20
 Frequency Matters
• In the last lecture, we explained that we may use a variable-length
code because symbols may be of different frequencies
• Relative Frequency of a symbol is defined as the number of
occurrence of this symbol divided by the total number of symbols,
in a specific symbol sequence (e.g. winzip), or over many typical
symbol sequences that we need to encode. It means the same as
probability and is the p(x) in Shannon’s entropy formula!
• For simplicity, we assume that neighboring symbols are
independent, meaning that the probabilities of a symbol do not
change because of its neighbors. In reality this may not be always
true. For example, in English, the letter “q” is almost always
followed by a “u”. On a fax page, a white dot is very likely to be
followed by another white dot. (Dependency among symbols will only give
more opportunity to compress; it will not make things worse. As shown in later
example, MPEG takes advantage of dependency across pixels in video)

P.21
Prefix Code – A Code Tree
• We see that there are many ways to set up the prefix code
tree
Our 5 Symbol Example An Alternative Prefix Code Tree

0 1 0 1
1
e
0 1 1 0 1
0
01 10 11
0 1 0 1 0 1 c d e

000 001 010 011 001

000
a b c d a b
• Which is better? How to construct the best tree?
P.22
 Huffman Code
• Invented as part of a master thesis at MIT, it is the most
commonly used variable-length, prefix source coding technique
• Assume knowledge of the relative frequencies/probabilities of
all symbols to construct the best tree
• Take complex problem into multiple simple problems
• Idea: for N symbols problem, combine the 2 most infrequent
symbols then have N-1 symbols problem……
• Good prefix code should have the key properties
1. The codewords of the most infrequent symbols should be longest
in length
2. The two most infrequent symbols should have:
a) the same length
b) the same sequence except the last bit
P.23
 Huffman Code Via a Simple Example
Example: Four symbols with probabilities:
A: 0.4, B: 0.3, C: 0.2, D: 0.1

0 A
0.4 0
1
10 B
0.3 0
0.6 1
110 C
0.2 0 0.3 1

111 D 1
0.1

Avg. bit/sym = 0.4×1 + 0.3×2 + 0.2×3 + 0.1×3 = 1.9 bits/sym

P.24
 A Slightly More Complicated Huffman Example
a (0.2), b (0.1), c(0.3), d (0.15), e(0.25)
(4) Done
(1) smallest: b(0.1), d(0.15) 1
-> b+d = 0.25 0 (3) e+c (0.55)

(2) smallest: a (0.2), b+d (0.25) (2) a+b+d (0.45)

-> a+b+d = 0.45 0 1
0 1
(3) smallest e(0.25), c(0.3) e(0.25) c(0.3)
-> e+c = 0.55 (1) b+d (0.25)
a(0.2)
(4) Only two left (e+c, a+b+d) 0 1
-> done
b(0.1) d(0.15)

Result: a: 00, b: 010, c: 11, d: 011, e: 10

bit/symbol = 0.2×2 + 0.1×3 + 0.3×2 + 0.15×3 + 0.25×2 = 2.25 bits/sym
P.25
Another Huffman Example: Encoding Binary
Symbols
Can we reduce file size if we have only binary source symbols
and know their relative frequencies/Probabilities?

e.g. Students either pass (P) or fail (F) an exam:

Prob(P) = 0.8; Prob(F) = 0.2

Can we have less than 1 bit per binary symbol?

YES, we can encode pairs/strings of symbols.

In source coding, often we must intelligently convert an original

set of symbols into a different set of symbols before encoding
P.26
“Less than 1 bit per Binary Symbol”
• Suppose we use the grades of a pair of students as our symbols.
• With the assumption of independence, the probability of PP, for
example, is 0.8 x 0.8 = 0.64

Grade pairs (from a list “PPPFFPF…”) Probability Code

PP 0.64 ?
PF 0.16 ?
FP 0.16 ?
FF 0.04 ?

P.27
Finding the Huffman Code for the Grade Example

PP = 0.64, PF = 0.16, FP = 0.16,

FF = 0.04
1 0
(a) smallest freq. FF and FP (or PF)
(b) Combine FF with FP to form FF+FP +PF (0.36)
FF+FP PP(0.64)
(c) smallest freq now is FF+FP and 1 0 Most freq.
PF Least bits
(d) combine FF+FP and PF to form FF+FP(0.20)
FF+FP+PF PF(0.16)
(e) combine the last two 0
1
Put 1 on left, and 0 on right
PP = 0, PF = 10 FP = 110 and FF(0.04) FP (0.16)
FF = 111 Least freq.
Most bits
P.28
“Less than 1 bit per Binary Symbol”
• Suppose we use the grades of a pair of students as our symbols.
• With the assumption of independence, the probability of PP, for
example, is 0.8 x 0.8 = 0.64

Grade pairs (from a list “PPPFFPF…”) Probability Code

PP 0.64 0
PF 0.16 10
FP 0.16 110
FF 0.04 111
Average bit/student pair = 0.64*1 + 0.16*2 + 0.16*3 + 0.04*3
= 1.56 bit/pair
Bit/student = 0.78 (indeed less than 1 bit to represent P or F!)

P.29
Encoding Groups of Three Students

• What about grouping three students?

Grade Triplets Probability Code
PPP 0.512 0
PPF 0.128 (7) 1 100
(5) 0.256
PFP 0.128 (6) 0.488
101
PFF 0.032 (4) 0.232
11100
FPP 0.128 (2) 0.064 110
FPF 0.032 11101
(3) 0.104
FFP 0.032 11110
(1) 0.040
FFF 0.008 11111

Average bit/student triplet = 1×0.512 + 3× (0.128×3) +

5× (0.032*3+0.008) = 2.184 bit/triplet
Bit/student = 2.184/3 = 0.728
P.30
Entropy – The Limit
• Should we keep going? How about grouping students into groups
of 4, or 5, or 100?
• Shannon proved that no matter what we do, the average number
of bits needed can never be smaller than the entropy, or the
inherent randomness in the information
• For the pass/fail example:
Entropy = 0.8 × log2 1/0.8 + 0.2 × log2 1/0.2
= 0.8 × 0.322 + 0.2 × 2.322
= 0.722 bit
• This means that the result achieved using groups of 3 students is
already quite close to the limit.

P.31
 Further Notes on Huffman
• Solution is not necessarily unique but gives the same average
bits/symbol
– 0 on left or 0 on right; larger probability on left or right
– Two symbols may have the same relative freq. and we have
to pick one arbitrarily: FP and PF (0.16), and b+d (0.25) and
e (0.25) in earlier example
• Huffman coding is uniquely decodable and completely reversible
(assuming no bit errors over transmission). It is lossless.
• But even a single bit error can be disastrous. There must be a
way to limit the propagation of error. Fax encoding, for example,
has a special end-of-line codeword that enables the decoder to
get back in sync
• Huffman’s idea is used in most source coding schemes, lossless
and lossy. It is used in Fax as well as JPEG.

P.32
 Lossy Compression – MP3
• Recall from Chapter 2 that digital CD audio signals consist
of 16-bit samples recorded at a sampling rate of 44.1 kHz.
• For stereo music, there are two channel. Without
compression, a 3 minute CD quality stereo audio signals
require 254,016,000 bits, or about 30 MB of storage! To
stream the signal and play it in real time would take 1.35
Mbit per second (Mbps).
• MP3 (Audio Layer 3 of the MPEG-1 Multimedia Coding
Standard), a popular audio compression standard, greatly
reduces the data required to store and transmit audio files
by the use of lossy compression

P.33
MP3 Audio Compression
MP3 encoded audio at 192 kbps is similar to CD-quality
audio at 1.35 Mbps (according to international listening
tests).

At this compressed size (7 to 1), the data can be downloaded

from the Internet easily (even on a slow modem), and it does
not take up very much hard disk space!

The quality of the sound in the compressed format is not as

good as that of the original CD, but the difference is not
noticeable to most people.

The compression scheme that MP3 uses is called

“perceptual noise shaping.”

P.34
 MP3 Audio Compression
Perceptual noise shaping is built around the way human
hear sound.

For example,
(1) there are certain sounds that we cannot hear.
(2) there are certain sounds that we hear better than other
sounds.
(3) If there are two sounds at adjacent frequencies playing
simultaneously, we hear the louder one but cannot hear the
softer one.

So if you eliminate all the parts of the song that one does
not hear very well anyway, you can reduce the file size quite
a bit without changing the sound noticeably!
P.35
 We can just remove
all other frequency
data in this particular
time.

Instead of coding values of the sampled waveform in time domain,

MP3 actually encodes the values of the different frequency
components in the frequency domain. This is common in audio
visual coding which can take advantage of the fact that human is
really perceiving the frequency components in the signal
P.36
A Simple MP3 Encoder
Input music
Convert the signal from time domain
to frequency domain

Time Domain to Perceptual

Frequency Domain Noise shaping
Conversion Analysis

Quantization and
Huffman Coding

Perceptual noise shaping

MP3 music

P.37
MPEG – A Family of Audio-Visual Coding Standard
• MPEG (pronounced M-peg) stands for Moving Picture
Experts Group. It is the name of the family of standards
used for coding audio-visual information (e.g., movies,
video, music) in a digital compressed format.
• The major advantage of MPEG compared to other video
and audio formats is that MPEG files are much smaller
for the same quality. This is because MPEG uses very
sophisticated compression techniques.
• MP3 is part of the MPEG standard.

P.38
 MPEG-2 for Cable TV and DVD
• MPEG-2 is the first popular MPEG standard. It is typically used to
encode audio and video for broadcast signals, including direct
broadcast satellite and Cable TV. With some modifications, it is
also the coding format used by standard commercial DVD movies
and HDTV (High Definition Television).
• Key techniques used include inter-frame prediction and coding in
the frequency domain (Discrete Cosine Transform coding/DCT).
• In predictive coding, the coder and decoder use an algorithm to
predict what many intermediate frames should be based on some
anchor frames. The coder encodes only the difference between the
actual frame and the predicted. If this difference is small most of
the time, there is not much information to encode.
• DCT takes advantage of the fact that most signal energy is at low
frequencies and that there are often many consecutive zero values
in the amplitudes of adjacent frequency components
P.39
 MPEG-4
• MPEG-4 is a more advanced standard for video and audio
compression introduced in late 1998. (HKUST played a key role
here!)
• The MPEG-4 standard is now used in web (streaming media) and
CD distribution, conversation (videophone), video-conference and
broadcast television.
• MPEG-4 absorbs many of the features of MPEG-2, and adds
advanced techniques such as motion estimation – predicting
how blocks of pixels will move across the picture in successive
frames.
• Most features included in MPEG-4 are optional and left to
individual developers to decide whether to implement them.

P.40
 Summary – Lecture 14
• Introduced the Huffman code as an algorithm for symbol-to-
codeword assignment. The Huffman code minimizes the average
number of bits needed to encode symbols by assigning long
codewords to less likely symbols.

• Huffman works by repeatedly combining the two least likely

symbols/sub-trees to form a bigger sub-tree. Huffman is used as
part of almost all source coding algorithms, lossless or lossy

• Demonstrated entropy as the fundamental limit on the least

number of bits needed to encode information

• Described MP3 as a lossy perceptual source coding algorithm

that takes advantage of limits in human’s perception ability

• MPEG as an advanced lossly video coding example. It uses

prediction and frequency domain coding (same for MP3).

P.41
ELEC1010 Electronic & Information Technology

Lecture 15 – Error Detecting Codes

• “To err is human”

• It is also the nature of data transmission (and storage).

P.42
Errors in Digital Transmission
• In the transmission and storage of digital information, errors
invariably occur. Signals suffer from attenuation, distortion and
noise in the process of transmission, and there can be scratches
or defective elements in CD and electronic memories.
• Bit error means that a “0” bit is corrupted into a “1” and vice
versa. A single bit error can lead to a large absolute error

e.g. Consider a uniformly quantized 8-bit data:

241 = 128 + 64 + 32 + 16 + 1 = 11110001

If one of the bits is erroneous, it can be

Received Dec. value Error
01110001 = 113 128
10110001 = 177 64
11010001 = 209 32
…
P.43
 Channel Coding
• To protect data transmission from error, bit streams can be
encoded before transmission in what is called channel coding.
• Channel encoding adds extra bits to create systematic
redundancy to gain reliability is at the expense of efficiency.
• Channel coding can be used for:
1. Error detection: detection of error in the sequence
2. Error correction: not only detect the error but also
correct the error in the sequence
• A code is defined by the set of allowable “codewords.”

P.44
Error Detection
• Detecting error allows a receiver to identify transmission errors and take
actions such as:
 ignore the received signal and reconstruct from other signals (e.g. in
GSM-based 2G mobile communications)
 request re-transmission (e.g. in web protocol)

• How to detect errors?

Use only a subset of possible binary sequences. Anything that is
outside that subset is an error.

• How to define the subset of possible binary sequences?

Add an extra bit (so called “parity bit” or “check bit”) to the stream such that
the number of 1s is always even number for instance

e.g. original 7-bit sequence coded 8-bit word with even parity bit
1110001 11100010
1011110 10111101

P.45
Error Detection Code
• The set of allowable encoded sequences, called
codewords, are only a subset of possible sequences, e.g.

Input (3 bits) Codewords (even parity, 4 bits)

000 0000
001 0011
010 0101
011 0110
100 1001
101 1010
110 1100
111 1111

0001, 0010 … are odd parity and are invalid codewords

Therefore, any single bit error can be detected.
P.46
Example: ASCII Standard for Alphanumeric Data
• ASCII is a standard for encoding alphanumeric characters
(upper and lower alphabet letters, numerals, and punctuations).
Originally designed for the transmission of characters between
user terminals and the computer, ASCII adds a parity bit to each
7-bit character (so up to 27 = 128 characters allowed) to form an
8-bit code.
• One may choose either even parity or odd parity. For even
parity, the binary code to be sent should have an even number of
“1”. A similar definition applies to odd parity codes. For example:
ASCII even parity odd parity
A = 100 0001 0100 0001 1100 0001
g= 110 0111 1110 0111 0110 0111

P.47
Parity Check Detects All Errors with Odd Number of
Bit Errors
• Suppose we use even parity, and an “e” (= 110 0101) is keyed in.
The “e” is then encoded as “0110 0101” to be sent out.
• Now, if the code is received as “0110 1101”, which consists of 5
“1”s, we immediately know that it is an invalid code. The sender
will be requested to resend the data.
• Clearly, if two errors occur, for example, the intended “e” is
received as “0100 1101”, then we may incorrectly decode it as an
“M”. Engineers who design the data transmission system should
ensure that it is very unlikely to have two errors in the same
transmission.
• Parity check achieves error detection of all odd number of errors.

P.48
Error Detection: HKID and Credit Card Number Examples
• When people provide their ID or credit card numbers over the
phone, there may be common errors such as flipping two
adjacent numbers
• We can use algorithms similar to parity check with special
focus on the common error types

Example 1: Hong Kong ID number

Example 2: Credit Card Number

P.49
A Common Error Detection Method
• Basic Principle

– Most ID/credit card number verification apply the Error Detection

method known as “Luhn algorithm.”

– Luhn algorithm can:

(1) Detect any single-digit error

(2) Detect the transpositions of adjacent digits
(e.g. 1234567 -> 2134567)

– This kind of detection can help make sure the ID card or credit
card number provided by the user is a valid number.

(Caution that a “valid” card number does not necessarily mean

this card number really exists in the system !!)

P.50
 Hong Kong ID Number (I)

Is this HKID number valid?

Ans: check the last digit “check bit (E)”

following the steps below

1. First write down the first seven digits of your ID Card Number (including the
first English letter) e.g. C668668

2. Convert the first English letter into a number according to the following table

1 2 3 4 5 6 7 8 9 10 11

A B C D E F G H I J K e.g. C=3
L M N O P Q R S T U V

W X Y Z

P.51
Hong Kong ID Number (II)
3. Multiply the first SEVEN numbers by 8,7,6,5,4,3 and 2, respectively.
3x8 = 24
6x7 = 42
6x6 = 36
8x5 = 40
6x4 = 24
6x3 = 18
8x2 = 16

4. Find their sum 24+42+36+40+24+18+16= 200

5. Find the remainder when the sum is divided by 11 200 / 11 = 18

remainder 2

6. Subtract the remainder from 11 (i.e. 11-2 = 9): the result is the check digit
-- If the result = 11 (remainder = 0)  check digit = 0
-- If the result = 10 (remainder = 1)  check digit = A

The correct ID should be C668668(9)

P.52
 Block Codes
• A common type of channel code is block code in which
the input is encoded one block at a time. Shown below
is an even parity block code:
input stream divide into
001010100101010 blocks
001 010 100 101 010….

encoder
encoded stream (even parity)
0011 0101 1001 1010 0101
Transmission

Note : some errors could not be detected.

encoded data - 0011(even parity),
incorrect received data – 0101(even parity)
P.53
 Summary – Lecture 15
• Introduced the concept of channel coding for error detection or
error correction

• Introduced the concept of even and odd parity check codes and
the ASCII standard as an example

• Introduced the concept of block code which encodes data in

blocks

• Discussed generalization of parity check code to Luhn’s algorithm

and block code

P.54
ELEC1010 Electronic & Information Technology

Lecture 16 – Error Correcting Codes

P.55
 Error Correcting Codes
• Suppose we receive 1011 in the even parity code example,
we know
1. there is an error because the parity is NOT even
2. If the transmission introduces one error, possible correct
codewords includes 0011, 1001, 1010, 1111. We cannot
tell which one is more likely!

• What should one do when 100% correct decoding of the

original data is needed?
- Retransmission? For recording on a CD there is no
retransmission. For a mission to Mars the
retransmission delay is very large
- Automatic correction is desirable!! How?

P.56
 Simple Approach: Repetition Code
• It is possible to correct a limited number of errors using error
correcting codes. For example, the “repetition code” for 1
bit (repeating the input bit for at least three times)

Input bit Codeword (Encoded Sequence)

0 000
1 111
Received Decoded
000, 001, 010 or 100 0 because more ‘0’s than ‘1’s
110, 101, 011 or 111 1 because more ‘1’s than ‘0’s

However, if we have a 2-bit or 3-bit error patterns, we will

decode incorrectly!!
Also, the code seems to be very inefficient – it uses 3 bits to
transmit 1 bit --- 3 times more!!
P.57
 Two-Dimension Parity Code
• Another simplified example of an
1 0 1 1 1
error-correcting code is the two-
dimension parity check code 0 1 1 1 1
• We arrange the data in a N x N 1 0 1 0 0
block. We add a parity bit for each 0 1 0 0 1
column, and a parity bit for each row
0 0 1 0
• We see that all single-bit errors can
be corrected

1 0 1 1 1 1 0 1 1 1
0 1 0 1 1 0 1 1 1 1
Bit Error! Violation
1 0 1 0 0
1 0 1 0 0 0 1 0 0 1
0 1 0 0 1 0 0 0 0
Violation
0 0 1 0
Violation
P.58
 Efficiency of Error Correcting Codes
• Efficiency, or the code rate is defined as
No. of Data Bits
Code Rate (R) =
No. of Data Bits + No. of Parity Bits
• For 2-dimension parity code, transmit N x N data bits and
2N parity bits. Code Rate is N2 / (N2 + 2N) = N/(N+2)
• For N = 2, we refer to the code as a (8, 4) code. Transmit 8
bits for every 4 data bits. efficiency, or code rate, is 4/8.
• The 2-dimension parity code can correct all 1-bit error
patterns in a data block of N x N bits
• For the same block size, we want to find codes with code
rates (or efficiency) closer to 1

P.59
 Structured Error Correcting Codes
• Even a code randomly designed is a GOOD code!!
• However, no structure means difficult to decode, i.e. finding
the bits difference between the received sequence and all
the possible 2k (e.g. for k=number of bits=100, 2100)
codewords has high computational complexity
• Solution: Having structure using mathematics (e.g. discrete
mathematics) to enable efficient decoding algorithms
• Example:
• Reed Solomon or BCH Code: used in CD, VCD, DVD
• Convolutional Code: used in Wireless communication

P.60
 Summary – Lecture 16

• Introduced the repetition code and 2-D parity

check as illustrations of error correcting
codes. These two are only as simple
illustrations; they are seldom used in
practice.

P.61
ELEC1010 Electronic & Information Technology

Lecture 17– Channel Capacity

P.62
 What is a Communication Channel?
• Examples of a Communication Channel:
• A wire/cable connecting two points that take a signal from
point A to point B, e.g. a copper wire, an fiber optic cable
• The space where a Electromagnetic (EM) wave transmits
from the transmit antenna to the receive antenna
• Even a storage media where bits are stored and then
retrieve later, e.g. Hard disc drive, optical disc such as CDs,
VCDs, DVDs,

P.63
 Models for Communication Channel?
• Model for a channel:
• Send a signal x(t) over a communication channel and
receive y(t) at the other end
• y(t) depends on x(t), e.g. y(t) = a x(t) where a is the
attenuation factor
• However, in general y(t) is a possible distorted and usually
corrupted by noise and possibly interference.

P.64
Simplest Binary Symmetry Channel Model
• Transmit a bit (i.e. ‘1’ or ‘0’) and receive a bit (i.e. ‘1’ or ‘0’)
• Most of the time (with probability 1- ε where ε is generally very
small), you get exactly what was transmitted. No error
• Occasionally, (with probability ε), the transmitted bit is flipped
(i.e. a bit error, or a ‘1’ becomes a ‘0’, and a ‘0’ becomes a ‘1’)
• The error could be caused by noise, interference, etc, but the
final effect is that a bit error occurs.
• Symbol rate gives the number of channel symbol per second
• Diagram for the BSC model:

1 1


0 0


P.65
 Shannon Capacity
• “Channel Capacity” is another revolutionary concept from
Shannon who defined the “Capacity” of a channel as the
maximum information (or data) rate that can be
transmitted reliably over the channel.
• He proved that if the data rate is below the channel capacity,
given that we are willing to accept large blocks, large coding
delay, we can achieve arbitrarily low error probability
• For Example, if a BSC with symbol rate 1k symbol/second has
an error probability equal to 0.1
• Data rate is1kbps and bit error rate is 0.1
• With repetition code (3 times), Data rate is 1/3 kbps and
block error rate is 0.028 (0.028=0.13+3*0.12*0.9)
• Question: What is the maximum data rate for a given error
rate??
• Shannon: For arbitrarily small error rate, max data rate is
CAPACITY P.66
 Capacity of a Binary Symmetric Channel
• Assume a Binary Symmetric channel with a bit error rate .
Shannon defines the capacity of the channel as follow:
•
Capacity = 1 +  log2  + (1-) log2 (1-)

𝐶𝑎𝑝𝑎𝑐𝑖𝑡𝑦 = 1 − 𝐸𝑛𝑡𝑟𝑜𝑝𝑦
= 1 − 𝐻(𝜀)
= 1 − 𝜀𝑙𝑜𝑔 + 1 − 𝜀 𝑙𝑜𝑔
= 1 + 𝜀𝑙𝑜𝑔 𝜀 + 1 − 𝜀 𝑙𝑜𝑔 1 − 𝜀


1 1


0 0


P.67
Shannon’s Channel Capacity
Channel Capacity

1
The curve to the right 0.9

shows the channel 0.8

capacity of a binary 0.7

symmetric channel 0.6

with random bit error 0.5

rate . 0.4

0.3

0.2

0.1

0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

P.68
Error-Free Communications
• Here is the surprising part of Shannon’s theory. Assume we
transmit b bits over the channel with a coding rate of r,
Shannon shows that if r < capacity, theoretically it is possible to
correct all error bits with certainty when b becomes large!

• Example: assume  = 0.2.

Channel Capacity
= 1 + 0.2 x log2 0.2 + 0.8 x log2 0.8 = 0.278

• This means that one can find a code, such that for every 0.278
data bit transmitted, there are at least 0.722 protection bits, we
can decode and correct all errors with almost certainty.

P.69
Approaching Shannon’s Limit
• To achieve Shannon’s limit, b must be large. That means large
delay and a large codebook (i.e., many symbols).
• Modern theories on error correcting codes are to find algorithms
to approach Shannon’s limit while keeping b small and the
encoding and decoding simple.
• Such theories make use of advanced mathematics (discrete
mathematics) and there have been many exciting inventions!
• Another very important class of error correcting codes which does
not operate on blocks of data bits is the convolutional code.

Galois (1811-1832)
His mathematical field theory provided the
foundation for discrete mathematics which The Rubik’s cube can be
allowed us to develop very efficient block codes. formulated as a discrete math
Unfortunately he was killed in a duel at age 20. problem
P.70
Error Correcting Codes in Daily Lives
• For example, all of our CD and DVD recordings use an error-
correcting block code known as the Reed-Solomon code. For
example, a (255, 223) Reed-Solomon code encode 223 bits of
data into 255 bits, adding 32 bits of redundancy. This makes our
music and video readable even when there are defects and
scratches on the disc.
• The wireless channel is very noisy and error-prone. The GSM 2G
mobile phone network, for example, uses different speech source
coding schemes together with a ½ rate of convolutional code (there
are as much redundancy as data bits!) to give us an acceptable level
of voice quality on the mobile phone.
• The modern world cannot function today without error correcting
codes!

P.71
 Summary – Lecture 17
• Introduced the Binary Symmetric Channel

• Introduced Shannon’s concept of channel capacity. Shannon

showed that in principle, arbitrarily reliable communications
(which bit error rate arbitrarily close to 0) can be achieved as long
as the information rate is kept below the channel capacity

• Introduced the notion of discrete mathematics for design highly

efficient block codes.

P.72