Review of Information Theory

Prof. Dr. Mahmood F. Mosleh

 Random Variables
There are two types of random variables, discrete and continuous. All random variables
have a cumulative distribution function.
1. Discrete Random Variables: A discrete random variable is one which may take on only a
countable number of distinct values such as 0,1,2,3,4,........ The probability distribution of a
discrete random variable is a list of probabilities associated with each of its possible values.

The probabilities must satisfy the following:

a) for each i
b) or,
2. Continuous Random Variables: It is one which takes an infinite number of possible values. A
continuous random variable is not defined at specific values. Instead, it is defined over an interval of
values, and is represented by the area under a curve. The curve, which represents a function p(x),
must satisfy the following:
c) The curve has no negative values (p(x) > 0 for all x)
d) The total area under the curve is equal to 1.
A curve meeting these requirements is known as a density curve.
 Logarithmic Measure of Information
The amount of self-information contained in a probabilistic event depends only on
the probability of that event: the smaller its probability, the larger the self-information associated
with receiving the information that the event. 10

Where is self information of and if: 7

i. If “a” =2 , then has the unit of bits 6

I(xi) b its
5
ii. If “a”= e = 2.71828, then has the unit of nats 4

iii. If “a”= 10, then has the unit of hartly 3

0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
 Properties of P(xi)

1)  0 (a real nonnegative measure)

2) = + for independent event.
3) is a continuous function of .
 Average Information (entropy)
In information theory, entropy is the average amount of information contained in each message received. If the source
produces not equiprobable messages then are different. Then the statistical average of over i will give the average
amount of uncertainty associated with source X. This average is called source entropy and denoted by given by:

Example: Find the entropy of the source producing the following messages:

 
Solution:
 Binary Source Entropy
In information theory, the binary entropy function, denoted H(X) or Hb(X) is defined as the entropy of a Bernoulli
process with probability p of one of two values. Mathematically, the Bernoulli trial is modelled as a random variable X
that can take on only two values: 0 and 1:

We have:

Then:

If

For binary source, if , then the entropy is:

Note that is maximum equal to 1(bit) if:
For any non-binary source, if all messages are equiprobable, then so that:
 Source Entropy Rate
It is the average rate of amount of information produced per second.

The unit of H(X) is bits/symbol and the rate of producing the symbols is symbol/sec, so that the unit of R(X) is bits/sec.
Where
is the average time duration of symbols, is the time duration of the symbol .
Example: A source produces dots ‘.’ And dashes ‘-‘ with P(dot)=0.65. If the time duration of dot is 200ms and that for a
dash is 800ms. Find the average source entropy rate.
 Mutual information for noisy channel
Consider the set of symbols , the transmitter my produce. The receiver may receive . Theoretically, if
the noise and jamming is neglected, then the set X=set Y. However and due to noise and jamming, there
will be a conditional probability
The amount of information that provides about is called the mutual information between and . This is
given by:

Properties of :
1. It is symmetric, .
2. if aposteriori probability > a priori probability, provides +ve information about .
3. if aposteriori probability = a priori probability, which is the case of statistical independence when
provides no information about .
4. if aposteriori probability a priori probability, provides -ve information about , or adds ambiguity.
Also
 Joint Entropy & Conditional Entropy
In information theory, joint entropy is a measure of the uncertainty associated with a set of variables.

The conditional entropy quantifies the amount of information needed to describe the outcome of a random
variable Y given that the value of another random variable X is known.
Transinformation (average mutual information)
It is the statistical average of all pair . This is denoted by and is given by:

or

The relationship between joint, conditional and transinformation:

Where, the is the losses entropy. Also we have:

Example: The joint probability of a system is given by:

Find: Marginal entropies, Joint entropy, Conditional entropies, Transinformation.

1.

2.  

H
Source Coding
Prof. Dr. Mahmood F. Mosleh
 Source coding
An important problem in communications is the efficient representation of data generated by a discrete source. The
process by which this representation is accomplished is called source encoding. An efficient source encoder must
satisfies two functional requirements:

i. The code words produced by the encoder are in binary form.

ii. The source code is uniquely decodable, so that the original source sequence can be reconstructed perfectly
from the encoded binary sequence.
 Code Efficiency

We have:

A code efficiency can therefore be defined as:

The overall code length, L, can be defined as the average code word length:

The code efficiency can be found by:

 Types of Source Code
1- Fixed- Length Code Words:

If the alphabet X consists of the 7 symbols {a, b, c, d, e, f, g}, then the following fixed-length code of block
length L = 3 could be used.

The encoded output contains L bits per source symbol. For the above example the source sequence bad...
would be encoded into 001000011... . Note that the output bits are simply run together (or, more technically,
concatenated). This method is nonprobabilistic; it takes no account of whether some symbols occur more
frequently than others, and it works robustly regardless of the symbol frequencies. This is used when the
source produces almost equiprobable messages or,

, then

for binary coding then:

1. bit/message if ( and r is an integer) which gives

2. bits/message if which gives less efficiency

 Examples for Fixed Code
Example
For ten equiprobable messages coded in a fixed length code then
and bits
and
Example: For eight equiprobable messages coded in a fixed length code then
and bits and
Example: Find the efficiency of a fixed length code used to encode messages obtained from
throwing a fair die (a) once, (b) twice, (c) 3 times.
 Variable-Length Code Words
When the source symbols are not equally probable, a more efficient encoding method is to use variable-
length code words. However such code is not uniquely decodable, even though the codewords are all
different. If the source decoder observes 01, it cannot determine whether the source emitted (a b) or (c).

To address this problem, the Prefix-free codes must be used: A prefix code is a type of code system
(typically a variable-length code) distinguished by its possession of the "prefix property", which requires
that there is no code word in the system that is a prefix (initial segment) of any other code word in the
system. For example:
 Huffman Code
The Huffman coding algorithm comprises two steps, reduction and splitting:
Example: Design Huffman codes for having the probabilities

The average code word length: . The source entropy:

bits/symbol. The code efficiency:
 Huffman Code
H. W
Develop the Huffman code for the following set of symbols

Symbol A B C D E F G H
Probability 0.1 0.18 0.4 0.05 0.06 0.1 0.07 0.04
 Shannon- Fano Code
In Shannon–Fano coding, the symbols are arranged in order from most probable to least probable, and
then divided into two sets whose total probabilities are as close as possible to being equal. All symbols
then have the first digits of their codes assigned; symbols in the first set receive "0" and symbols in the
second set receive "1". As long as any sets with more than one member remain, the same process is
repeated on those sets, to determine successive digits of their codes.
Example: Develop the Shannon - Fano code for the following set of messages, then find the code efficiency.

Code
bits/symbol
0.35 0 0   2
bits/symbol
0.2 0 1   2

0.15 1 0 0 3

0.12 1 0 1 3

0.10 1 1 0 3

0.08 1 1 1 3
 Shannon- Fano Code, with r=3
Code

0.35 0   1
ternary unit/symbol
0.2 1 0 2
ternary unit/symbol
0.15 1 1 2
𝐻(𝑋)
𝜂= × 100 % =91 . 636 % 0.12 2 0 2
𝐿𝐶
0.10 2 1 2

0.08 2 2 2
 Shannon Code

For messages , , ,… with probabilities , , ,… then:

1) if
2) if
Also define
then the codeword of is the binary equivalent of consisting of bits.
 Shannon Code, Example
Develop the Shannon code for the following set of messages,
, then find: (a) Code efficiency, (b) at the encoder output

0.3 2 0 00 2

0.2 3 0.3 010 2

0.15 3 0.5 100 2

0.12 4 0.65 1010 2

0.10 4 0.77 1100 2

0.08 4 0.87 1101 1

0.05 5 0.95 11110 1

Data Compression
By
Noor Dhyaa
Data Compression:
In computer science and information theory, data compression, source
coding, or bit-rate reduction involves encoding information using fewer
bits than the original representation. Compression can be either lossy or
lossless.
A. Lossless data compression
algorithms usually exploit statistical redundancy to represent data more
concisely without losing information.
B. Lossy data compression
is the converse of lossless data compression. In these schemes, some loss
of information is acceptable.
• The compression ratio Cr is then defined as:

• The saving ratio Sr is then defined as :

• Higher saving ratio indicate more effective compression while negative

ratios are possible and indicate that the compressed image has larger
memory size than the original.
 Run-Length Encoding
(RLE):
Run-Length Encoding is a very simple lossless data compression technique
that replaces runs of two or more of the same character with a number
which represents the length of the run, followed by the original character;
single characters are coded as runs of 1.
Example:
Input:
AAABBCCCCDEEEEEEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Output: 3A2B4C1D6E38A
Example :
Consider these repeated pixels values in an image … 0 0 0 0 0 0 0 0 0 0 0 0 5
5 5 5 0 0 0 0 0 0 0 0 we could represent them more efficiently as (12, 0)(4, 5)
(8, 0)
24 bytes reduced to 6 which gives a compression ratio of 24/6 = 4:1.
Example:
Original Sequence (1 Row): 111122233333311112222 can be
encoded as: (4,1),(3,2),(6,3),(4,1),(4,2).
21 bytes reduced to 10 gives a compression ratio of 21/10 =
21:10.
Example:
Original Sequence (1 Row): – HHHHHHHUFFFFFFFFFFFFFF
can be encoded as: (7,H),(1,U),(14,F) .
22 bytes reduced to 6 gives a compression ratio of 22/6 = 11:3 .
7
LZW (Lempel–Ziv–Welch) Compression technique

• The LZW algorithm is a very common compression technique.

• This algorithm is typically used in GIFs.
• It is lossless, meaning no data is lost when compressing.
• LZW compression works by reading a sequence of symbols, grouping
the symbols into strings, and converting the strings into codes.
Because the codes take up less space than the strings they replace,
we get compression.
 (New string)

Saving ratio =(14 -9) /14

= 35%
 New string

Saving ratio =(12 -9) /12

= 25%
Convolutional Code
 Introduction

• Convolutional coding were introduced in 1955 by Elias.

• Convolutional codes offer an approach to error control coding substantially
different from that of block codes
• Convolutional encoders are linear and time-invariant system given by
the convolution of a binary data stream with generator sequences.
• They can be implemented by shift registers.
• Can achieve a larger coding gain than can be achieved using a block coding
with the same complexity
 Convolutional Code

• The information, consisting of individual bits, is fed into a shift register in order to be
encoded.
A convolutional encoder is generally characterized in (n, k, or (k / n, K )
form
K)
where :
– k inputs and n outputs
• In practice, usually k=1 is chosen.
Rc  k / n is the coding rate, determining the number of data
bits per coded bit.

– K is the constraint length of the convolutinal code (where the encoder has
K-1 memory elements).

S is the number of memory elements

The encoder shown is a (2, 1, 3), 1/2 code rate, and 4 possible states
Convolutional Codes

Convolutional encoder
 Convolutional Encoder

 Convolutional encoder (rate ½, K=3)

– 3 shift-registers, where the first one takes the incoming data bit
and the
rest form the memory of the encoder.

c1
Input data bits

c2

Different drawings of the same encoder

 Non-Systematic Convolutional (NSC) Codes

Here is an example of a NSC code with a

c1(t) constraint length K = 3and generator polynomial

G(D) = [1 + D + D2, 1 + D2]

To make the generator polynomials more compact

1 1 1 we convert them to a binary string and group the
bits into threes to write them in octal form
m(t)
D D In this case 1 + D2 is 101 and 1 + D + D2 is
111.
1 0 1 101 is 58, where the subscript 8 indicates it is
an
octal number. Similarly, 111 is 78

c2(t) Therefore, this is the (7, 5)8 NSC code

 Using the (7, 5)8 NSC Code to Encode
n=2 ,k=1, K=3 convolutional encoder the Message 101000

c1(t) Output calculation:

Input m(t) S1 S2 c1 (t)  m(t)  S1 
c2(t) S2
Encoding Process (Initialised state S1S2 = 00) c2 (t)  m(t)  S 2
At time t = 1 1 At time t = 2
1

1 0 0
0 1 0
1
0

At time t = 3 0

1 0 1

0
 Using the (7, 5)8 NSC Code to Encode the
Message 101000

n=2 ,k=1, K=3 convolutional encoder

At time t = 4 1 At time t = 5
1

0 1 0 0 0 1
0 1

At time t = 0
6
0 0 0

The codeword is [c1(1)c2(1), c1(2)c2(2), c1(3)c2(3), c1(4)c2(4), c1(5)c2(5), c1(6)c2(6)]

=[11 10 00 10 11 00]
 State Table of a Convolutional Code

• The memory elements can contain four possible values: 00, 01, 10, 11
• These values are called states and only certain state transitions are allowed. For
example, we cannot go from state 00 to state 11 in one step or stay in state 10.
• All possible state transitions, along with their corresponding input and
outputs, can be determined from the encoder and stored in a state table.
• The state table for the (7, 5)8 NSC code is shown below:
Input Initial State Next State Output
S1 S2 S1 S2 c1 c2
0 0 0 0 0 0 0
1 0 0 1 0 1 1
0 0 1 0 0 1 1
1 0 1 1 0 0 0
0 1 0 0 1 1 0
1 1 0 1 1 0 1
0 1 1 0 1 0 1
1 1 1 1 1 1 0
 State diagram of a Convolutional
Code
• A more compact form of the state table is the state diagram
• The state is represented by the content of the memory, i.e., the (K- 1)k previous bits,
namely, the (K-1)k bits contained in the first (K-1)k stages of the shift register. Hence,
there are 2 (K-1)k states.
1/10 Input/Output

11
What is the codeword for the message
1/01
m = [1 0 0 1 1 1]?
0/01 0/10

01 10

1/00
0/11 1/11

00

Start at the all-zero state

0/00
Tree Diagram of a Convolutional Code

• A Tree Diagram is obtained from the state

table and is one method of showing all
possible code words up to a given time.
• A input of 0 is represented by moving to
an upper branch in the tree and an input
of 1 corresponds to moving to a lower
branch.
• The states are denoted by a, b, c
and
11 d: a = 00, b = 01, c = 10, d =
• The line is the input 1, 0, 1, 0, which has
the codeword 11, 10, 00, 10
 Trellis Diagrams

State Input/Output

• One problem with the tree diagram is that the 00 0/00

number of branches increase exponentially
0/11
• We can represent all possible state transitions
with a graph, where nodes represent states and
edges represent transitions. Each edge is labelled 01 1/11

with an input and its corresponding coded output. 1/00

• By making copies of the graph and joining them 0/10

together we create a trellis with each path 10 0/01
corresponding to a codeword.
1/01

11 1/10
 Trellis Diagrams

• A trellis diagram of the (7, 5)8 NSC code is given below

• The red line shows the message 10101 giving a codeword 11 10 00 10 00
State t=1
t=2 t=3 t=4 t=5
 Convolutional Encoder

Example1: Consider the binary convolutional encoder with constraint length K=3, k=1, and
n=3. The generators are: g1=[100], g2=[101], and g3=[111]. The generators are more
conveniently given in octal form as (4,5,7)

Q1 Q2 Q3
[g1]=[1 0 0], C1=Q1
[g2]=[1 0 1] , C2=Q1+Q3
[g3]=[1 1 1 ], C3=Q1+Q2+Q3
 State Table of a Convolutional Code

draw the state diagram

For example, for data sequence :

1011….
output will be: 111 001 100 110
 Tree Diagram

convolutional encoder3 =n ,1=k ,3K=

Input bit:
101

The state of the first (K-1)k

stages of the shift register:
a=00; b=01;
Output bits:
111 001 c=10; d=11
100
Trellis Diagram
Example2: Consider the 2/3 rate convolutional encoder with generators
[g1]=[1011], [g2]=[1101], [g3]=[1010](in octal these generators are(13,15,12).

C1=Q1+Q3+Q4

C2=Q1+Q2+Q4
C3=Q1+Q3
State Table of a Convolutional Code
Example of State Diagram (2)
Example of Trellis Diagram (2)

K=2, k=2, n=3 convolutional code