Scribd
Upload
49 views

Uncertainity, Information, and Entropy

1. The document discusses discrete memoryless sources and defines entropy as a measure of the average information content per source symbol. 2. Entropy is bounded from 0 to the log base 2 of the number of possible symbols, with 0 representing certainty and the maximum representing total uncertainty. 3. According to Shannon's source coding theorem, the minimum possible average code word length is the entropy, so an efficient source encoder will have an average code word length approaching the entropy.

Uploaded by

Harsha
Copyright
© Attribution Non-Commercial (BY-NC)
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
49 views

Uncertainity, Information, and Entropy

1. The document discusses discrete memoryless sources and defines entropy as a measure of the average information content per source symbol. 2. Entropy is bounded from 0 to the log base 2 of the number of possible symbols, with 0 representing certainty and the maximum representing total uncertainty. 3. According to Shannon's source coding theorem, the minimum possible average code word length is the entropy, so an efficient source encoder will have an average code word length approaching the entropy.

Uploaded by

Harsha
Copyright
© Attribution Non-Commercial (BY-NC)
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 7

' $

Uncertainity, Information, and Entropy


Probabilistic experiment involves the observation of the output
emitted by a discrete source during every unit of time. The source
output is modeled as a discrete random variable, S,which takes on
symbols form a fixed finite alphabet.
S = s0 , s1 , s2 , · · · , sk−1
with probabilities
P (S = sk ) = pk , k = 0, 1, · · · , K − 1
We assume that the symbols emitted by teh source during
successive signaling intervals are statistically independent. A source
having the jproperties jist described is called discrete memoryless
source, memoryless in the sense that the symbol emitted at any
time is independent of previous choices.

& %
We can define the amount of information contained in each
' $
symbols.
I(sk ) = log( p1k )
Here, generally use log2 since in digital communications we will be
talking about bits. The above expression also tells us that when
there is more uncertainity(less probability) of the symbol being
occured then it conveys more information. Some properties of
information are summarized here:
1. for certain event i.e, pk = 1 the information it conveys is zero,
I(sk ) = 0.
2. for the events 0 ≤ pk ≤ 1 the information is always I(sk ) ≥ 0.
3. If for two events pk > pi , the information content is always
I(sk ) < I(si ).
4. I(sk si ) = I(sk ) + I(si ) if sk and si are statistically independent.
The amount of information I(sk ) produced by the source during an
& %
' $
arbitrary signalling interval depends on the symbol sk emitted by
the source at that time. Indeed, I(sk ) is a discrete random variable
that takes on the values I(s0 ), I(s1 ), · · · , I(sK−1 ) with probabilites
p0 , p1 , · · · , pK−1 respectively. The mean of I(sk ) over teh source
alphabet S is given by

H(S) = E[I(sk )]
X
K1
= pk I(sk )
k=0

X
K1
1
= pk log2 ( )
pk
k=0

This important quantity is called entropy of a discrete memoryless


source with source alphabet S. It is a measure of average

& %
information content per source symbol.
' $

Some properties of Entropy


The entropy H(S) of a discrete memoryless source is bounded as
follows:
0 ≤ H(S) ≤ log2 (K)
where K is the radix of the alphabet S of the source. Furthermore,
we may make two statements:
1. H(S) = 0, if and only if the probability pk = 1 for some k, and
the remaining probabilities in teh set are all zero; this lower
bound on entropy corresponds to no uncertainity.
1
2. H(S) = log2 (K), if and only if pk = K for all k; this upper
bound on entropy corresponds to maximum uncertainity.

& %
' $

Shannon Source Coding Theorem


An important problem in communication is the efficient
representation of data generated by a discrete source. The process
by which this representation is accomplished is called source
encoding.
Our primary interest is in the development of an efficient source
encoder that satisfies two functional requirements:
1. The code words produced by the encoder are in binary form.
2. The source code is uniquely decodable, so that the original
source sequence can be reconstructed perfectly from the
encoded binary sequence.
We define the average code word length, L̄, of the source encoder
as
& %
' $

X
K−1
L̄ = p k Ik
k=0

In physical terms, the parameter L̄ represents the average number


of bits per source symbol used in the source encoding process. Let
Lm in denote the minimum possible value of L̄. We then define the
coding efficency of the source encoder as
Lm in
η= L̄

The source encoder is said to be efficient when η approaches unity.


According to the source-coding theorem, the entropy H(S)
represents a fundamental limit on the average number of bits per
source symbol necessary to represent a discrete memoryless source
in that it can be made as small as, but no smaller than, the entropy
H(S). Thus with Lm in = H(S), we may rewrite the efficiency of a
source encoder in terms of the entropy H(S) as
& %
' $

H(S)
η= L̄

& %

You might also like