Information Theoretic Approach

Information Theoretic Approach

• Information theory has its own special set of terms.
• In the figure there is a source s which generates messages.
• A message is an ordered sequence of k symbols
• s = (s1,…,sk)

Fig. The communication channel. A message s is encoded as

codewords x before being transmitted through a channel, which may
corrupt the encoded message by adding noise η to produce outputs y
= x + η. A receiver decodes the inputs x, which are then interpreted as
a message s.
 Information Theoretic Approach
• Where each symbol can be a number or a letter
corresponding to the value of a random variable.
• The entire sequence s is represented by a letter
in bold typeface, and its symbols are enclosed in
round brackets.
• Each symbol is an outcome value of a random
variable S which can adopt any value from an
alphabet of α (alpha) different symbols,
• As = { s1, …, sα }
 Information Theoretic Approach
• The probability of each symbol being
generated by the source is defined by the
probability distribution
• p(S) = {p(s1), …, p(sα)}
• where , by definition, the sum of p(s) values
must add up to one,
•
 Information Theoretic Approach
• A communication channel is used to transmit data from its input to its
output. If these data are transmitted from input to output without
error then they have been successfully communicated. Before being
transmitted, each message s is transformed by an encoder, which we
represent as a generic function g, into the channel input x = g(s), which
is a sequence codewords
• X = ( x1, …, xn),
• Where each codeword is the value of a random variable X which can
adopt any one of m different values from the code book
• Ax = {x1, …, xm}
• The probability of each codeword is defined by the probability
distribution
• p(X) = { p(x1), …, p(xm) }
 Information Theoretic Approach
• We can choose a message to transmit as it is, so that the message and
transmitted codewords are identical (i.e. x=s).
• However several symbols may be combined into a single codeword, or
the message may be compressed by removing its natural redundance.
• Consequently, the number of codewords in an encoded message x =
g(s) may not match the number of symbols in the message s.
• If the form of compression allows the message to be decompressed
perfectly then the compression allows the message to be
decompressed perfectly then the compression is lossless, but if some
information is discarded during compression then the message cannot
be recovered exactly, and the compression is lossy.
• A code is a list of symbols and their corresponding codewords. It can
be envisaged as a simple look-up table (e.g. Table 2.1).
 Information Theoretic Approach
• In order to ensure that the encoded message can withstand the
effects of a noisy communication channel, some redundancy
may be added to codewords before they are transmitted.
• Transmitting a message s encoded as the codewords x produces
the channel outputs
• y = (y1, …, yn)
• Each output is the value of a random variable Y, which can adopt
any one of m different values
• Ay = { y1, …, ym}
• If the channel is noisy then the output yi may be different from
the codeword xj that was transmitted.
Information Theoretic Approach

Table 2.1 : A code consists of a set of symbols (e.g. decimal numbers) or

messages which are encoded as codewords (e.g. binary numbers) . Here,
the eight symbols are numbers which increase in steps of three, but , in
principle, they could be any eight numbers or any eight entities.
 Information Theoretic Approach
• The probability of each output is defined by the probability distribution
• p(Y) = {p(y1),…,p(ym)}
• Each output sequence y is interpreted as implying the presence of a particular
input sequence x.
• If this interpretation is incorrect (e.g. due to channel noise) then the result is an
error.
• For a given channel, the mapping between messages and outputs, and vice versa,
is provided by a code.
• A code consists of a message, an encodr, and a decoder.
• The decoder converts each output y to a (possible incorrect) message.
• Channel noise induces errors in interpreting outputs; the error rate of a code is
the number of incorrect inputs associated with that codebook devided by the
number of possible inputs.
• Channel capacity is the maximum amount of information which can be
communicated from a channel’s input to its output.
 Information Theoretic Approach
• Capacity can be measured in terms of the amount of
information per symbol, and if a channel
communicates n symbols per second then its capacity
can be expressed in terms of information per second
(e.g. bits/s). The capacity of a channel is somewhat like
the capacity of bucket, and the rate is like the amount
of water we pour into the bucket. The amount of
water (ate) we pour (transmit) into the bucket is up to
us, and the bucket can hold (communicate, or transmit
reliably)less than its capacity, but it cannot hold more.
 Information Theoretic Approach

• Consider an alphabet of α symbols, where

• α =2 If the data is binary.
• If a noiseless channel transmits data at a fixed rate of n symbols/s then it
transmits information at a maximum rate or channel capacity of nlog α bits/s,
which equels n bits/s for binary data.
• However, the capacity of a channel is different from the rate at which information
is actually communicated through that channel. The rate is the number of bits of
information communicated per second, which depends on the code used to
transmit data. The rate of a given code may be less than the capacity of a
channel, but it cannot be greater;
• The channel capacity is the maximum rate that can be achieved when considered
over all possible codes. E.g. a code for binary data in which 0s and 1s occur
equally often ensures that each binary digit (symbol) conveys one bit of
information, but for any other code each binary digit conveys less than one bit.
 Information Theoretic Approach
• Thus, the capacity of a noiseless binary channel is
numerically equal to the rate at which it transmits
binary digits, whereas the capacity of a noisy
binary channel is less than this, shown in fig 2.4
• Some of the definitions require a different
interpretation for continuous variables, and we
may sometimes use non-bold letters to represent
messages, encoded messages and output
sequences.
Information Theoretic Approach

• Fig. 2.4 The channel capacity of noiseless and noisy channel is the maximum
rate at which information can be communicated. If a noiseless channel
communicates data at 10 binary digits/s then its capacity is C=10 bits/s. The
capacity of a noiseless channel is numerically equal to the rate at which it
communicates binary digits, whereas the capacity of a noisy channel is less
than this because it is limited by the amount of noise in the channel.
Information Theoretic Approach
Information Theoretic Approach