Scribd
Upload
14 views18 pages

1 Introduction To Information Theory and Coding

The document provides an overview of information theory in the context of digital communication, focusing on the roles of information sources, encoders, channels, and decoders. It highlights Shannon's contributions, including the fundamental theorems regarding data compression and reliable transmission rates. The document emphasizes the theoretical limitations and potentials of encoding and decoding systems in communication systems.

Uploaded by

adamsadsm0
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
14 views18 pages

1 Introduction To Information Theory and Coding

The document provides an overview of information theory in the context of digital communication, focusing on the roles of information sources, encoders, channels, and decoders. It highlights Shannon's contributions, including the fundamental theorems regarding data compression and reliable transmission rates. The document emphasizes the theoretical limitations and potentials of encoding and decoding systems in communication systems.

Uploaded by

adamsadsm0
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 18

ECE421 Digital Communication

Introduction to Information
Theory

Lecturer: Dr. Reham Samir


References

n K. Deergha Rao “Channel Coding Techniques for


Wireless Communications” Springer, 2015.

n Behrouz A. Forouzan “Data Communication and


Networking” (5th Edition), McGraw Hill, 2015.

n Behrouz A. Forouzan “Data Communication and


Networking” (3rd Edition), McGraw Hill, 2004.

n Peyton Z. Peebles, Jr., Ph.D “Digital Communication


Systems” Prentice-Hall, 1987.
Introduction

n Information is the source of a communication system, whether it is


analog or digital.

n Information theory is concerned with the analysis of a


communication system.
Introduction

n Information Source
n The source of information is the person or machine that produces the
information to be communicated.

n Encoder
n The encoder produces a signal suitable for transmission over the
channel.
n In telephony this operation consists merely of changing sound pressure
into a proportional electrical current.

n In telegraphy we have an encoding operation which produces a sequence


of dots, dashes and spaces on the channel corresponding to the message.
Introduction

n Encoder
n In a multiplex PCM system the different speech functions must be
sampled, compressed, quantized and encoded, and finally interleaved
properly to construct the signal.

n Modulation is an example of complex operation applied to the message


to obtain the signal.
Introduction

n Channel
n The channel is the medium over which the coded message is
transmitted. It may be a pair of wires, a coaxial cable, a band of radio
frequencies, a beam of light, etc.
Introduction

n Decoder
n The decoder operates on the output of the channel and attempts to
extract the original message for delivery to the destination.

n In general, this cannot be done with complete reliability because


of the effect of "noise," which is a general term for anything
which tends to produce errors in transmission.
n Examples of noisy channels:

n A telephone line suffers from cross-talk with other lines; the


hardware in the line distorts and adds noise to the transmitted
signal.
Introduction

n Fundamental theorem of information theory states that "it is


possible to transmit information through a noisy channel at any rate
less than channel capacity with an arbitrarily small probability of
error ".

n Information theory addresses and answers the two fundamental


questions of communication theory:
1. What is the ultimate transmission rate of communication?

(answer: the channel capacity, C, is its rate limit.)

2. What is the ultimate data compression?

(answer: the entropy of the data, H, is its compression limit.)


Introduction

n All communication schemes lie in between these two limits on the


compressibility of data and the capacity of a channel.

n Information theory is concerned with the theoretical limitations and


potentials of encoding and decoding systems.

n Coding theory is concerned with the creation of practical encoding


and decoding systems.
n Information Theory (IT) includes:
Where Information Theory all begins …
Shannon’s Vision

n Sha nnon's inform a ti o n t h e o r y d e a l s w i t h l i m i t s o n d a t a


compression (source coding) and reliable data transmission
(channel coding).

n How much can data can be compressed?

n How fast can data be reliably transmitted over a noisy channel?


Shannon’s Vision

n The fundamental problem of communication is that of reproducing


at one point either exactly or approximately a message selected at
another point. (Claude Shannon, 1948)

n Shannon wants to find a way for “reliably” transmitting data


throughout the channel at “maximal” possible rate.
Shannon’s Vision
Shannon’s Vision

n Example: Disk Storage


Shannon’s Vision

n Example: Cellular Phone


Shannon’s Vision

n
Shannon’s Vision

n Two basic “point-to-point" communication theorems (Shannon


1948)
n Source coding theorem: the minimum rate at which data can be
compressed losslessly is the entropy rate of the source.

n Channel coding theorem: the maximum rate at which data can be


reliably transmitted is the channel capacity of the channel.

You might also like