Scribd
Upload
19 views9 pages

IT Lecture 0

Information theory deals with the quantification, storage, and communication of information. It studies how information can be encoded efficiently for transmission through a channel. The purpose is to transmit information with minimum time and space. Sources of information produce symbols from a finite alphabet. An information source is memoryless if the probability of each symbol is independent of previous symbols, and stationary if the probability distribution does not change over time. Efficient encoding and error correction are needed to transfer information reliably across noisy channels.

Uploaded by

19PD34 - SNEHA P
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
19 views9 pages

IT Lecture 0

Information theory deals with the quantification, storage, and communication of information. It studies how information can be encoded efficiently for transmission through a channel. The purpose is to transmit information with minimum time and space. Sources of information produce symbols from a finite alphabet. An information source is memoryless if the probability of each symbol is independent of previous symbols, and stationary if the probability distribution does not change over time. Efficient encoding and error correction are needed to transfer information reliably across noisy channels.

Uploaded by

19PD34 - SNEHA P
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 9

Information Theory

- What is Information Theory?

- What is the purpose of information theory?

- Why we need to study information theory?

What is information theory (IT) ?

IT is the science that deals with the concept “ Information” :


Its measurement & its application

What is the purpose of IT?

Transmission of information in an efficient way : minimum time & space

Why we need to study information theory?

Because of the revolution of communication dealing efficiently with information and its
transmission becomes a necessary requirement for a computer engineer.
What is Information?

There are 3 types of Information

Syntactic Information Semantic information Pragmatic information

Related to the structure Related to the Related to the usage and


of the messages meaning of the effect of the messages
(characters) that forms messages
the information

Example:
i. I eat sushi by hashi
ii. By hashi I eat sushi
iii. There is a typhoon in Japan
iv. There is a typhoon in Kanto area in Japan
i. and ii. are syntactically different but semantically & Pragmatically equal
iii. and iv. are syntactically, semantically & pragmatically different
(iv. gives more information than iii.)
INFORMATION TRANSFER ACROSS CHANNELS

Sent Received
messages messages
symbols

source encoderChannel
Source
channel
channel
Channel
decoderSource receiver
coding coding decoding decoding

Compression Error Correction Decompression


Source Entropy Channel Capacity

Capacity vs Efficiency
Digital Communication Systems

Information User of
Source Information

Source Source
Encoder Decoder

Channel Channel
Encoder Decoder

Modulator De-Modulator

Channel
Information Source

Examples:

1. English text
2. A man speaking
3. Photographs
4. Motion of films,
....... etc.

A chief aim of information theory is to study how such sequence of symbols (signals)
can be most effectively encoded for transmission (by electrical means).

For Information source we have:

· Information expressed as a finite set of symbols : Source Alphabet

· A ( discrete) information source : is a source that generates a sequence of symbols

· The symbols are denoted by a1, a2, …., am and the alphabet by A = { a1, a2, …, am }

· A finite sequence of symbols is called word. The set of all words is denoted by A*
Information Source

Memoryless

Memoryless means the generated symbols (of a source message ) are


independent.

Stationary

The idea of stationary of a source demands no change with time

Discrete
The source produces independent symbols in different unit times
Memoryless

Memoryless means the generated symbols (of a source message ) are independent.

X
source

P(X = i | X1, X2, …, Xn ) = P ( X = i )

i.e. The probability of the output X is conditionally independent of previous channel


inputs or outputs X1, .., Xn

Example:

Coin toss 8 times : the probability to get head


Each time is ½ x …x ½ = (½)8 = 1/256 = 0.0039 = 0.4 %
Coin toss number 9 : The probability to get head is still ½

So it is independent of the previous 8 toss


Stationary

A process is called Stochastic if its output is associated with a probability distribution.

A stochastic process is said to be stationary when the process is (temporally


homogeneous ) remain invariant under every translation of the time scale

Example 1 : Assume a source produces an infinite sequence of the form:

Source AE AE AE AE AE etc.

i.e. What comes later is like what has gone before. Stationary is a designation of such
source of characters

Example 2 : Assume the source that produces:

Source AE AAEE AAAEEE etc.

such source is not stationary

The idea of stationary of a source demands no change with time

i.e P(X = i ) = C (constant)


Digital Communication Systems

Source of User of
Information Information

Source Source
Encoder Decoder

Channel Channel
Encoder Decoder

Modulator De-Modulator

Channel

You might also like