Information Theory and Coding

Computer Science Tripos Part II, Michaelmas Term 11 Lectures by J G Daugman

1. Foundations: Probability, Uncertainty, and Information 2. Entropies Dened, and Why they are Measures of Information 3. Source Coding Theorem; Prex, Variable-, & Fixed-Length Codes 4. Channel Types, Properties, Noise, and Channel Capacity 5. Continuous Information; Density; Noisy Channel Coding Theorem 6. Fourier Series, Convergence, Orthogonal Representation 7. Useful Fourier Theorems; Transform Pairs; Sampling; Aliasing 8. Discrete Fourier Transform. Fast Fourier Transform Algorithms 9. The Quantized Degrees-of-Freedom in a Continuous Signal 10. Gabor-Heisenberg-Weyl Uncertainty Relation. Optimal Logons 11. Kolmogorov Complexity and Minimal Description Length

Information Theory and Coding

J G Daugman
Prerequisite courses: Probability; Mathematical Methods for CS; Discrete Mathematics Aims The aims of this course are to introduce the principles and applications of information theory. The course will study how information is measured in terms of probability and entropy, and the relationships among conditional and joint entropies; how these are used to calculate the capacity of a communication channel, with and without noise; coding schemes, including error correcting codes; how discrete channels and measures of information generalise to their continuous forms; the Fourier perspective; and extensions to wavelets, complexity, compression, and ecient coding of audio-visual information. Lectures Foundations: probability, uncertainty, information. How concepts of randomness, redundancy, compressibility, noise, bandwidth, and uncertainty are related to information. Ensembles, random variables, marginal and conditional probabilities. How the metrics of information are grounded in the rules of probability. Entropies dened, and why they are measures of information. Marginal entropy, joint entropy, conditional entropy, and the Chain Rule for entropy. Mutual information between ensembles of random variables. Why entropy is the fundamental measure of information content. Source coding theorem; prex, variable-, and xed-length codes. Symbol codes. The binary symmetric channel. Capacity of a noiseless discrete channel. Error correcting codes. Channel types, properties, noise, and channel capacity. Perfect communication through a noisy channel. Capacity of a discrete channel as the maximum of its mutual information over all possible input distributions. Continuous information; density; noisy channel coding theorem. Extensions of the discrete entropies and measures to the continuous case. Signal-to-noise ratio; power spectral density. Gaussian channels. Relative signicance of bandwidth and noise limitations. The Shannon rate limit and eciency for noisy continuous channels. Fourier series, convergence, orthogonal representation. Generalised signal expansions in vector spaces. Independence. Representation of continuous or discrete data by complex exponentials. The Fourier basis. Fourier series for periodic functions. Examples. Useful Fourier theorems; transform pairs. Sampling; aliasing. The Fourier transform for non-periodic functions. Properties of the transform, and examples. Nyquists Sampling Theorem derived, and the cause (and removal) of aliasing. Discrete Fourier transform. Fast Fourier Transform Algorithms. Ecient algorithms for computing Fourier transforms of discrete data. Computational complexity. Filters, correlation, modulation, demodulation, coherence.

 The quantised degrees-of-freedom in a continuous signal. Why a continuous signal of nite bandwidth and duration has a xed number of degrees-of-freedom. Diverse illustrations of the principle that information, even in such a signal, comes in quantised, countable, packets. Gabor-Heisenberg-Weyl uncertainty relation. Optimal Logons. Unication of the time-domain and the frequency-domain as endpoints of a continuous deformation. The Uncertainty Principle and its optimal solution by Gabors expansion basis of logons. Multi-resolution wavelet codes. Extension to images, for analysis and compression. Kolmogorov complexity. Minimal description length. Denition of the algorithmic complexity of a data sequence, and its relation to the entropy of the distribution from which the data was drawn. Fractals. Minimal description length, and why this measure of complexity is not computable. Objectives At the end of the course students should be able to calculate the information content of a random variable from its probability distribution relate the joint, conditional, and marginal entropies of variables in terms of their coupled probabilities dene channel capacities and properties using Shannons Theorems construct ecient codes for data on imperfect communication channels generalise the discrete concepts to continuous signals on continuous channels understand Fourier Transforms and the main ideas of ecient algorithms for them describe the information resolution and compression properties of wavelets

Recommended book * Cover, T.M. & Thomas, J.A. (1991). Elements of information theory. New York: Wiley.

Information Theory and Coding

Computer Science Tripos Part II, Michaelmas Term 11 lectures by J G Daugman

1. Overview: What is Information Theory? Key idea: The movements and transformations of information, just like those of a uid, are constrained by mathematical and physical laws. These laws have deep connections with: probability theory, statistics, and combinatorics thermodynamics (statistical physics) spectral analysis, Fourier (and other) transforms sampling theory, prediction, estimation theory electrical engineering (bandwidth; signal-to-noise ratio) complexity theory (minimal description length) signal processing, representation, compressibility As such, information theory addresses and answers the two fundamental questions of communication theory: 1. What is the ultimate data compression? (answer: the entropy of the data, H, is its compression limit.) 2. What is the ultimate transmission rate of communication? (answer: the channel capacity, C, is its rate limit.) All communication schemes lie in between these two limits on the compressibility of data and the capacity of a channel. Information theory can suggest means to achieve these theoretical limits. But the subject also extends far beyond communication theory.
1

Important questions... to which Information Theory oers answers: How should information be measured? How much additional information is gained by some reduction in uncertainty? How do the a priori probabilities of possible messages determine the informativeness of receiving them? What is the information content of a random variable? How does the noise level in a communication channel limit its capacity to transmit information? How does the bandwidth (in cycles/second) of a communication channel limit its capacity to transmit information? By what formalism should prior knowledge be combined with incoming data to draw formally justiable inferences from both? How much information in contained in a strand of DNA? How much information is there in the ring pattern of a neurone? Historical origins and important contributions: Ludwig BOLTZMANN (1844-1906), physicist, showed in 1877 that thermodynamic entropy (dened as the energy of a statistical ensemble [such as a gas] divided by its temperature: ergs/degree) is related to the statistical distribution of molecular congurations, with increasing entropy corresponding to increasing randomness. He made this relationship precise with his famous formula S = k log W where S denes entropy, W is the total number of possible molecular congurations, and k is the constant which bears Boltzmanns name: k =1.38 x 1016 ergs per degree centigrade. (The above formula appears as an epitaph on Boltzmanns tombstone.) This is
2

equivalent to the denition of the information (negentropy) in an ensemble, all of whose possible states are equiprobable, but with a minus sign in front (and when the logarithm is base 2, k=1.) The deep connections between Information Theory and that branch of physics concerned with thermodynamics and statistical mechanics, hinge upon Boltzmanns work. Leo SZILARD (1898-1964) in 1929 identied entropy with information. He formulated key information-theoretic concepts to solve the thermodynamic paradox known as Maxwells demon (a thoughtexperiment about gas molecules in a partitioned box) by showing that the amount of information required by the demon about the positions and velocities of the molecules was equal (negatively) to the demons entropy increment. James Clerk MAXWELL (1831-1879) originated the paradox called Maxwells Demon which greatly inuenced Boltzmann and which led to the watershed insight for information theory contributed by Szilard. At Cambridge, Maxwell founded the Cavendish Laboratory which became the original Department of Physics. R V HARTLEY in 1928 founded communication theory with his paper Transmission of Information. He proposed that a signal (or a communication channel) having bandwidth over a duration T has a limited number of degrees-of-freedom, namely 2T , and therefore it can communicate at most this quantity of information. He also dened the information content of an equiprobable ensemble of N possible states as equal to log2 N . Norbert WIENER (1894-1964) unied information theory and Fourier analysis by deriving a series of relationships between the two. He invented white noise analysis of non-linear systems, and made the denitive contribution to modeling and describing the information content of stochastic processes known as Time Series.

 Dennis GABOR (1900-1979) crystallized Hartleys insight by formulating a general Uncertainty Principle for information, expressing the trade-o for resolution between bandwidth and time. (Signals that are well specied in frequency content must be poorly localized in time, and those that are well localized in time must be poorly specied in frequency content.) He formalized the Information Diagram to describe this fundamental trade-o, and derived the continuous family of functions which optimize (minimize) the conjoint uncertainty relation. In 1974 Gabor won the Nobel Prize in Physics for his work in Fourier optics, including the invention of holography. Claude SHANNON (together with Warren WEAVER) in 1949 wrote the denitive, classic, work in information theory: Mathematical Theory of Communication. Divided into separate treatments for continuous-time and discrete-time signals, systems, and channels, this book laid out all of the key concepts and relationships that dene the eld today. In particular, he proved the famous Source Coding Theorem and the Noisy Channel Coding Theorem, plus many other related results about channel capacity. S KULLBACK and R A LEIBLER (1951) dened relative entropy (also called information for discrimination, or K-L Distance.) E T JAYNES (since 1957) developed maximum entropy methods for inference, hypothesis-testing, and decision-making, based on the physics of statistical mechanics. Others have inquired whether these principles impose fundamental physical limits to computation itself. A N KOLMOGOROV in 1965 proposed that the complexity of a string of data can be dened by the length of the shortest binary program for computing the string. Thus the complexity of data is its minimal description length, and this species the ultimate compressibility of data. The Kolmogorov complexity K of a string is approximately equal to its Shannon entropy H, thereby unifying the theory of descriptive complexity and information theory.
4

2. Mathematical Foundations; Probability Rules; Bayes Theorem What are random variables? What is probability? Random variables are variables that take on values determined by probability distributions. They may be discrete or continuous, in either their domain or their range. For example, a stream of ASCII encoded text characters in a transmitted message is a discrete random variable, with a known probability distribution for any given natural language. An analog speech signal represented by a voltage or sound pressure waveform as a function of time (perhaps with added noise), is a continuous random variable having a continuous probability density function. Most of Information Theory involves probability distributions of random variables, and conjoint or conditional probabilities dened over ensembles of random variables. Indeed, the information content of a symbol or event is dened by its (im)probability. Classically, there are two dierent points of view about what probability actually means: relative frequency: sample the random variable a great many times and tally up the fraction of times that each of its dierent possible values occurs, to arrive at the probability of each. degree-of-belief: probability is the plausibility of a proposition or the likelihood that a particular state (or value of a random variable) might occur, even if its outcome can only be decided once (e.g. the outcome of a particular horse-race). The rst view, the frequentist or operationalist view, is the one that predominates in statistics and in information theory. However, by no means does it capture the full meaning of probability. For example, the proposition that "The moon is made of green cheese" is one which surely has a probability that we should be able to attach to it. We could assess its probability by degree-of-belief calculations which
5

combine our prior knowledge about physics, geology, and dairy products. Yet the frequentist denition of probability could only assign a probability to this proposition by performing (say) a large number of repeated trips to the moon, and tallying up the fraction of trips on which the moon turned out to be a dairy product.... In either case, it seems sensible that the less probable an event is, the more information is gained by noting its occurrence. (Surely discovering that the moon IS made of green cheese would be more informative than merely learning that it is made only of earth-like rocks.) Probability Rules Most of probability theory was laid down by theologians: Blaise PASCAL (1623-1662) who gave it the axiomatization that we accept today; and Thomas BAYES (1702-1761) who expressed one of its most important and widely-applied propositions relating conditional probabilities. Probability Theory rests upon two rules: Product Rule: p(A, B) = joint probability of both A and B = p(A|B)p(B) or equivalently, = p(B|A)p(A) Clearly, in case A and B are independent events, they are not conditionalized on each other and so p(A|B) = p(A) and p(B|A) = p(B), in which case their joint probability is simply p(A, B) = p(A)p(B).
6

Sum Rule: If event A is conditionalized on a number of other events B, then the total probability of A is the sum of its joint probabilities with all B: p(A) =
B

p(A, B) =
B

p(A|B)p(B)

From the Product Rule and the symmetry that p(A, B) = p(B, A), it is clear that p(A|B)p(B) = p(B|A)p(A). Bayes Theorem then follows: Bayes Theorem: p(B|A) = p(A|B)p(B) p(A)

The importance of Bayes Rule is that it allows us to reverse the conditionalizing of events, and to compute p(B|A) from knowledge of p(A|B), p(A), and p(B). Often these are expressed as prior and posterior probabilities, or as the conditionalizing of hypotheses upon data. Worked Example: Suppose that a dread disease aects 1/1000th of all people. If you actually have the disease, a test for it is positive 95% of the time, and negative 5% of the time. If you dont have the disease, the test is positive 5% of the time. We wish to know how to interpret test results. Suppose you test positive for the disease. What is the likelihood that you actually have it? We use the above rules, with the following substitutions of data D and hypothesis H instead of A and B: D = data: the test is positive H = hypothesis: you have the disease H = the other hypothesis: you do not have the disease
7

Before acquiring the data, we know only that the a priori probability of having the disease is .001, which sets p(H). This is called a prior. We also need to know p(D). From the Sum Rule, we can calculate that the a priori probability p(D) of testing positive, whatever the truth may actually be, is: p(D) = p(D|H)p(H) + p(D|H)p(H) =(.95)(.001)+(.05)(.999) = .051 and from Bayes Rule, we can conclude that the probability that you actually have the disease given that you tested positive for it, is much smaller than you may have thought: p(H|D) = p(D|H)p(H) (.95)(.001) = = 0.019 p(D) (.051) (less than 2%).

This quantity is called the posterior probability because it is computed after the observation of data; it tells us how likely the hypothesis is, given what we have observed. (Note: it is an extremely common human fallacy to confound p(H|D) with p(D|H). In the example given, most people would react to the positive test result by concluding that the likelihood that they have the disease is .95, since that is the hit rate of the test. They confound p(D|H) = .95 with p(H|D) = .019, which is what actually matters.) A nice feature of Bayes Theorem is that it provides a simple mechanism for repeatedly updating our assessment of the hypothesis as more data continues to arrive. We can apply the rule recursively, using the latest posterior as the new prior for interpreting the next set of data. In Articial Intelligence, this feature is important because it allows the systematic and real-time construction of interpretations that can be updated continuously as more data arrive in a time series, such as a ow of images or spoken sounds that we wish to understand.
8

3. Entropies Dened, and Why They are Measures of Information The information content I of a single event or message is dened as the base-2 logarithm of its probability p: I = log2 p (1)

and its entropy H is considered the negative of this. Entropy can be regarded intuitively as uncertainty, or disorder. To gain information is to lose uncertainty by the same amount, so I and H dier only in sign (if at all): H = I. Entropy and information have units of bits. Note that I as dened in Eqt (1) is never positive: it ranges between 0 and as p varies from 1 to 0. However, sometimes the sign is dropped, and I is considered the same thing as H (as well do later too). No information is gained (no uncertainty is lost) by the appearance of an event or the receipt of a message that was completely certain anyway (p = 1, so I = 0). Intuitively, the more improbable an event is, the more informative it is; and so the monotonic behaviour of Eqt (1) seems appropriate. But why the logarithm? The logarithmic measure is justied by the desire for information to be additive. We want the algebra of our measures to reect the Rules of Probability. When independent packets of information arrive, we would like to say that the total information received is the sum of the individual pieces. But the probabilities of independent events multiply to give their combined probabilities, and so we must take logarithms in order for the joint probability of independent events or messages to contribute additively to the information gained. This principle can also be understood in terms of the combinatorics of state spaces. Suppose we have two independent problems, one with n
9

possible solutions (or states) each having probability p n, and the other with m possible solutions (or states) each having probability p m. Then the number of combined states is mn, and each of these has probability pmpn. We would like to say that the information gained by specifying the solution to both problems is the sum of that gained from each one. This desired property is achieved: Imn = log2(pmpn) = log2 pm + log2 pn = Im + In A Note on Logarithms: In information theory we often wish to compute the base-2 logarithms of quantities, but most calculators (and tools like xcalc) only oer Napierian (base 2.718...) and decimal (base 10) logarithms. So the following conversions are useful: log2 X = 1.443 loge X = 3.322 log10 X Henceforward we will omit the subscript; base-2 is always presumed. Intuitive Example of the Information Measure (Eqt 1): Suppose I choose at random one of the 26 letters of the alphabet, and we play the game of 25 questions in which you must determine which letter I have chosen. I will only answer yes or no. What is the minimum number of such questions that you must ask in order to guarantee nding the answer? (What form should such questions take? e.g., Is it A? Is it B? ...or is there some more intelligent way to solve this problem?) The answer to a Yes/No question having equal probabilities conveys one bit worth of information. In the above example with equiprobable states, you never need to ask more than 5 (well-phrased!) questions to discover the answer, even though there are 26 possibilities. Appropriately, Eqt (1) tells us that the uncertainty removed as a result of solving this problem is about -4.7 bits.
10

(2)

Entropy of Ensembles We now move from considering the information content of a single event or message, to that of an ensemble. An ensemble is the set of outcomes of one or more random variables. The outcomes have probabilities attached to them. In general these probabilities are non-uniform, with event i having probability pi, but they must sum to 1 because all possible outcomes are included; hence they form a probability distribution:
i

pi = 1

(3)

The entropy of an ensemble is simply the average entropy of all the elements in it. We can compute their average entropy by weighting each of the log pi contributions by its probability pi: H = I =
i

pi log pi

(4)

Eqt (4) allows us to speak of the information content or the entropy of a random variable, from knowledge of the probability distribution that it obeys. (Entropy does not depend upon the actual values taken by the random variable! Only upon their relative probabilities.) Let us consider a random variable that takes on only two values, one with probability p and the other with probability (1 p). Entropy is a concave function of this distribution, and equals 0 if p = 0 or p = 1:

11

Example of entropy as average uncertainty: The various letters of the written English language have the following relative frequencies (probabilities), in descending order:
E T O A N I R S H D L C ... .105 .072 .066 .063 .059 .055 .054 .052 .047 .035 .029 .023 ...

If they had been equiprobable, the entropy of the ensemble would 1 have been log2( 26 ) = 4.7 bits. But their non-uniform probabilities imply that, for example, an E is nearly ve times more likely than a C; surely this prior knowledge is a reduction in the uncertainty of this random variable. In fact, the distribution of English letters has an entropy of only 4.0 bits. This means that as few as only four Yes/No questions are needed, in principle, to identify one of the 26 letters of the alphabet; not ve. How can this be true? That is the subject matter of Shannons SOURCE CODING THEOREM (so named because it uses the statistics of the source, the a priori probabilities of the message generator, to construct an optimal code.) Note the important assumption: that the source statistics are known! Several further measures of entropy need to be dened, involving the marginal, joint, and conditional probabilities of random variables. Some key relationships will then emerge, that we can apply to the analysis of communication channels. Notation: We use capital letters X and Y to name random variables, and lower case letters x and y to refer to their respective outcomes. These are drawn from particular sets A and B: x {a1, a2, ...aJ }, and y {b1 , b2, ...bK }. The probability of a particular outcome p(x = ai) is denoted pi, with 0 pi 1 and i pi = 1.
12

An ensemble is just a random variable X, whose entropy was dened in Eqt (4). A joint ensemble XY is an ensemble whose outcomes are ordered pairs x, y with x {a1, a2, ...aJ } and y {b1 , b2, ...bK }. The joint ensemble XY denes a probability distribution p(x, y) over all possible joint outcomes x, y. Marginal probability: From the Sum Rule, we can see that the probability of X taking on a particular value x = ai is the sum of the joint probabilities of this outcome for X and all possible outcomes for Y : p(x = ai) =
y

p(x = ai, y)
y

We can simplify this notation to: p(x) = and similarly: p(y) =

x

p(x, y)

p(x, y)

Conditional probability: From the Product Rule, we can easily see that the conditional probability that x = ai, given that y = bj , is: p(x = ai|y = bj ) = p(x = ai, y = bj ) p(y = bj ) p(x, y) p(y)

We can simplify this notation to: p(x|y) = and similarly: p(y|x) = p(x, y) p(x)

It is now possible to dene various entropy measures for joint ensembles: Joint entropy of XY H(X, Y ) =
x,y

p(x, y) log

1 p(x, y)

(5)

(Note that in comparison with Eqt (4), we have replaced the sign in front by taking the reciprocal of p inside the logarithm).
13

From this denition, it follows that joint entropy is additive if X and Y are independent random variables: H(X, Y ) = H(X) + H(Y ) i p(x, y) = p(x)p(y) Prove this.

Conditional entropy of an ensemble X, given that y = bj measures the uncertainty remaining about random variable X after specifying that random variable Y has taken on a particular value y = bj . It is dened naturally as the entropy of the probability distribution p(x|y = bj ): H(X|y = bj ) =
x

p(x|y = bj ) log

1 p(x|y = bj )

(6)

If we now consider the above quantity averaged over all possible outcomes that Y might have, each weighted by its probability p(y), then we arrive at the... Conditional entropy of an ensemble X, given an ensemble Y : 1 (7) H(X|Y ) = p(y) p(x|y) log y x p(x|y) and we know from the Sum Rule that if we move the p(y) term from the outer summation over y, to inside the inner summation over x, the two probability terms combine and become just p(x, y) summed over all x, y. Hence a simpler expression for this conditional entropy is: 1 H(X|Y ) = p(x, y) log (8) x,y p(x|y) This measures the average uncertainty that remains about X, when Y is known.
14

Chain Rule for Entropy The joint entropy, conditional entropy, and marginal entropy for two ensembles X and Y are related by: H(X, Y ) = H(X) + H(Y |X) = H(Y ) + H(X|Y ) (9)

It should seem natural and intuitive that the joint entropy of a pair of random variables is the entropy of one plus the conditional entropy of the other (the uncertainty that it adds once its dependence on the rst one has been discounted by conditionalizing on it). You can derive the Chain Rule from the earlier denitions of these three entropies. Corollary to the Chain Rule: If we have three random variables X, Y, Z, the conditionalizing of the joint distribution of any two of them, upon the third, is also expressed by a Chain Rule: H(X, Y |Z) = H(X|Z) + H(Y |X, Z) (10)

Independence Bound on Entropy A consequence of the Chain Rule for Entropy is that if we have many dierent random variables X1, X2, ..., Xn , then the sum of all their individual entropies is an upper bound on their joint entropy: H(X1, X2 , ..., Xn)
n i=1

H(Xi)

(11)

Their joint entropy only reaches this upper bound if all of the random variables are independent.

15

Mutual Information between X and Y The mutual information between two random variables measures the amount of information that one conveys about the other. Equivalently, it measures the average reduction in uncertainty about X that results from learning about Y . It is dened: p(x, y) I(X; Y ) = p(x, y) log (12) x,y p(x)p(y) Clearly X says as much about Y as Y says about X. Note that in case X and Y are independent random variables, then the numerator inside the logarithm equals the denominator. Then the log term vanishes, and the mutual information equals zero, as one should expect. Non-negativity: mutual information is always 0. In the event that the two random variables are perfectly correlated, then their mutual information is the entropy of either one alone. (Another way to say this is: I(X; X) = H(X): the mutual information of a random variable with itself is just its entropy. For this reason, the entropy H(X) of a random variable X is sometimes referred to as its self-information.) These properties are reected in three equivalent denitions for the mutual information between X and Y : I(X; Y ) = H(X) H(X|Y ) I(X; Y ) = H(Y ) H(Y |X) = I(Y ; X) I(X; Y ) = H(X) + H(Y ) H(X, Y ) (13) (14) (15)

In a sense the mutual information I(X; Y ) is the intersection between H(X) and H(Y ), since it represents their statistical dependence. In the Venn diagram given at the top of page 18, the portion of H(X) that does not lie within I(X; Y ) is just H(X|Y ). The portion of H(Y ) that does not lie within I(X; Y ) is just H(Y |X).
16

Distance D(X, Y ) between X and Y The amount by which the joint entropy of two random variables exceeds their mutual information is a measure of the distance between them: D(X, Y ) = H(X, Y ) I(X; Y ) (16) Note that this quantity satises the standard axioms for a distance: D(X, Y ) 0, D(X, X) = 0, D(X, Y ) = D(Y, X), and D(X, Z) D(X, Y ) + D(Y, Z).

Relative entropy, or Kullback-Leibler distance Another important measure of the distance between two random variables, although it does not satisfy the above axioms for a distance metric, is the relative entropy or Kullback-Leibler distance. It is also called the information for discrimination. If p(x) and q(x) are two probability distributions dened over the same set of outcomes x, then their relative entropy is: p(x) DKL(p q) = p(x) log (17) x q(x) Note that DKL(p q) 0, and in case p(x) = q(x) then their distance DKL(p q) = 0, as one might hope. However, this metric is not strictly a distance, since in general it lacks symmetry: DKL(p q) = DKL(q p). The relative entropy DKL(p q) is a measure of the ineciency of assuming that a distribution is q(x) when in fact it is p(x). If we have an optimal code for the distribution p(x) (meaning that we use on average H(p(x)) bits, its entropy, to describe it), then the number of additional bits that we would need to use if we instead described p(x) using an optimal code for q(x), would be their relative entropy D KL(p q).
17

Venn Diagram: Relationship among entropies and mutual information.

Fanos Inequality We know that conditioning reduces entropy: H(X|Y ) H(X). It is clear that if X and Y are perfectly correlated, then their conditional entropy is 0. It should also be clear that if X is any deterministic function of Y , then again, there remains no uncertainty about X once Y is known and so their conditional entropy H(X|Y ) = 0. Fanos Inequality relates the probability of error Pe in guessing X from knowledge of Y to their conditional entropy H(X|Y ), when the number of possible outcomes is |A| (e.g. the length of a symbol alphabet): Pe H(X|Y ) 1 log |A| (18)

The lower bound on Pe is a linearly increasing function of H(X|Y ). The Data Processing Inequality If random variables X, Y , and Z form a Markov chain (i.e. the conditional distribution of Z depends only on Y and is independent of X), which is normally denoted as X Y Z, then the mutual information must be monotonically decreasing over steps along the chain: I(X; Y ) I(X; Z) (19)

We turn now to applying these measures and relationships to the study of communications channels. (The following material is from McAuley.)
18

 bo

H PB B ` q a s WB H s D R g q H 8 I P WB ` R T W 8 a 8 `B H W R T D s I H 8 bSEE)f)mY$FH)idlY1Ym)XVYbb)ESYXVFDw)97 8 T a e R H 8 g R H q ` 8 P s a 8 W 8 HD R g q H 8 I P P W 8 H 8 8 a qD P W B T 8 R P I HB @ 8 @ W 8 P Vf)YSciXdVSwSbYblEiS1Y~bSVza fESbF8V12SUECcGbS3  (  ( 4  ( ' %  ( 4 ggw!&)&1 x

8 T a e R H H H 8D q a R g 8 g 8 I P R R a P W 8 8 I P 8 x s I 8 @ r 8 P s P H 8D WB H s a R qD a s 8D Vf)XdSVEYfXibplYwcq fbY)lGXd$SiEYmS~CX31EfbEGh Xx )ES)ED))ES)E R i!

| H W RB P s e w VYXES$))V8 @ 8 I P g R a ` 8 P sD e TD s T q T W e T T R 8 P s P H 8 I P I PB @ ` 8 P I B 8 @ H 8 eD s x $R v YXf!QVS$mYbE$bxVYs bVXSwS2YECQVSYFVGVYm$$q fb8 RaP W 8 P s P H D s e `B xB ` WB 8 H 8 I P R g e H 8 I P 8 R P g 8 P H q H 8 I P R S$S~$)gEXE)Ym2dVYw)2Y2)tSbSSYSY$tq fb2YtbYQYs RaP W8 8 IP W8 IP ` W D )ES) FR)Ez A5i

| a 8 W W s g D s g a R W 8 I P WB 8 P s P H I T s 8 VbY)wcSXYiYmS$S1V$b$R q fbYY)pbCG&SzuVmiSiXS$1XESSEibCYiECx  S$SSP R a P W 8 8 I P 8 W n 8 ` W s T 8 @ P H aB { D R g q H 8 g R H R W RB H HB g 8 8 I P I PB @ 8 P s P H R S$SYfrYmX$i$cfEEEY)Xfa YYmV)ES)GVF8dFPEEY)Xza XESESYSiEC@ 8 P s P H g R a WB x R g R q PBDB s R 8 I P WB 8 H B BDB s R W RB PB H W s a P I PB r H 8P sPH I B @ HH 8 T R 8P sPH 8 B W n q W s 8 W n8 ` q s g 8 @ W 8 IP D sa 8 W 8 W Xk &3VVVVSb~yd VS$SFPdVVXfa S$SdFPYtYY)$c&GrbY$$SbYbr! HH8 SVVT R x R v a s u 8 P s P H R @ P ` W s 8 T a e R H H H 8D q a R g 8 g WB P W 8 H 8 8 a H s a } | p o 8 a e B $fa ywS9S$dGf2YVf)XdSVEYfXibpYmSbSVfa fYI S~&)b1f)XE{
B, 1/4 E, 1/4 C, 1/8 A, 1/2 A, 1/4 C, 1/8 B, 3/8 B, 1/4 D, 1/8 C, 1/8

D sB B a P HB 8 g s y 8 B I P H W RB P s e w 8 @ Vm$mYFxzCElED )VFHYxYXES$))V$R v WB H e q T W e T T R 8 P s P H 8 I P a R 8 xD R H W 8 I P W s T 8 7 p o 8 a e n WB k j r i r h r f e YudY2VYs bVXd$SYtXsrES1bYqblU)bcf)Yg1mlGc)ggr bd PV))I $YCX3dVVXfa $$d9d$S1G1UECbYVd9EFDwbFTYI SXA$YH 8 s D s 8 I P a R H H 8 T R x R v a s u 8 P s P H R @ P s I PB @ a 8 I P 8 R P q D s B s a W @ R I FH1b)VhV$bQXVS$dbYb2E)SU3dVVXfa bV1)X31)bVSSVVYfa B 8 ` 8 I T s 8 W R ` 8 P s a 8 W 8 HB D R g q H s H H 8 T R P W 8 x 8 a e R a e H 8 H H 8 T R x R v a s u a 8 `B H W R T R HD s 8 @ R H r q a R g 8 g I PB @ H 8 T a e R H a 8 `B H W R T R P I HB @ DB @ 8 ywS91b)ESYXVtSE$2G9SGfXpbihEC1VVf)Xdcb)ESYXVUSQECGFDECA97 6   (  4 0 # ( ' % #      $5321)&$"!

A, 1/2 D, 1/8

  oo o oo ) o ) 8 ` R )YGh oo o o bo bo 8 ` R )YGh h f e bo o )  o o 8 ` R )YGh bo bo $bo bo x i

|H8 `RT IP W8 bV)XVqYbED 8D sB a s x 8D B H H 8 g R H ` W s P 8 s D s D R g q H a e R 8 gB H a e R WB s s a 8 B H W R T 8 EYmfyiEYudSR iXdYV))I w)idG)X3ED mS)X1m$X$b)F`dYXV97 H s a 8D 8 P R H q s ` 8 I P g R a WB P s ` 8 ` R T 8 H a R u HB WB ` R T W 8 W s I T e H R 8 g s y bYI SXVEVS$GYgY5XziYmSwg)XVSfXCE1Ym)XVYbVY$ED )V8 W s H 8 ` R T I P W 8D 8D sB a s x R 8 H e q WB ` R T W 8 R g a R ` 8 H H 8 g R T 8 a R g 8 g R rV)XVUYbEcEYmf$C$iSY99Ym)XVYbG$SX3VSdVfa XVifXpppXSH 8 x B I T s R P R I D e R @ 8 @ ` W s r HD R g q H 8D s R B e w 8 8 x s I P R W R ` 8 @ D s a 8 W 8 W VF8YV$S8 YF`YYUYbE)i)SEY)Xfa Y)Vb)iY)wSbYbU! )$h$gm9g$gl m"m

r 8 a s o fmD H P 8 H s P s I P 8 P R D R g q H a H B R a 8 g e W 8 s a 8 x s 8 I P @ R W HB 8 a 8 I SVhi$l$)S3iS~b8 dFPYC$~b)i)Yp$SbY$YpE1fbY@ o E~ RD | WB s P R 8 @ W 8 I P r H PB B m$YXG1bYbSEE)` q a s WB R v T RD s H s I P W 8D R 8 T W 8 e w 8 H D B H H I T s 8 WB ` R T W 8 a 8 `B H W R T ` W s f)mY$iVXEYl&w9cYbE$VYbYVdF8YmSdR V$biYm)XVYbb)ESYXVY9 I P W 8D R HD R g q H R H 8 T W 8 e w 8 H a 8 `B H W R T 8 @ B I P 8 g R T a 8 x R R P q a P R P P W 8B T 8 WB H YbE$GE)i)SwVVYbY)VSb)ESYXVCGFHYpXVfb$ldlf~S)bbEVYm&EB WB ` R T HB I P r R @ P R a 8 @ s P R W B W 8 I @ P e r HD R g q H 8D s R B e w 8 I PB @ DDB Yu)YVGEYgGf$bG$R GYCFHbYY))bEiSEY)Yfa YVECEESP WB ` R T 8 ` qD 8 e wB W e gYm)YVV)2EVYEY)&$R 8D s T WB 8 a s 8 @ P s I P H W s 8 g HB I P D R g q H 8 W R W s I P 8 a R g R P WB ` R T W 8 8 W R P H s 8 EYs bYmCfw)&YbptEYiY)i)SCYX)GfYiSiYm)YVYbYXS$bED P$cS$bYEFDwSYilGS$bFHY2mh5iVEYwS1fG&C$)SCGi s 8 P s T R D s P H e g 8 @ 8 H s T B I P WB 8 x 8B I T s R P q a P 8 @ B P s I P 8 P R ` Ws o Y1 gGC$rbGyR iFHY&EY)Yfa YVczE)SY9bY7 r R @ P R a 8 @ s B ` W s r 8D s R B e w 8 8 a s HD R g q H 8 I P W 8 I | q W 8 xB AbEFHB WB ` R T 8 I P R q T W 8B T 8 8 I W 8 I P @ R W v 8 @ H s ` W s rD R g q Yu)YVGYwcVYbEVCYCihbY3 3 h$YYCwpYX3)i)SH a H PB W 8 I P HB 8 I W s I P H H 8D a 8 8 P WB P H 8 a sD 8 I P HB 8 a 8 I b8 SEYbYE3bYCi)SVEbVSmSVfmiYE x fbY@ 8 HB @ a 8 I P SECfbYR R a8 @ s &$GbG$R 5 o S3 3 3 3

| ` 8 8 W 8 @ HD R g q H 8 H 8 I P 8 ` R T 8 ` W s T 8 @ 8 a e H W 8 R H PB B ` q a s WB VVYlE)i)SSVYi)YVV)1bGlf)YbcbSEE)f)mY v T RD I P W 8D ` 8 y n s H s r R a P W 8 r k D R g q H 8 I P WB ` R T W 8 a 8 B H W R VXEYG3YbE2VYgiwgl1q zbX! yd FH)idlYYu)YVYbb)F`dYXGh )"w$ggh)C a 8 ` R T W 8 ` W s 8 T a e R H H H 8D q a R g 8 g 8 P 8 a T HB i | o 8 a e B b)XVYbAYlVf)XdCSVEXzXibiiSVfbSE)b1f)XE{

Source encoder

x"m

Symbols

Encoding

o )
H I o P  Q o F | HB W RB PB ` W R T y n 8 8 I P q B P s H R P F @ C C bEAXESE)YXVgfa Y1szFHd$iSGE D&C B AYbElFP)fXGC)YVz)mYYmX))YVX3iYFRdFP)YYVbEVH @ @ I P W 8 D I B @ H ` a R @ 8 ` R T q a s W B W B x s I 8 ` R T s a R W B B ` W R T P W 8 B T e P R W q a s H H 8 T 8 W r HB P s I q PBD s e w 8 WB 8 8 6 4 8 I P q HB P s H P H e g B r P B y 8 R Y3fSVVVYis bEG$)CGEE$))VYmlE"!9V7fl5Yc3fES$SYiCFPYbdFHYViSP 8 ` R T y n 8 s a R P s I P ` W n 8 @ q P a R y n 8 8 I P ` 8 W RB P W 8 g q ` s 8 aD s 8 x s I 8 )YVgfa CX3G$)Yrffb8 fa gfa YAVYXESbp2)$bfm$b)i97

3 "$l2

m"m

8 T a e R H 8 I P R HD R g q H 8 I P P W 8 H 8 8 a R P ` 8 aB e w 8 a 8 s a 8 x s W Vf)YSY$EiSYCbdVfa fSAVfmYVzwSbXR HSEY$b)YCY~XGEimFBY$bigY)!&SbSVfa fbYiXESVY)!q fbGYC PB R a 8 g e W 8 I P W R PB g D D s P W 8 g s ` W e s H P W 8 H 8 8 a W 8 I P W RB P T W e R a P W 8 8 I T P 8 W B e a a R T PB B VbSV1YFRdP SSXVEYg8 3x HVSdVVXfa SEY$1XESwSb)ESYXV&YYYECiYu)YVECGFHwb)E$bfYVYiYu)YVT 8 H H 8 T R 8 HB R W R W RB P s a 8 `B H W R T P e R I PB @ WB ` R T I PB @ D s 8 ` PB H s g 8 a R 8 I P WB ` R H H 8D 8 HB R W 8 I P ` 8 D s T H 8 gB P 8 g R H R HD s HB HB I a 8 x s 8 SVEVSEYYAVEFDwbVimSVpXScSE$ECEYCi Vbb97 1XY))I VSwXfa Xx 0 W R W W s 88H R R a R {
' ) |Ps IP IT eHr b$)qYXi 8 P s a 8 s a 8 x s W s P s HD R g q H 8 I P 8 ` R T W 8 R P D B H H HB PB ' 8 B B H SwSl$Sbb$E)i)SiYc)YVYbSF8YmSdR EE(bFxdFPdR q W s a R X3thq zbUSEYFPVz)XS1SVEXfYibiSVzbSE)91Xg{ R a P W 8 8 PB W n I B @ 8 T a e R H H H 8D q a R g 8 g 8 P 8 a T HB ` s a R

| HB I B I @ % $ # 8 I P HB g 8 a R 8 I P P H a n  W R W W s bEVFTY5&fX"b!fXYEbfXVYCSfGH XY))I 8 ` RT H eR8 )YVCYXVYW s P W s P H WB W s P R W HB ` W s q P a R y n 8 8 I P 8 x s I P R W H 8 R ` PB P s I P 8 x a 8 H dYu1YE2Yffb8 fa gfa Yl)YVY)EC$)fbSYXR rV)m$X$q fblY1Sw)VElS$dYm)YV2Y3F8YgXV!)EEYiEEYI G WB s s R a P W 8 8 I P R P D s e w 8 HB 8 P s a WB ` R T ` W s D s ` R T 8 ` HB HB I P 8DB 7 R a P W 8 8 I P R P D s e w 8 HB 8 P s a WB ` R T 8 I P 8 H s T HB I P W q fbYS$))VES$SYm)XV~YS$bEY1! fXGC)YVbYXE$)dbSS$s EYCi8 3x2gfa EEYifXG)XVYW ` a R @ 8 ` R T a 8 W RD s R W a 8 P P PB 8 g s H B y n 8 HB I TB I @ ` a R @ 8 ` R T R FHifbYlEw)XESE)YXVlfYVgfES$ASE$lEYbYf)lzXiXs B 8 a 8 I P HB P s I P r W RB PB ` W R T    8 I P H 8 n HB P s H R HD s HB I P a 8 I P a e { 8 a R g a R WB PB s @ P e R I PB @ 8 ` R T 8 ` W s T 8 @ D R g q H ` 8 ` R T W 8 8 I P R H PB 8 I P 8 x s I 8 YmSE$ClYYYEC)YVV)biG&)idV)YVYbcYG$lSEYYc)c@ 8 T W R ` 8 aB e w 8 a HB WB v T s a P v T s R W HB P s I P 8 ` R T 8 ` W s T 8 VYXVfmYVztEiYmV$SV$)cYE&$)pg33Vc)XVV)1b@ P s I P WB WB P H 8 a 8 P WB HB 8 ` R T HB I P a 8 I P a e 8D s ` R T 8 ` qD 8 e wB W e HB 8 ` R T HB I $)mYmSSVzbSmEc)YVEYCbYf)"EYgYVV)9EVYEY)lEi)XVEYCC R a P W 8 8 I P W s I P H H D B I TB I q fblY1)dVF8FHqEY@ ~ hC &o ! | q W 8 xB HB rD R g q bEE)3iSH ab8 dFPY$b)YwSb&XS$ScYm)YVYY)YVGFHYX3$)fbSY H B R a 8 g e W 8 s a 8 x s a R r 8 P s a WB ` R T 8 I P r 8 ` R T B I P a R P s I P 8 x a 8 H q a R P T s HB P s H W e HB WB v T s a P v T s ` W s fXSV$!fES$Y)EXYmV$SV$)2Y3 g s 8 a P H 8 I P WB ` s 8 I s a 8 I P a e H PB P s WB v R D q D 8 e wB W e ` 8 D R H 8 a 8 `D e R bfdY1m$bYbYf)tSEY$YmXYFRFqVY)EY)VFxSVfl)2EYXVT q PB e B g s W s I T e H 8 a 8 I @ 8 ` R T s W 8 x 8 rD s a 8 W 8 WB r a 8 I P a e { q a R P T s HB P s H W EYXEYiVYfbYl)YVibV)3$SbYbmYVbYf)zXSV$!fES$Y)e 8)XViVYCV$clEYCglXR e f e $c)YVV)SbYVY$YXY)` ` R T s I T e H H 8 v s g HB I h i a Hs 8 ` RT8 ` RP a8 IP8 I @ @R W v PR W R 8 @ r o o 8 v D 8 T W 8 e w 8 H s I PB @ ` 8 P W 8 H 8 P s I P ` W n 8 @ r WB ` R T W 8 HB I P WB H G$)c$mFBVYbY)VS~1ECVdbSVza $)1YgYm)XVYbGEYiYmSCCo | W a e P WB 8 ` R T I T s 8 a 8 B H W R T 8 S)2m)YVV$bb)F`dYXV97

 
H p

Xr p tsqc

e | P e R WB I bYX1YmYP 8 g R H P 8 DDB @ 8 @ rD R g q H P WB q a 8 x 8 a R P s I P q P a R 8 I P 8 x s I 8 @ P s I P 8 P R iXdVEECGYs)idYe m1fbVX3G$)cffb8 fa Yb)GG$)S GXDc bb GtXq$c GtX&c ` r e ` r ` r x y w tXDc bb gtXq$c tXQc r e r r bb tXr c tXr c X 7 tr e c |  $I3i% $ DgVX!YwbdSEf)bAFY 8 I P H s W 8 P P B a @ 8 W s T H B I fEEFDY)Xfa w)XESE)YXVYE1 YtXr p c q PB B s R D s W RB PB ` W R T 8 I P HB VSdFPpbcEB qc 3VY))V1YSmQVSVVDmiEB X i$)fEEFDY)Xfa YiEi$)C ` 8 P B g 8 H p rD 8 W W s I T 8 I P R P WB ` 8 P T 8v WB H B P s I P q PB B s R 8 I P HB P s I X uqc 5 tXsqc f r p r p | H 8B PBDB s R W RB PB H W s a P R P 8 H s H s ` 8 P W 8 H 8 8 a 8 bVESEEEY)Xfa XESESYS$CVSiC$VSbSVfa fi) W 8 I P W s T D 8 W W s I T H H D q a R g 8 g 8 P 8 a T HB ` 8 I a R a a 8 8 g R H WB q B P W 8 `B a 8 ` R T 8 ` 8 I bYqbVY))CdVF8YfXpbiSVzbSE)lYC&XfSbiYSYmX3Sb)Eb)XVV)~YP HP W8H8 8 SbSVfa za i fbYYk ! o d V)I $Ye YX1YY$o d V))I wYe mB 8a8 I @ r i P8 s Ds P P eR 8 IP ` Ws k P8 s Ds P W q a s WB 8 I P 8 x s I q s g 8 @ 8 g s y 8 a R 8 g s H 8 I P 8 P R W ` 8 8 g ` W s P s I P 8 P R f)mYYb)icCED )VX3liY&YVVYW hYG$)GS H P 8 s D s 8 H 8 I P a 8 x R 8 W s a I TB I @ f ` W bSV))I wSVYb$YSVEYbYs VF8YuzyX)YdYlk DdSbbbQd H D sB a s x g R ` W s a ` W s e c c a P8 s Ds P P e R ` Ws ` X X bV))I $Ye YXYk VYSbbQd 1V))I $Ye mt1b)F`dYXV97 W P 8 s D s P WB W s a 8 B H W R T 8

D 8 W W s I T s a 8 x R a 8 H W s a P a R HD R g q H R WB ` R T 8 ` ` W s WB ` R h | o 8 a e B sVY))iGb$Gb3fYdXsE)i)S$Ym)YVV)YYu)YG&bcf)XE{

8 ` R T P P e R WB PD e H 8 a 8 I P R P WB H H 8 W g R ` W s a R P W 8 g 8D 8 8 I P H 8 T e ` R a P WB ` W s )YVYe YXYmSEYVflYcdmCSVYX)YS&$bibEVYVVYgYfuAY)I a 8 I P R 8 I P W R 8 B R ` 8 H a 8 x 8 a 8 qD a s 8D T W s T H H 8 T R HB I P 8 ` R T P P e R 8 g s H 8 I bYY1XldFHC&VdfbVfi)2EfbEVbCSVVYfa EY)YVCYe YXiYP 8 I P WB H PD e H 8 a H q s D s 8 ` R T P WB W s W 8 I @ HB W RB P a R P HB i 8 HB R W ` W s r W RB P a R P HB ` Ym&SEYVf&YCF@w)XVGYe mibY&E~XESfXdSESEYYYXESfXdSE)8 3B r HD R g q H ` 8 ` R T W 8 8 I P R W RB P s T B ` R g TB P s g 8 P H q H W 8 8 @ P 8 I B e WB P HB ` `D e R I H 8 bEiSV)XVYbGYg$pXES$bgn)XptESwcbSSYS~bVGVFHYYYudSE)cEYXYG97 W RB H a 8 x ` 8 B ` R g 8 g R XESfbAVgn)XilpXSH 8 P s a 8 W 8 qD B H H S$dbYbEYmSSR YYm)XVYe u1Y93)XpSGVY))VYib)ESYXV1iEB ` W s WB ` R T P WB 8 I P q B ` R g R P D 8 W W s I T 8 I P a 8 `B H W R T 8 @ H P$)cbdSXdY~SudFHY"wlb)EYVYgXfmiSE$G)FHY)Vi1Yu)YVYb8 s I P g 8 P H q H 8 I P R P WB 8 B R W R s 8 `B 8 I P 8 T e ` R a P WB R HD s 8 @ g B W s I T 8 g WB ` R T W 8 P sB R s 8 g R H q 8 T a e R H s I T e H q ` 8 P s a 8 W 8 HD R g q H H H R P I HB @ 8 @ I TB I S$mfa fa ) iYSiVz)XS1VYiVd$SbYbGE)i)S&d$s SECCGcVEY@ I e R a I P D 8 W W s I T s a 8 `B H W R T 8 @ R W r 8 T a e R H 8 P 8 a T HB ` 8 I P ` 8 a 8 B H W R T 8 x s I 8 XYXz)CVY))Vb)ESYXVG@ $YGVf)XSASVfbSE)2YVzb)F`dYXV)297
U V

(  4 ## U 6   &1$g( $(

X
Symbols Source encoder

Channel Decoder

Symbols

S T

( ( ' ))$# 9R

  p Xx
i

k X

m Gl bo o lsbo k Xr t& c

| R a 8 j 8 a s W RB P s e w 8 R P I B gfbV5zcu1XESw)V$CXEfa 8 I P W R g e H 8 I P WB H g a 8 P 8 I P R P H R g P s I P WB x a 8 H R W 8 I qD P T 8 a YY)cY9miSbdcYC$Spl$)2YmYfbSYXUbYCESVV3 b8 VEVVVfa ` 8 xB 8 T 8 H q s @D s 8 a s a 8 ` WB s g 8 a 8 I P ` W s r R a R a a 8 WB ` 8 xB 8 T 8 a WB 8 R q PBDB s R XCE$zb)Ym$cbfcYYgbo &$xiXfSbAm9VEVVVf2YmV$EEEY)Xfa s H s I P H a n 8 I iI T B I @ R r H D R g q H i o 8 x s I 8 @ 8 a 8 I @ r 8 a e n 8 I P a 8 B H W R T r 8 x R $)SfY VEYg$$bE)Sq)GGfbYCf)YgYb)F`dYXVbys P ` 8 B a T H 8 V)mfbSV)` 8 P s a a R a a 8 8 I P R q P a R WB s a 8 x s 8 I P ` W s P H a 8 ` W e R P r a 8 x 8 @ R S$SXfdbY$ffb8 fa Yu$SbcY9YSfb)Y)SgbVGy s 8a )lf)e n r v WBD H W RB P s TB W e g g R T s R D 8 ` R g TB P HB gB H s 8 `D e R T HB I P P P e R 8 I P P $g")uE~YXES$bEY)icXV1$V)Yi2ESSEED mS1EYXVlEYbYe YXiY$s P W W s R WB ` R T 8 ` h P T 8 a a R T WB g R r r q BDB s R W R g g R T s ` W s k Ye uB $9Ym)XVV)AVVfSYVYudw"fFPEEY)Xza XicYVY$$o d bSP W8 PB a @ qDD s e H e HD R g q H P P e R ` W s P WB R @ P H s I D 8 W W s I T TB a P 8 g g q H q a s WB 8 I fEfiEE$)Y3E)i)SYe YXcYYe mGf$)VY))GEfVpcdz)mYYC d d )x$bXfe!hDg3i ux
X

xE

o H s W B 8 T 8 a P T 8 a a R T R q PBDB s R 8 s a 8 x s 8 I P r D WB ` W H 8 a a R T ` W l$2YFRdP VVfCVVfSXV$2fEEEY)Xfa $SbblYFqYu)YYR VfdXVYs

p H H `

X 3

Xr p tqc

Hp
e H qp

Qc f r p

H p

| H s a R a a 8 D R g q H R q PB B s R bwYfSbiS$fEEFDY)Xfa 8sa8xs 8 IP 8 W n8 ` W8 IP 8 i $dbYCY)~bY7 Xk 7 o d Ye YXiYG$o d Ye uGY~mGED )VX3 P P eR ` Ws k P WB 8 I P WB 8 g s y 8 a R r HD R g q H P P e R R P 8 H e H 8 g R H P T 8D 8 H q s g 8 @ a D R g q H P WB 8 g R H R PD e H 8 bEiSGYe YY$CVSY)iYSCVVEVSccGG&3)i)SCYe miXSwCEYVfa ` 8 aB H 8 ` 8 I P WB 8 H s D R g q H P P e R I T s 8 q B P W 8 `B W 8 P R 8 @ r g 8 x s I 8 @ VzudV)cY2YmVl$)idYe YXV$bsSb)EAbSswGtQc)pGGx YX 3 tXr
p qc p c

H
`

f 5

qc p

| WB a s p D R g q H P P e R R q BDB Yuzb8 ) s qc iSYe YX$AfFPEEY s R 8 I P B P s I P r R W RB P e B a P HB ` q PB B s R D s WB a s g 8 I P ` W n W 8 I P W s T 8 )Xza YFHG$)Xi$AXESY)mfSE)fEEFDY)Xfa $)mflYYbYb97

p qc

X 3 Xuqc rp 5f X

p c qY

| f ` W Ys VEYmfyY)YSYC$tYFRdY)uzSE)fEEEY)Xfa mfYm$YXUbYP H 8D sB a s x g R ` W s a 8 I P R W B P e B a P HB ` q PBDB s R P WB R v 8 I P WB s P R W 8 I 8 7 k o 9Y Vb 5l X d Vz)XSGSVEXzXibiiXf!Ye YX$qXESY)mfSE)cfEB 8 T a e R H H H 8D q a R g 8 g s g R a P P e R R W RB P e B a P HB ` q P EY)Xza YVY))VCY$GV))I $CYe mCYpECcVS$mVYSS$)GlR 3VYW DB s R 8 I P D 8 W W s I T 8 I P R P 8 s D s P WB 8 I P I PB @ ` 8 P sB T R H H s 8 x s I 8 @ D 8 W s I T s R P P WB 8 I P H s 8 T a e R H H H 8D q a R g 8 g 8 P 8 a T HB ` s R P P e R 8 I P 8 v s P 8 @ P y 8 )VSGYe mCY$CVf)Xd&SVEYfXibiSVfbSE)$Ye YXY&$CGXV

 $ 8 P H e g W RB P s g a R WB I T e g @ R I v H s P I B g 8 W o 8 a e n 8 rD 8 W W s I T 8 I P lSYiXES$cSYsmVYiyYA$iXEi1Yc)cf)Xgg 3VY))VcYG$R P P e R ` W s P WB 8 I P I P R R H W RB P s x a 8 H R g R a W RB P s g a R WB I PB @ ` 8 8 B x 8 ` W RB Ye YYY&Ye uGY~)$YXESwyfbSYYXzXES$cSYsm~ECiVstVFTYV)~XESP VfSYViECGbYVSVY))VlYCYX))uYdEmR VEX~ES$)SbSE$W e T 8 a a R T s I PB @ a 8 I P 8 R P D 8 W W s I T 8 I P P e R s v WB I P R P HB P WB @ 8B x 8 xB P s W a 8 PD s D 8 W W s I T 8 I P P e R s WB I P 8 g R H H e HDD 8 P W RB P s g a R WB D s e P e sVY))YYX)YmYViXdCYEEVSqXES$cSYsm$)Yig 8 I YC 9VsVY))1$R 5Ye YXiYYmXzbSYXbSsys YX)VVYgVfa q ` 8 D 8 W W s I T s f P P e R 8 I P WB x a 8 H R a 8 P P eR s `8T e `8 q P WB s P a 8 T W e a R W B P s g a R WB R P W e R g s 8 I P R 8 a e H s 8 g s I PB @ H e H 8 `B x R HB I m$fbVY)Y2YFRd$cSX3m$)XiY$if)wbiECYV)EYyza EYC a W W a W r qAq x |D 8 W W s I T 8 I 3VY))VYP R % 6 % 8 I P 8 B a @ W 8 I P 8 7  8 I P HB W $x3iYw"GYqX1YcdFPzbYi9i$vdf3& 3xAYlEa r q Xx
p qc t

p c qtr

X o Y

s u

RD E RD E)
p qc

p c qY

X X

H Hp
` e

cr p tr X

s h

p c qtr

H Hp
` e

Qc 3 p

f r  W

H p

a r q W

| WB s s 8 s a 8 x s 8 I P 8 v s P W s T 8 @ R H r 8D sB a s g R ` W s a s B HB I m$Y$$SblY$1bGcSEYmfyx X)YScFHEYP
t q Qr p c

s h

RD E)

cr p r X

H
`

p c

fr

W q

| qc bYmXzbSYXbS3q fbY~Y)l9f f WB x a 8 H R a 8 P s R a P W 8 8 I P 8 W n 8 ` 8 7 p 8Y&$iYm$$YYuYfbSYY9VYm$YXR YX)YFRd$cSX3mlYb)ESYYVT $YW I P R 8 eD s x 8 I P WB x a 8 H R q ` 8 WB s P P e R s W B P s g a R WB 8 I P a 8 `B H W R @ R W s T 8 @ r 8 T a e R H 8 P 8 a T HB ` 8 I P a R R a P W 8 ` W s W RB P s g a R WB R H s 8 `B 8 I P WB ` W 8 P y bGYVf)XS~SVfbSE)YX3iq fbYXESwcSX3m$$b)EY1Ym)YbSYj 6 o  (      U  % "p5& g &%
S n 

D R g q H q v T eD W e 8 I P R D s rD 8 W W s I T TB a P 8 g g q H q a s WB f s | 8 a e B )SYmY)iY$~F8wC)3VY))VlEfViS1f)mC)f)XE{
1 1-p 1 1-p p p 999999 1.0 1.0 1.0 0.1 0.9 999999 3 2 1 0

X
0

Y
0

3 2 1 0

  qD H e RB x 8 ` 8 I BD s P H 8 8 @ q P D s e w 8 WB 8 I P R 8 H e q ESYXEXVza VYFHEYSVG2fEFBw)VYmY$SY Xx

t

z H p

pqc X X
p c qY s p c qY

RD E)

p c Qd

X
p c qtr

H
`

{2 a W
x

HB I P 8 P e PB P H e H ` W s EYcdYESSY)Y

p X C qc 3

|YxYscYFRd$)VYquB 8 R W BPs e w8 8 IP W 8 R 8 HB I P @ R I H R X GP$s)IPiSPYWcG@rbEY5$Y1 z a W )hyCAq o

x

| W B P s g a R WB D s e P e g 8 I P R H 8B P a R H e RB a s x 8 T e ` 8 ` W s T 8 YFRd$cSX3m$)YiY$VESzb8 fa YXEf$lVYgV)qb97
t s u

p c qtr

X X

D FR

p c qY

H p

H a W Aq
` x

| W RB P s g a R WB D s e P e g 8 I P a R W RB H H 8 y 8 W s WB s P R 8 @ 8 T W 8 XES$SX3u"$)YilYX31XESSVza Vqm$YXGlVYb RD S X uE)

p c Qd

H p

H
`

e H p e

RD S X uE) X 3
Xr p tuqc

Xr p tuqc

H
`

H p e

S X EE X RD

H H
` `

o S X EE X RD

1q W

| W RB P e B a P HB ` q PB B s R XESY)mfSE)fEEFDY)Xfa PmyYb$)9$c1qXESVY)!Uq zbYASEfVfbYf)b97 WB R v 8 I P a 8 x R g e H s H s W W RB P T W e R a P W 8 8 I P 8 PB a @ 8 a a 8 I P a e W s T 8 r 8 f B i) R P P e R H W a e P HB I P P WB 8 I P P T e a P H W R T 8 a R P D 8 W W s I T W RB P T 8 a a R T 8 I P W RD s ` 8 H H SYXCYS)EYbYe mlYCVYdSYXVVf1SVY))VXESVVzSXVlYcYXE$VdS$s g 8 P H q H W RB P T 8 a a R h | o 8 a e B bdSXd1XESVVfSXt&)lf)YF{

Observer Correction data

Source encoder

Decoder

Correction

H P 8 s D s P WB TB a P 8 g g q H bSV)I $Ye uEfVic)S$s a R W RB P s g a R WB D s e P e g W R P T 8 8 8 I P H @ R I H 8 a e B { I e R a I P W RB P s g a R WB X3XES$cSYsm$)YiXGVVdYGC$Y1f)XEYYXf)1XESwcSX3m$R P W e R g s g e gB y s g 8 I P P 8 R HD s 8 @ 8 aB @ 8 I P W R HD R g q H 8 I P 8 H H W s a P qD sBD 8 )Yi)imY$ccYVASE$cfmCiYAXlE)ScYdR YS9EYmEVfa 8 @ B o a R HB q PB B s R W RB PB H W s a P 8 I P W 8 I @ ` 8 jB gB y s g B ` W s r o HB q BDB GcGCYXtEfEEFDY)Xfa XESESYSYbYVVimXwcCFH1YY)&lE1fFPEEY s R W RB PB H W s a P 8 I P W 8 I @ R a 8 j R P R a ` D 8 W W s I T 8 I P R q PB T s T 8 I P P s I P 8 P R )Xza XESESYSY1bYifbVSGH zgVY))VY$fEV$s bYG$)S q BDB s R W RB PB H W s a P P H WB s s D 8 W W s I T 8 I P R q PB T s T 8 I P H @ R I H s 8 a e B fFPEEY)Xza XESESYSSYm$X$CVY))VY$9fEV$s bYiC$Y2f)XE{ o RD o gCfE)gCzgEFRD &o C{2q a W o D o RD g2GfFRg2GfgEEC a r x W o 1q W | q W 8 xB 8 a s W RB P s g a R WB D s e P e g ` W s R a P W 8 D s W B B ` W R T r R a P W 8 8 I P W 8 I bEfAXES$cSYsm$)Yi2Yq fb$)YFRdFP)YYVXq fbYqbYC t o Co s | HB yB a P s g D 8 W W s I T 8 I P ` W s k H 8B PB B s R bEEf$cVY))YYYbo bo d VESEEFDY)Xfa ` 8 P sB T R H H s ` W s k VS$mVYSS$Y$o d |V))I $2w)I Vz)XS2Ye mYC 3VY))VFTzVig W P 8 s D s H s 8 T a e R H P WB 8 I D 8 W W s I T B a P 8 g q H q a s WB 8 I P a R W RB P s g a R WB D s e P e g ` W s R a P W 8 D s W B B ` W R T 8 I P a 8 B H W R ~)S2z)mYpYCX3qXES$cdXsm$)Yi9Y2q fb&$)YFRdFP)YYVYb)F`dYXGh )hDg3i x"I

p c qY

a q9ahAq W W a W | 8 P a qDD a e s a R g 8 I P B W RB P s x g a R WB D s e P e g 8 I WbSPSEBf@EE$sS)Pw)W8fYiYFHqXES$cdXsm$)YilYC Hp H 5a q W RD X EE

p c qY

| W RB P e B a P HB ` q PBDB s R P WB R v 8 I P W R ` 8 H s r a 8 XESY)mfSE)EEEY)Xfa mfY1YVS$)gVbYW W s g H e RB x R 8 I P WB H 8D sB a s x g R ` W s a R @ P R R a P W 8 P WB R v 8 I P 8 W n 8 ` 8 7 cYXEXXYquVEYmf$X)YScGf$cq fbCmfYlY)9G a a a W cW r 9&CAq | P P e R 8 I P R R a P W 8 8 I P R H g a 8 P WB W RB P s g a R W Ye YXY$cq fblYx $CidbSm1XES$SX3uB D s e P e g 8 I P a R W RB PB W n 8 ` TB a P 8 g g q H H e RB x R 8 I P R P H ` s 8D WB ` 8 T 8 8 I $)YiYYs XESEY)cEfViS~YXEXXcYSl)$bEYm)VVVfa YCC 3
~  qc p

a W {2q
x

Xr p uqc

} b

RD E) RD E)

p c qd

iYi W a H p H x
e `

cr p t X

X Y

c p d

yYX

H p

H
`

| Bs R @ r Wm$PYX8G) X 3 tXsQc & qc 3 Qcr X lYmSC p p WB H r p W Vcda {2 EfVic)SEB a W r TB a P 8 g g q H H x x x Ra8j H8 g RT8 gfbVyViXVV) g Sab8dPEDc8YIPc8VTYWb8YIV qc 3 X C pqY X "bPb)YWb8 )`YmBzs TYs x R r p c r W8 ` 8 W 8a f ` W W ` bPb8)YWb8 8)YmB2EE$bTESSEBS$Sfaf|`Ys ~CAq ` W qDD s B P H P s P H 8 s W B a W
x

p  q$)EYu)YVYVY))gwVSY)$bf~mcVfb)ESYXV)SVV8 $mYbESfs Y W s g P s I P HB WB ` R T D 8 W W s I T R H 8 H e D s 8 a WB ` 8 a 8 `B H W R T 8 R P P T H s a sD e TB P a 8 W H 8 g 8 I T H WB ` R T I P W 8D D sB a s x ` W s H 8 T a e R H D s a 8 W 8 8 a R g R VibYdYm)YVYbE~F8YuzyAYVVf)Xd$SbYbfXpSP H 8B s W RB H H e T HB ` 8 I P r a 8 x 8 @ R I HD R g q H 8D s R B e w H s I I TB I @ r W P 8 s VEED ) 2XESdYbSE)YYVbVG$YgbE)i)SiEY)Xfa Y)V8 $)AEYYAV)I $iVf)XS8 3V)YVVXEYVSVYgCwbi)XfmYb~SVEVSEY1m9V)YVYb8 D s 8 T a e R H s B H 8 ` R T v T R D 8 j B H ` 8 y n H s P W 8 g W R a B x W 8 H H 8 D 8 H B R W s W B ` 8 ` R T W 8 DD s a e P s W D e R @ I B I @ H 8 T a e R H R P ` 8 P B a P H 8 a 8 a s 8 a 8 I ` 8 H e 8 a s P s I P H 8 g 9FqE$d)$)F`YYUVFTYlVVf)Xd2S9VdVFTzSVfzfbYVSYfl$)iVED s VYfXfSbF8YmSdR iXVfb$1SSVVYfa Ym)YVYbiYYmf)gVYgY)gVfa y 8 8 I H a R a a 8 D B H H 8 g R T a 8 x R R P H H 8 T R WB ` R T W 8 8 I P WB a e ` q T W s ` W e ` 8 WB ` ` s a 8 B H W R T P I B g 8 @ r g 8 P H q H 8 I P WB 8 B R W R g 8D R 8 I P 8 g R T a 8 x R R Yu))wb)F`dYXVXEi2GGbSdXSAYQmdFHYr$bEYXza YiYVfb$"  ' U (  4 !&)V&


D 8 W W s I 3VY))VT 8 I P R P ` 8 I T P s g WB 8 H s 8 T a e R H 8 I P 8 B a T H 8 ` 8 @ 8 P s a B I P 8 x B I T s 8 @ W 8 I YtShVYVS$YmV)$Vz)XSY9uzbSV)2GS$d2FHYVF8YV$QbY7
d&

o b)3

a W Aq
x

ys $g

|Hs r q `8P R W8 ` r  $ 8 IP 8 W n8 ` 8 7 P8 s D b$b2VSYb))$q d&GXwcYiY)9)iV)I $s P WB 8 I P a R H W RB P e B a P HB ` q PBDB s R DD s a 8 x R W RB P s jB gB y s g s g a R a P H e g 8 Ye uYX3GYXESY)mfSE)cfEEEY)Xfa E$b$XES$bimY$cl5dXsb8 SYiG@ rXESwcSX3m$)YiYlVimXwcdECG3VY))AbEpCX3G$)~SYW W RB P s g a R WB D s e P e g 8 I P 8 jB gB y s g R P I HB @ 8 @ B rD 8 W W s I T W 8 xB s a R P s I P 8 P R 8 7 yB a P s g D 8 W W s I T 8 I P q W 8 xB P 8 s D s P P e R P W 8 ` W 8 ` ` W s P 8 s D 91V)Ef$cGVY))VY9bEx2V)I $cYe YXib)Yb8 )YpV)I $s P WB 8 8 a s R D 8 ` R g 8 I P WB H e rD 8 W W s I T s R q PB T s T 8 I P 8 W n 8 ` R P I HB @ 8 Ye u1Vf!1$V)XiYYmSY&3VY))VAwfEV$s b1Y1Y)2SUEC197 6 '  o  ' U (  4 V3)v)V& TB a P 8 g g q H s r TB a P 8 g g q H s D 8 W W s I T q a s WB R q PB T s h | 8 a e B EfViSwC)gEfViSVY))Vf)mY$1fEV$s C&lf)XE{
0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 P(transition) 0.8 0.9 1

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 I(X;Y)

0.5

0.1 0.2

0.3 0.4

0.5 0.6 P(transition) 0.7 0.8

0.9 I(X;Y)

0.9 0.8 0.7 0.6 0.5 0.4 P(X=0) 0.3 0.2 0.1

  8 a e n WB ` 8 P s a P H W R g 8 ` HB a R a a 8 R q PB B s R D s e `B H 8 a 8 I P P H WB s lf)Xg1mVSwSSYXpb)EXfSb$fEEFDY)Xfa $)gESVflYCdYuwX$s 8Ye)EBY)VdlYFRdFPdV8 fGYwS$SG)XVYCVf)Xgn b) ElXES$))V8 w W IT8P W B BP 8a 8 IP R 8Psa 8 ` RT 8 I s 8a e o H B o W R B P s e w q W 8 xB H s q PB T s T D 8 W W s I T 8 I o R q PBDB s R W RB PB H W s a P s a 8 B H W R AbEC$2fEV$s bVY))VlYC&) &$2fEEEY)Xfa XESESYSpb)F`dYXGh
H " I t l "7 gCz 5 s o o " | B P s I P r a R a a 8 WB ` 8 xB 8 T 8 a 8 a s H B 8 a R g a FH$)YVXfdbm2VEVVVflzdFPYfXpCXR o B a R a a 8 W s WB s P R DDB @ ` W s rD R g q H a H B o @ PB g H W s a P 8 @ 8 T W 8 YfSb~cm$YYgEECcYYs)idb8 dFPYCEiYSCGCVYb WB P R x q PB a R v s gYmScfEfXfqcg dXsb8 ga Ra ` W s H 8 gB P R a 8 g e W ` ` R W s H PB B ` 8 H 8 I P P s 8 a D B @ 8 ` R T W B B P 8 a 8 I P k YCVimS$b)i)X)YiCSEE)dVY&$b8 fEFD)YVcYFRdFPdV8 fY$o d W 8 I P B HB I P a R WB ` R T W 8 q a s WB D s a e P s W 8 I o D o RD bY~FHEYYsqYu)YVYbf)mY$S)$)cYC~VgzFRgz"gEEGo fEV$s bVY))VtVFx2EY~EEEY)Xfa XESESYStEClVY))VT q PB T s T D 8 W W s I T s H 8 B HB I P q PBDB s R W RB PB H W s a P s I PB @ D 8 W W s I TB a P 8 g g q H q a s WB s 8 v s P 8 7  s HB H 8 ` R T P H 8 gB H 8 I P R 8 W EfViSQf)mYt2$A9bx3x3bfh2E1V)XVASVED mS2Y$Y

"lbmbY

H8sHH8 bV$SVig ` 8 ` R T W 8 8D B H e s W 8 8 @ P 8 h 8 T W s P HB ` g P W s T B W B H s 8 x 8B I T s R P q a P ` W s r a 8 ` R V)YVYb9EYmSYmD bVGfV)!VYSE)STbgnYXES2VEYV$9S9fYVb)YVT 8 ` ` W s D 8 W W s I T 8 I P R H TB P HB a 8 P T s a s I T a R a a 8 8 I P a 8 `B H W R T R P H ` 8 8 W WB ` R T 8 I )UYVY))VcYG$iVESSEfbSV$d)YfSbcY~b)ESYXVd)VVY9Ym)YVcYC I T s 8 TB W s v T 8 a @ R P q H s 8  P V$b8 VEYcVVfSd$bGH &3C 8 T W s ` s R P WB R 8  8 @ r 8 T W a e R ` W s 8 8 a I P WB a f VYgicSYmfaVYb8 )Xs1YVf)YuzCo | WB a s 8 I HB g W s g e I s R P 8 e ` a R a a 8 8 I P a 8 B H W R T r a R a a 8 a 8 P T s a s I gYmfbYfEic)XSYgGXzSbYb)F`dYXV)VXzSbbSV$S)VT g R ` W s a I T e H P H WB s s H P T 8 P R 8 T a e R H HB I P R q T W s ` W e ` 8 a 8 I P 8DB I @ r a 8 x 8 @ R X)YShVYcSYm$Y$SVVSfa Vf)YSEY$QVYgY)gVf2YEEY&bVGy I T 8 H 8 HB W R T 8 a R g q H s 8  P VVV8 ~SEYXVVf1id$bGH &3C 8 T W s x ` s R P WB R 8  8 @ r H P W 8 8 T a R WB 8 a B a f VYy)$cS1Ymfa G)bSblVzXsmVf1VjzCo | ` a s @ a R P I B s a P H qD 8 xB P s a g R T 8 ftzXsXE$SdEVES$ds XV~) `D e R @ H a 8 P T s a s I T o WB o R W B e a a R T R P 8 e ` WB @ R D R 8 I P g R a W RB P T e a P H W R T 8 EYXGfbSV$d)mG$iYFRdP SSXVSYglYmC$EFDsGYXf!~XESVYSdYXVVG I T 8 H 8 HB W R T 8 a R P q H s 8  P VVV8 ~SEYXVVf1Sd$bGH &3C 8 T W s x ` s R P WB R 8  8 @ r H P W 8 g 8 T a R WB 8 a WB a f VY$)$SYuza GgbdbibVzXsmVf1YuzCo | 8 g 8 P H q H y D 8 qD v TB H a 8 I P s a s H H R a T s W RB H HB g H W s a P q ` 8 T e ` R a P WB 8 mg VcbdSXdVF8V"2EVESGbYwScCSfbw2XESSEiYS1VVYgXzm~) P I B g H s I T e H H a 8 P T s a s I T o WB o 8 s a 8 x s W R ` 8 e a a R T qD g R ` W s a I TB I @ D 8 W W s I XEi&$cVYxzbSV$S)VbmG$dbYcVSP SSYVEiY)YScVEYgVY))VT s I e R a I P P y 8 P g s a R a 8 P g R T a R I HBD W j D e WB W s 8 g q D s B P W s g 8 H ` W s P T 8 1XYXf)YVSSYfa bSYe XVtXEEYYwYuYbi2EFDwbFTdbSYVVfa XVEE$bESV$)S$1VVEB Ym)YbSrb)F`dYXGi)$bfm$VYgY)gVzt$)Yis a R T qDD s TB P T s P W q H R 8 T 8 s WB ` W 8 H a 8 B H W R h q ` s 8 aD s q T W s ` W e ` 8 a R P W e R g P W s T B W B H s 8 x s I H W RB H HB g H W s a P a R WB ` R T W 8 WB ` 8 P H 8 a 8 P WB 8 a s 8 @ I B I @ H 8 T a e R bgnYXESrb)GYXESSEiYSYsYm)XVYbmVSdVfbSufVFTYVVz)XSH

  | H a 8 I P R R W ` W s HD R g q H P P e R q BDB VfbYcY2Y3cE)i)SCYe YYC)fFPEEY )Xza &9EC8 3x2F8Y)Xfa YVXEV)lG)r m i ! XESwbdVfa fa s R o I PB @ B D s R B e w 8 P I B 8 8 x s I 8 @ W RB P s P W 8 H 8 8 FPF)Af)mY9EC@ X Ye mAbE1XsYSbSS$s Ye YXYiYe mB m )c97 B B ` q a s WB I PB P WB W 8 xB s a R H W a 8 P P P P e R ` W s P W 8 x s I 8 D 8 W W s I T H B I P R q P B T IVY))VFY$1fEV$s bT s 8 I P HB P s I 7 WB ` R T 8 ` 8 e wB W e a R qD a s 8D T HD R g q H P WB 8D s R B e w YCEG$)VXYm)XVV)~YEY)GX3 m hEfbEVxEiSYe mlEY)Xza Y)V8 P T 8 v WB 8 7 W RB 8 T 8 a P T 8 a a R T H s 8 g s H 8 I P HB P W 8 x 8 W RB e a a R T B s R q P 2VVfDm9XESP VVf1VVfSXVcw2iAYEcbVtXESP dSXV1FPYtl$fEB EY)Xza YlwiGVY))iYP uEiS)XVYbSVSYUbVS&$AVXEY DB s R 8 I P 8 v s P 8 @ D 8 W W s I T 8 I WB HD R g q H 8 ` R T W 8 R P ` 8 H e W 8 x 8 H R v T RD I T s 8 WB PB 8 W R e a a R T qD g R ` W s a q s g I TB I @ D 8 W W s I T D sB T B P a s a 8 I P s a s a 8 B H W R V$bim&EYCYXP SSYVlEiX)YdlbcEYVY))V)$mVgn dfbY$db)F`dYXGh 3$cgmhA x"I

bEiSYe mCED SEYiYm)YVcVVEYV$lbS3$f&V)YVbSSV)$S$bSVEVSEYW HD R g q H P WB 8 B PD e g WB ` R T q ` 8 x 8B I T s W 8 P R 8 a s H 8 ` R T a 8 P P 8 r 8 H s T H H 8D 8 HB R 8YECVEFDwSs !SEYVY))Y$GVESSEfbSVwS)VYiVd$VE)CE)YVT I P I PB @ D 8 D s a W 8 HB R W D 8 W W s I T 8 I P R H TB P HB a 8 P T s a s I T 8 I P q ` 8 P s P TB ` HB 8 ` R ` R R s R 8 TB R I T 8 I P a sD e TB P a YXAl$VEYVYimYbESfs !tS$bG$SbYbYtu1V)YVtYiVYSSYXVT W 8 H s T D s a 8 W 8 8 I P WB H 8 ` R T I T e H P T e a P H W R R P H W s 8 g s W s I P a 8 I P s a R R 8 T W 8 P HB y 8 W s HB g 8 a R 8 I P HB I P R R R  W R W W s SYbi)bY$SC$Xza VYbSSEYV2EbfXVYEYG$wXfa H XY))I ' W s I P H H D 8 P s a a R a a 8 D s e `B H 8 a s I PB @ ` 8 T e ` R 8 a HB 8 T a e R H 8 I V1)dVF8S$dCXfSb"$)gESVfcAEC2VVYgYfa fEVf)XSYP P s I P I T e H 8 g 8 I T H WB ` R T s H P HB y 8 8 a 8 I P r ' DD s g H qDB a s a B a s a R W 8 I w)VYiibYVSYm)XVildSEXV~fbYVE$2EEfSFPYdCX31bYP r 1 C tq fb&$iVf)XS9Ys |fEV$s b&$VY))V1CXg{ B R a P W 8 R 8 T a e R H s ` W q PB T s T R D 8 W W s I T s a R | % $ # $ 8 I P r g 8 a R 8 I P ` W R T 8 H  W R W W s P s 8 xB a a s 8 DfY&GY!cY)qbfXVYAYXVVSCH YY))I $lEfdl97  (  ( 4  U (  4 g&1!&~V&

o R q P DB s R W RB PB H W s a P s a R 8 ` R T W RB PB P 8 a R q T W 8B ) &$fEFBEY)Yfa XESESYSiGX3)YVAXESESV8 f$2VYbEVT j r H 8 P s a H H R @ RD P s D 8 W W s I T TB a P 8 g g q H q a s WB R q PB T s h s | 8 a e B c)bVS$dlSED $E$&VY))VEfVic)S2f)uYwEV$s CC)cf)XE{

0.9 1e-08 0.91 0.92 0.93 0.94 Capacity 0.95 0.96 0.97 0.98 0.99 1 1e-07 1e-06 1e-05 0.0001 0.001 P(transition) 0.01 0.1 Residual error rate 1e-10 0.01 0.0001 1e-09 1e-08 1e-07 1e-06 1e-05 0.001 0.01 0.1 0.1 Code rate Repetitiion code H(0.01) 1

I(X;Y)(p) I(X;Y)(0.01)

  Ha Raa8 o bfXfdbh VVfSXViSSEYSY2VY$VbVEE$bT P T 8 a a R T R P H PB 8 H e q 8 I P H s 6  ` 8DD s 8 a s H 8 ` R T WB g g s 8 I H PB D 8 H e o fcV)YV2YmicY2bSEYY!e SY) b3fYStYlYSbdS$s YfSb8 a 8H W saP ` W s H Wa 8PP a Raa o GVVfSXV~S&SEYVSY)YVlYmii$SbYb!bSEYY!SYbsfYSP P T 8 a a R T R P H PB H 8 H e 8 ` R T WB g g s s D s a 8 W 8 W H PB D e 8 H e a 8 H W s a ` W s H Wa8PP a Raa8 o Y YSbSSws XzSbib5 3VVfdXVdSEYt2VSY9)XVtYmic2EYC p P T 8 a a R T R P H PB H 8 H e 8 ` R T WB g g s HB I a R a a 8 WB B 8 I P H VXzSbmFPYlYEB 8 H D 8 r a R a a 8 R W H B 8 a 8 I P W 8 I P 7!lSEV$VXfdbYGEfbYibY $V b)i)Yf)mYY XESP VVf1 a 8 g e W q a s WB 8 I P B W RB 8 T 8 a W m Q!Q w r ` W Ys m Q! m Q i Q r ` W Ys m Q i m Q i Q! r ` W Ys m Q i e! |  twgqYgr q % ` W s q W 8 xB 8 a s H B WB WB s g 8 a 8 I P v T RD ` 8 ` R T W 8 8 I P ` W r H PB R bE~fdFPY2YmYm$cbzpYYEYV)XVYb~YwR m Ys i !r lSEYSP PB R T W s HB I P W RB P s P W 8 H 8 8 a q a s WB 8 I P 8 v s P 8 @ r HD R g q H o R o D s x a 8 P W Eq V2`YlEY$RqXES$bSVfa ff)mYY$GbE)SiDt$~G$yfbdmB 8igmS1V$b$sCSEib1bTEYVz)XSH)XXg"SX3V)YVYbVY))VY1mGbSVfa EB B P I T s 8 P H PB g 8 I B I @ 8 T a e R a e R a R { g a R ` 8 ` R T W 8 D 8 W W s I T 8 I P WB P W 8 H 8 H HD g q H 8 T a R H 8 I P WB ` R T W q a s WB H e B x R 8 I P I B I @ WB 8 W R HB 8 ` R T TB P s g 8 ER)SVf)eXSY$RcYm)XVYb8f)mYYXREYYCY1VFTYcmCYXGE)XVCESwcbSP H q s HB I g a R a R 8 ` R T % H 8 ` x R 8 ` R h WB g g s p 8 I fYSHqEYP dXsb8 SPl)YV 3iD3D!1sV)EBX$fa )YGYmiX$3YC 8 x R &$)s9VuzbSV)`ffb8 Rfa aXfdb8YcSCVVfs)SEYVY))VT ` 8 B a T H 8 q P a R a a 8 I P R P P T 8 v e H H PB D 8 W W s I p W H PB W RB P g a R WB 8 R T W 8 R P g H W s I T g ` n @ W s h WB ` R T D 8 W W s I UuB~SEY9XES$sSX3u)`XVYbSEBY)VV8isYW~8GCcgYm)YV&VY))VT 8 I P PB B ` q a s B a H PB W B P s g a R B p HB 8 W s 8 I P R q PB T s T 8 I Yl$RAEE)hf)WmYb8 SEYhYFRd$cSX3Wm9XEDVY)W)ITYlwhfEV$s bYC p
}

o o p

~) ED hCp R o o R ED g Cp o

p Xuqc p r p s F uqc RD Xr p tp o o a a dW r 3 X 1 |D R g q H a q PB T s T W RB P s g a R WB 8 I P WB a 8 `B H W R T W 8 I 3)i)Sb8 EV$s b1XES$cSYsmlYYmfb)ESYXVqbYC

m i ! m i ! m i ! m i m i ! m i ! m i ! i !

o ) o X VX p | W RB P s jBD s g a R W q BDB s R ` 8 P sB T R H H s 8 I P 8 x s I 8 7 p XES$bE$cSXYfFPEEY)Xza VS$mVXdS$Y)i9V VX W B P T W e q B H W 8 ` q P DB s R B p q BDB s R I PB @ YFRdVY)UfFPdYb)hfEFBEY)Yfa 8 3r X { GX 9fFPEEY)Xza EC&r VV Y o Y h r X p X VYm$yx XV$UVFTY@ EYmfyX)YSAib)F`dYXVF8D VYg{ H 8 eD s W R H 8 v s P I B I 8D sB a s x g R ` W s a s a 8 B H W R T r gs y8 a R SwbCSVfbSE)CYXf!SVVYfa YmSEimE$GSEYVfCYim$YXcYif1F`YYVC97 8 H s T 8 P 8 a T HB ` 8 I P g R a H H 8 T R WB PB gBD s H s H PD e H 8 a 8 I P WB s P R ` W s q a P D e R T 8 W RB P e B a P HB ` q PBDB s R H e R e WB P W R T s q W 8 xB XESY)mfSE)2fEEEY)Xfa YYYXmSXViAbE 8 a s H W B P T W e 8 H 8 I P R H B BDB s R ` 8 P sB T R H H s 8 I P ` W s r H W RB P T W e H e R e WB P W R f2YYFRdVY)qSVYl$AVF8dFPEEY)Xza Vd$mVXSd$YQYYXESVY)1YYYXmSXVT 8 a s PB g H W s a P R P I HB @ 8 @ HD R g q H 8 I P I P R B P s I P 8D sB a s x qD H e R e WB P W R T 8 a fcEiYS9SEC11E)i)SYt)FHi$)EYmfyESYXYYmSYVfs a 8 H W s a P R P I HB @ 8 @ H 8 s H H 8 g a R HD s W B H 8 I P I TB I @ WB 8 H s T 8 I P a 8 B H W R T @ R W 8 bszYScdECGCV$dViCXE$)XESYqEYqud$bYb)F`dYXV$Yl97
vD5dv p

H I t l m o d{gCf p s m | H 8B a 8 H D sB g R WB 8 I P bVEfbS$miXYmYlY$R a 8 ` WB s g 8 a 8 I P H s a R a a 8 D s e `B H 8 a R q PBDB s R 8 I P WB s P R 8 @ WB s v T RD PB b)YuwcbfY$&YfSb$)gESVf$cfEEEY)Xfa Yim$YYGim$X e VYVXEYCEYlp a H a R a a 8 B 8 a R g a R R @ P R q PBDB B H H 8 I P @ RDD s 8 @ B ` 8 P e B a P HB ` qD g a R B W b8 zXfSbFPYCfYiXGf$cfEEEYudSR Yh$EE$G$8 3x9VSY)mfSE)cEiSYsY)e H a R a a 8 PB g R ` W s a 8 x s I 8 @ 8 a 8 I @ r 8 H s T D s a 8 W 8 8 a R g 8 I P a 8 `B H W R T W 8 I P W s T 8 fXfdbEYX)YSl)lGlzbY)S$b"$SbYbfYiYb)ESYYVbY1b~97

8 P s a a R a a 8 D s e `B H 8 a 8 ` R T WB g g s p | 8 a e B SwSGXfSb$)gESVfi)XVcYmiGXw3Cg$if)XE{
1e-20 1e-08 Residual error rate 1e-10 1 1e-07 1e-06 1e-05 0.0001 0.001 Mean error rate 0.01 0.1

H a R a a 8 PB R @ P R e H P T 8 a a R T I TB I @ P s H P HB y 8 8 ` R T ` 8 g s W W e W s r H a R a a bzXfSbEYGfcSP CSVVzSXVVEYp 3$dSEXV~)XVAVi))))bfYfSb8 FPYVf)9dP SVVfSXVEY)YVVXEY~bo 3h2E2)YVmYdFHD e B 8 8 a I P R e H P T 8 a a R T I TB I @ 8 ` R T v T RD s HB 8 ` R T q sD R } 8 I P R H a R a a 8 PB 8 W R P T 8 P 8 ` ` W s bfXzSbCEYlYXCVVSV)Y l t 3fm$s E$GX3GSEYVCV)YVcYmiclYC o H aB DD s a R P HB y 8 H 8 ` R T WB g g s 8 I

  X3CYYFRdY)uzSE)Ye mEYmSSR E$Gb$R a R H W B P e B a P HB ` P WB 8D B H H DD s a 8 x W RB P s g a R WB D s e P e g HB I P WB jB gB y s g q W 8 xB D 8 W W s I T 8 I P R q PB T s T 8 I P I PB XES$SX3u"$)YiEYYmVimY$cAbEVY))Y$fEV$s bYqEC@

c X ~ X

cr tX }

RD c E dX

f { 

|D s a 8 P WB 8D e R ` 8 I P WB H 3$dXVSuiEY)X)Y1YmSYe ` 8 W n 8 ` HB f ` W VY)GEhYs VEYmfyx X)YSGYXYYudXV$pXES$cSYsm)$)YigEE$)m{ H 8D sB a s g R ` W s a H e R e WB P W R T R W RB P s g a R WB D s e P e g r qDD s WB h

c X | I PB EC@

~ q } E) RD r ~ hq } E) Q r q RD ~ q } E) Q { q RD o | H W RB P e B a YXESY)mfP B ` H e R e WB P W R T a R H 8 R a P W 8 D s W RB PB ` W R T ` W s P WB R v 8 I P 8 W n 8 ` R D s W 8 I P W s T 8 zFH)YXYYmSYVGX3GVEB fb$)XESE)YXVYuYiY)cdFHw2bY1b~97 l ~ } E) q X$3 RD o | q P `D R ` W s W RB P s P R W a R P T 8 x ` WB I 8 D s a 8 P WB `D R @ 8 I 8 f2EY1XES$YCXSVVYmYb$SXVdm2E33YP WBD s 8 T W R T @ W RB H W 8 gB ` R a R P T 8 x g R ` W s a H e R e WB P W R T s R P HB I P ` W 8 P y 8 W s T 8 YmE$bVYXVUXESYbpu)G$XSVVX)YSYXYYmSXVc2dlEYYbSYVbi97 l b~ X 3 } E) X R D 1q W X o |  R 8 a e H s 8 g s H s qD W g a 8 $vdf3&x3g&f t$z)$bic$EYXR SbSP PSzYYmSYtbEYXfa YcY)~V)fFPVws bCYq zb"$)Yi$R H a n 8 I P WB H e q q s @ s g 8D R 8 I P 8 W n 8 ` 8 @ r q B T s T a R R a P W 8 D s e P e g W B P s a 8 `B H W R T 8 I P WB B H 8 R a P W W 8 8 @ P 8 H 8 T W 8 a 8 B ` 8 I P WB ` 8 P H 8 a 8 P WB W 8 P YFRd$Sb)ESYXVcY9m8 ssqVEB zb8 bVfVpVVYbzb)1YUuUVSSVfbSmbS3$R 8 a s 8 @ H s r a 8 x 8 @ R P HB y 8 P R W H 8 R ` B gBD a 8 P P sD 8 I P H s WB q a a R @ a 8 I P s a HB HB I fiG$bVGyibSEXVYlVX)~FPpuEbSd$mY$YmYfSXGbYwSlElEYC

3
X

X  ED k mFB1 q R g D W l l & ~ ~ hD X X X q ED k mFB~ X R g D } } RD E) X o p p & ~ p X q E9 X GX } E GX 5k uED RD p gB RD sX X o p & ~ p X GX k uED 1q } ED X GX R p gB W o | PB g D 8 I P WB v s P ` W s R a P W 8 8 P 8 a T HB ` a R sD e g a R a e R WB H bEimFBYcYmYq fb~SVfbSE)X3~mYdXs)XYmS

&

  a 8 W W s g H e RB x R 8 I P WB H W RB H W 8 gB ` 8 B PD e g R P H ` W 8 P y 8 HB I bY)cYXEYYYAuYXESYbim)iED SEY1S)YbdXVCEYC R a P W 8 D sB P W 8 a 8 B ` 8 I q zb"$mSbzb)YP a 8 P 8 a I P r a 8 @ 8 I P a 8 P s 8 a 8 I P BBB 8 T W 8 I r W s 8 g 8 I P R P W 8 ` W 8 ` WB bS$sbfYl8Y)b$R YGbd$bfX~YmEEVYbYbiY&$lb)Yb8 )YmFHB ` W s T W B a s 8 I P W R qD W R P W 8 ` W 8 ` HB R a P W 8 D sB P W 8 a 8 B ` 8 I P BB r Y8VYsmf$xcYAX9EYXib)Yb8 )lEAq zb$mSbfb)Y~uEV f r 8 sB a s x I B f D sB a s g R ` W s a q W s a R B | P s I P 8 x a 8 H R 8 iAVTYWmf$2FP@ yF8Yuzyx X)YS2X3GmGb$)lfbdYXl97 3 3FRD o |8T W8 VYbYI p X Gl  l o 7 Y | g 8 P H q H 8 I P W R H P WB s a P H W R T 8 I P a R ` a s 8 a 8 e ` I PB @ P s I P R bSSYSY1XSm$dSYXVYtXs1zXVfiYg1ECC$)iSH 

RD o " X E9C | WB xD R H q ` 8 WB s P R B I TB I gYmXES1VYm$YXFHqEY@

7 X E X b X E" X FR X b D X | WB B g gYmVjpuB y s g ` W s H a B B PD e g 8 W s a s WB H e ` 8 xD R H HB g 8D R W RB P s jB gB R HB I $cYzbF8ED SEYiYdX$AYmSY2VESGEbEYXfa XES$bimSP EYC

3
X

X
X

e X X

| 8a8 I fbY@ X X X Q

| H P WB s a P H W R T 8 I P I B @ bSm$SdYXVYFPXr F8YuzyY)YS$)XESYbim)!1YGb)ESYXGh D sB a s x g R ` W s a D s W RB H W 8 gB ` o 8 I P a 8 `B H W R a8 @ Ws8 g `8 BT H b$R bi9VgnVV8 is a R R a P W 8 D sB P W 8 a 8 B ` g e gB y s g 8 I P ` W n R P H g 8D R 8 I P R P W 8 g 8 P s P X3cq fb$mSbfb))imX$lY2Y~SEB bEYXfa Y$GbpbS$SH fA2VYbCVbG$R $dbwf)$bi2cE1VYmfy1Y~$cFHYcSYYT 8 a s 8 T W 8 a 8 @ 8 s a 8 x s R 8 a e H s 8 g s HB 8 T W sB a s x 8 I P H s B I P 8 H R R I 8 @ 8 T W sB a s ` 8 B T H a R GhVYmf$x VgnVV8 cYsq zbwudbfb))imX$g Y2Ew)I G R a P W 8 D sB P W 8 a 8 B ` g e gB y s 8 I P HB P s 7 c )" X B& Y"X c o W8 I @ `8 B g g5 X ibYVVjpuB y$cE W 8 I P 8 g e D R x 8 I P W B I P B @ R a 8 j W R W q D W R H B B 8 g e D R x 8 g R s g HB X 1bYGbi)mYmYEC1fbV9XYEYXCE)8 3xi)miXSH R P ` 8 PB gBD H SVSEimEEB X sgE)i)SSEY)Xfa Y)VECQVVimX$E9q f&Co h HD R g q H g 8D s R B e w 8 I PB @ ` 8 jB gB y s g HB RaP W j H W B H W 8 gB ` @ R 8 bYYFRdYbim)2$~) R P W 8 v s P 8 D e R I H T P 8 8D sB a s x r W RB P e B a P HB ` I T s 8 P e r q P `D R 8 I R a ` 8 SAb$1)tF`YYYcSV1EYmfyXESY)mfSE)wbY)8 fEcYP fgiG@ WB @ RDD R 8 I P 8 H s T 8 P 8 a T HB ` 8 I P R P H e R sD s W s H 8B P a R H e RB a s x WB s P R 8 YuyEE3iYW SwbSVzbSE)cYSYX$m$)cVESfb8 fa YXEfym$YYc97 #  ( o  g( wwgCpc

 $ H eB ` s 8 a 8 H @ 8 I P 8 `B H P e R R a 8 j HB I TB I @ W RB P e B a P HB ` q PBDB s R Ym)$da zbYI !Y)ESdYXfbV1EQEYXESY)mfSE)QEEEY)Xfa D W RB H W 8 gB ` @ W s q ` 8 P W 8 H 8 8 a HB 8D g 8 H W 8 8 I P P s I P ` W n 8 @ r $s)XESYbpu)!VSbSVfa fE9EYbdYbY$)YAVS 3 D$$fabSm1uB1EYXcfbV9YY3E$)XESAVSEimFB&pudY SxhVSEimEYY) s x 8 P W B W q D W R R a 8 j W R W H D s W B H ` 8 P B g D 8 g B P ` W s R P ` 8 PB gBD ` W s r V 3abG$R `VSEimFB$2EYibSYb1Yib)ESYXVGCYEFDwbgn VV8 cfYG q 8 @ 8 PB g D R 8D g 8 H W 8 8 I P a 8 `B H W R T 8 @ B r q D s T B T H 8 a R u X X YSbbYVX W RB P B a P HB q PB B R @ D s W B H W 8 gB ` @ W s q ` 8 P W 8 H 8 8 a 8 R XESYe)mfSE)`fEEFDYs)Xfa (q1q$)YFRdYbim)!)1VSbSVfa z)cSP 8D g H W 8 8 I a 8 `B W R W s 8 ` W s H P W 8B T 8 R T R a 8 j W R EYib8SYbYPb)ESHYXVTtbT1G@hY1SbEVYV9fbV9YYW (rEYXtm$P qD W R WB s Y9G@bY~rl$$fabSuBUigmSPSP5VSEimEYYFRdVY)Ab)ESYYVhSE$Gcx R 8 W 8 I P D s x 8 P W 8 B R ` 8 PB gBD H W B P T W e a 8 `B H W R T R HD s 8 @ X }
~

| P s I P g 8 a R 8 I P WB g s H 8 I P g R a a 8 g 8 g 8 a 8 @ 8 H a e R T R 8 a 8 I $)bfYVYYmED YXf!bbpbflGSf)XV$fbY@ T WB !T3)YmSH X k Qd X | H W RB P T W e ` 8 PB g bYXESVY)!AVSEimFBD YY)$iEYibSYblYl! Qd VEYmf$x X)YSCX3)lGEE$SbYbfYG ` W s R 8D g 8 H W 8 8 I P k X H 8D sB a s g R ` W s a a R 8 x s I 8 @ r qDD s a 8 W 8 8 a R u bG$R a8 @ 8$Sb1EC1Yfb SVSEimEYY)F8YibSYbi!SEYSEYSyYGFHB s a 8 x s I PB @ ` W s j P a 8 R P ` 8 P B g B D ` W s r D g 8 H W 8 h 8 H B R W 8 P B I @ g 8 I P T3)YudH  ! d T WB k W RB P sB x 8 ` ` a s ` W s P | W8 IP bY9 XES$mYV)zgYSH IECYFRdY)uzSE)mSdYl9tSYm)fXVV$9Ym$yx X)YStXcV$ PB @ W B P e B a P HB ` W sB H H e s } s R P WB ` a R T T s 8 eD s g R ` W s a s W R H 8 v s P IV$bfbY9k Vbb Y o d Yf d VEYmfyY)YS$VSUb)ESYXGC Ts8 8a8 I @ | H 8D sB a s x g R ` W s a R P 8 H s a 8 `B H W R h WB q 8 v P B I H 8 H s a 8 `B H W R T H e R e WB P W R T a R 8 P 8 a T B gYmYV$3 YlS$)I b)ESYXVYXYYmSYV&YlSVfbdFH)` 8 q s g 8 a 8 I P s I P 8 P R 8D g 8 H W 8 W s 8 x s I 8 @ r q s H r W RB P e B a P B )9cfbY2$)cSC9EYibSYb)GV$qXESY)mfdFH)` q P DB s R s I B @ a 8 I P 8 R P ` W s r W RB P T W e P W 8 a 8 B ` s H 8 W n 8 ` R 8 eD s x I T s fEFBEY)Yfa lFPbYVdYXESVY)!bzb)G&VY)t$Ym$yV$&j WB H $)ud
q

| H 8D g 8 H W 8 8 I P a 8 `B H W R T r 8 g s y 8 a R bVEYibSYblYb)ESYXV)ED VGYg{ $ % H s W @ R W v HB v a R $dXiY!qiw$YXEiSXG@ R s 8 a s D s a 8 W 8 HB I P % R s 8 B 8 I P R P H e H WB a I TB I @ D 8 W W s I T 8 I P R P W $bfwSbYbCEYc!E!AD&$pb)F`YdYYmf)EY"VY))YdmB ` 8 P T 8 v WB WB 8 W B P T W e W 8 xB s R q PBDB s R 8 I P 8 ` eD T WB W 8 I P R HD s P H e g 8 VdVVDm9YmV)YFRdVY) bEA$fEEEY)Xfa Y2)YmVYmUbYSE$SYi97 8Hs S$bT 8 P 8 a T HB ` 8 I P R H P 8 s D s P P e R ` W s P WB 8 I P R 8 T s 8 I P 8 v s P W 8 I P H W RB P T W e dVfbSE)Y$SV))I $CYe YYYYe mY$~V$mD Y$1bYCYXESVY)! 8 H 8 I P P e R 8 I P P s ` 8 P T 8 P 8 ` WB 8 H W RB P T W e 8 I P R W RB H a 8 x ` 8 a e P a 8 g R SVYCcYe YXcYwVSVVdV)2YmV)cYXESVY)!iYG$XESfbVS)fb8 pXSH I PB @ rD 8 W W s I T a e R R P P WB H s r H W B P T W e 8D B H H R P 8 H s g R a W 8 H R I T r 8 gB P q s EC)3VY))V)YlSGYe m&$)bYYFRdVY)EYmSSR $&VSr5Xf!ibdYVbimSiH R H W B P T W e WB ` 8 P H 8 a 8 P WB 8 a s 8 @ HD 8 W W s I T ` W s HD s W B H H e R e WB P W R T R 8 H s T 8 I P W $srYYFRdVY)mcVSdVfbSufGEVY))VYtE$)YFd&YXYYmSYVwSwbCY! # U  (# g( B$&i n

 
k ~ &gbED p 3 o } R o k 3 GED R 3 o k GED R f c o | H s r q PB T s T D 8 W W s I T 8 I P 8 P s eD s x 8 W 8 I P W s T 8 b$gfEV$s bVY))VYlSw)uwyV1bYqb97 k VYmfyC$)I 1Ys VYmfyx 8 T W sB a s x H s f ` W 8 T W sB a s Hs $)I mYbESfs W bYXESY)mfSE)qmSSY~)SYig Ys )9YG@ a sD e TB P a H W RB P e B a P HB ` W sB H H e s } 8 x s I P H e f ` W IPR ` W n 8 8 T W 8 I r W RB P e B a P HB ` W sB H H e s } s 8 x s I 8 @ W 8 I @ ` 8 x 8B I T s HB B I P W 8 8 H 8 x s I 8 @ VYbYXESY)mfSE)qmSSY~c)lGAbYVVEYwEFHY2bVd)GH e x p w | W RB P s PB gBD a 8 @ 8 s a 8 x s 8 I P R P P T 8 v e XES$EimEGbG$R $SblYSCVVfs)H $R W RB P B a P B ` 8 I R P P T H 8 a I PB @ WB jB gB y s g q ` 8 x 8B I T s HB HB I P H W RB P e B a P HB XESYe)mfSHE)iYPS~VV8 Vz9EC9YmVimXwcVVEYV$EiEY2~YXESY)mfSE)` PYe uB"FDwabyFHBYYmVimXwc2VYm$YX1bYCEVY))VY$1fFPVws bYC W D s 8 x R I P WB jB gB y s g q ` 8 WB s P R W 8 I P HB D 8 W W s I T 8 I P R q B T s T 8 I f f Tc{  |8T W8 VYb k R ED 3 r f o |Ym$$xCSEB2YnWXES$s))V8Xf!1YgVz)XSldFHYlY1Xb)Yb8 )` 8 eD s H P ` W RB P e w g R a ` W s r 8 T a e R H 8 B R W 8 I P W R P W 8 ` W 8 FqVEDdYmV)cH$9q fb8C$s)YFRBdFP)`YYVTYIPtm$YX1G@ $b)Yb8 )YmcEB pH e D 8 R H WB 8 s R a P W D W B W R 8 WB s P R 8 R P W 8 ` W 8 ` WB H q PB H W 8 ` D s a P T H a 8 @ 8 I P H 8 a 8 I fESYb)$SVV8 GbG$R YEB k lfbY@ k  | 8 T W sB a s VYmfyx ` W s W s 8 g R a 8 j I B @ W RB P e B a P HB ` W sB H H e s } ` 8 PB g D ` W s 8 I g R a H 8 a 8 I Ybi2zbV|FPXESY)mfSE)AmSSYlVSEimFBCYY)cYP Xf!EB p cfbY@ @ Vb 5cGr p h p p f o | q W 8 xB HB P P e R ` W s P WB 8 I bEECYe YX2YYe mYP W 8 8 @ P 8 B I H W RB P sD 8 a 8 I P W 8 I HD s W B H P P e R ` W s P WB 8 I P I P R 8 B a 8 P T s a s I bVGfV) YYXES$mVflY1bYbE$)YFdYe YXYYe uY1)ldFHzbSV$S)VT R P ` 8 H e 8 W s T P s I P f ` W p H 8 g s H @ 8 I P a 8 `B H W R T W 8 I P 8 SVSY9)Qbw)i p EYs h VED 5Y1b)ESYXVQbYA97 R P ` 8 PB gB hSVSEimED I P B @ ` W s ` W R P ` 8 PB gBD 8 gB P )F`)Y)Ys )dVSEimEppudCr bG$R $dbFP$)XESU)997 a 8 @ 8 s a 8 x s I B @ D s W B H s 8 x s I 8 8D g 8 H W 8 R H H sD T D s W n 8 I P R P H W RB P s a 8 `B H W R T P B a P H 8 a 8 @ a 8 I P a e EYibSYb$S$mV$)YcSYXES$Sb)ESYXVVFTzSVflGGbYz){ bVIgb VdCy3 Y5b3 VEE$b$SY&E&dVfbSu$CVf)YSCSEYmYbFTdfs $ ` 8 D D s T R H 8 I P H B P H 8 a 8 P W B R 8 T a e R H 8 H B R W a s D e B P a 8 I P D s W B H 8 I P R qD P W 8 ` W 8 ` WB ` 8 P T 8 v WB HB 8 HB R W I TB I @ WB D 8 W W s I T a 8 `B H W R T 8 Y)3$)YFdCY$ESb)Yb8 )YmVSVVfDuGESEYVEY1mGFHVY))Gb)ESYYVC97

6 '  o  U (  4 V3)v~V&

xn

W B P T 8 H WB W 8 B H W RB PB W n 8 ` R a P W 8 D sB P W 8 a 8 B ` H e R e WB P W R T D s W RB H W 8 gB )YFRdVVSmbFxYXESEY)q fb$mSbfb)YYYXmSXV$)XESYbim)` 8 B W n 8 I P R P WB g 8 I P P W s T 8 @ r H D g 8 H W 8 R H q P a 8 P P sD 8 I P WB a 8 `B H W R T q dFPY1YdmbYcgn bG&VF8YibSYb$cV8 f~bSS$m1YYmfb)ESYYVf

V e G3dVVVT e ED SEYe AXESEYEl)YGCmYbESfs u i h HH 8 T 8 B PD u W RB HB xB i 8 ` R h a sD e TB P a WB ` W s W RB P sD e ` R g R H 8 e wB W I T 8 g e a P T H ` s 8 H a R HB H s 8 I P HB HB I mYQXES$mYgYirwVY)EY)VSP )SVV8 $bfa iYsiESw)2YEcEYC h 8 B R W 8 vBD H v R D g 8 T a e R H 8 I P W 8 I @ ` 8 x 8B I T s B D 8 W W s I T 8 I P R q B T s T 8 I g!dFHY$mEXYFRdVz)XSGY~bYiVVEYw&FH)VY))GYg$~fFPVws b&YCC PB gB W R W W s 8 I P ` 8DD s T HB HB I bEimXY))I YVEE$bECEYC  l FRD k | W 8 I P B 8 eD s x WB B gBD 8 I bYFH~Ym$yAYudFPpuEYCq q ` 8 W n 8 ` HB I B I @ r r  # R H g a 8 P WB W 8 P PB a @ 8 a W 8 P R HB HB I VY)E1VFTY)(3xVb)yub&C$GiSbSqu1bdSEfVf1bS3$ECEYCC
ED k R | WB s P R 8 m$YY@ C C &DC o W BfuD | WD a R W RB H W y 8  a RD q s WB H PB gBD s R P P e "utXsXESYs VGH !XEYbYudCCbEimE~SCY) H 8 H s 8 a T WB I P `B @ ` W s 8 I P H s qDD s TB W R P R W R g H 8 H s 8 a T WB q B T s T 8 I P D a s B g Vd$bfbYm1)EC)Y)lYG$EE$bEYXSYXpVS$bfbYm1fFPVws bYFqzmFDpuB

H 8 H s 8 a T WB bVSwbfbYm YG$s 8 IP H ` W e R P e R I PB @ ` W s qDD s TB W R P R W R g H 8 H s 8 a T WB q PB T s T 8 I P P s I P 8 x a 8 H YYXYXYECtYEE$bEYXSYXicVS$bfbYm9fEV$s bYl$)cfbSYCo RB P s a 8 HB R W R P D s W B H 8 I P B W RB P s e w 8 WB RD 8 I P WB I PB @ g a 8 P ` W R T 8 H 8 I V( cES$SCSEYS)$)YFdYFHiXES$))V~mEYimYECdbSiYXVVSGYC V 8a8 I fbY@ x"m



H PB $YEY

ab8@$R VSSEiYd$SbYCEB ` 8 P PB g H W s a P 8 s a 8 x s 8 I P H g} E o RD
~ k

| q W 8 xB H R P ` 8 PB g bEEB ASVSEimFBD Y`Y)cYs q P B H W 8 ` D s a P T H a 8 @ R 8 H B R W W s B H H e s } 8 P B I @ 8 x B P B ` ` Ws ` W k fESYb))$dVV8 "bG$R $CdFYlmSSYlSEYGESE))$s q `8 a ePa r jPa8 VS)fb8 fb SQVSEimE)Y)CVY))q$9fEV$s bcYC R P ` 8 PB gBD ` W s D 8 W W s I T s R q PB T s T 8 I | % $ # $ 8 I P r g 8 a R 8 I P ` aB I P  W R W W s i&fX2"DgGX!f&!gAYbfXVYzuYH YY))I R H PB $YEY
~ k

o
}

E RD

| q W B P s T B PD e g r @ g R a B r @ q W RB P s B B PD e g q 2YFRd$bEFBD SEYiglXzY8 slqu9XESwbFTED SEYi r ` W R T 8 H a 8 P s a s WB s P R W s T 8 @ R H rD R g q H D 8 W W s I T a H PB WB HB q PB T s T HB I YXVVd&b8 S$dl1uwYXib~SY3)i)SVY))b8 SEYm&EcfEV$s bGEYC

1. Foundations: Probability, Uncertainty, and Information 2. Entropies Dened, and Why they are Measures of Information 3. Source Coding Theorem; Prex, Variable-, and Fixed-Length Codes 4. Channel Types, Properties, Noise, and Channel Capacity 5. Continuous Information; Density; Noisy Channel Coding Theorem 6. Fourier Series, Convergence, Orthogonal Representation 7. Useful Fourier Theorems; Transform Pairs; Sampling; Aliasing 8. Discrete Fourier Transform. Fast Fourier Transform Algorithms 9. The Quantized Degrees-of-Freedom in a Continuous Signal 10. Gabor-Heisenberg-Weyl Uncertainty Relation. Optimal Logons 11. Kolmogorov Complexity and Minimal Description Length.

Jean Baptiste Joseph Fourier (1768 - 1830)

37

Introduction to Fourier Analysis, Synthesis, and Transforms

It has been said that the most remarkable and far-reaching relationship in all of mathematics is the simple Euler Relation, ei + 1 = 0 (1) which contains the ve most important mathematical constants, as well as harmonic analysis. This simple equation unies the four main branches of mathematics: {0,1} represent arithmetic, represents geometry, i represents algebra, and e = 2.718... represents analysis, since one way 1 to dene e is to compute the limit of (1 + n )n as n . Fourier analysis is about the representation of functions (or of data, signals, systems, ...) in terms of such complex exponentials. (Almost) any function f (x) can be represented perfectly as a linear combination of basis functions: f (x) =
k

ck k (x)

(2)

where many possible choices are available for the expansion basis functions k (x). In the case of Fourier expansions in one dimension, the basis functions are the complex exponentials: k (x) = exp(ik x) (3) where the complex constant i = 1. A complex exponential contains both a real part and an imaginary part, both of which are simple (real-valued) harmonic functions: exp(i) = cos() + i sin() (4) which you can easily conrm by using the power-series denitions for the transcendental functions exp, cos, and sin: 2 3 n + + + + , (5) exp() = 1 + + 1! 2! 3! n! cos() = 1 sin() = 2 4 6 + + , 2! 4! 6! 3 5 7 + + , 3! 5! 7! (6) (7)

Fourier Analysis computes the complex coecients ck that yield an expansion of some function f (x) in terms of complex exponentials:
n

f (x) =
k=n

ck exp(ik x)

(8)

where the parameter k corresponds to frequency and n species the number of terms (which may be nite or innite) used in the expansion. Each Fourier coecient ck in f (x) is computed as the (inner product) projection of the function f (x) onto one complex exponential exp(ik x) associated with that coecient: 1 +T /2 ck = f (x) exp(ik x)dx (9) T T /2 where the integral is taken over one period (T ) of the function if it is periodic, or from to + if it is aperiodic. (An aperiodic function is regarded as a periodic one whose period is ). For periodic functions the frequencies k used are just all multiples of the repetition frequency; for aperiodic functions, all frequencies must be used. Note that the computed Fourier coecients ck are complex-valued.
38

W8tG A5 c yUC

i thf

R{ ii if 97 { i g if 7 3 Y { x g&huT5|zp&hQtUthf p hf ig  x k i g d 9 Y 5 Sa G E 3a 7 9E C E 9E S 3 A Cc G Y 5 SEc C c q i g Uphf$56`3cU8Db}6h}U`CF'8'`CFD6$U@R'8D88Uphf R{ i g if 9 i&hu5T7 { p&hQtUY { hf p hf i g if 7 3 x ig  x k7 vECc C 973Y A 5 G C 97 a3 v S7 5 G C f i g ''8T5tUW88T5RDtD4phf pt&|RW&xtp|Dt|

r7 vECc C 97 A 5G C 973Y a3 7 vEC '85@W8t85@tU0R}w'c T54iphfdW56`39UY85DaC8zcR6`385DUtUD`A6RFD!U`CF'8'`CFD6p8}RDD68e 9 g c S a3 5 97 G V C 5 G 7C 9 P3E V 7 9EC E 9E S 3 C c cG c C V 3 C p i 6f i g8&hT5qhd XWl d z ri g if 97 i gf om { p ithf v i 8r&hf'tUqhd Fm l d x g i g i 7 3 Y i gf o { k CE G 7c 5 9Y C 3 2 E 9E S 3 C c CEC tFtt'`CUgz84}'`CFD8!8zF'8y xQ{ i thf i i g i 9 7 { i f Y { x pbf05T|zxg&ihQ7t3Utbf p x k 9 i gf d E 3a 7 9E C E 9E S 3 C c 5 C '`74ph46jU`CF'8('BCXD68'8 C 9 EC c `7}X'8t3 v { y i h s |zw k 5 G C v 3c 5 9cY 5 Sa w E C YC 5 3E C c C7 S C &t'g@$6`3U8D}'RUU86F!88!e p i 'Ff i~ h  v 6@$h'tUq&h0T57 Xm l g ri g hf 7 3 Y i g if 9 o C 9 EC c { w `7}X'8t3 |y xd s 6@$hT5q&hT57 p g r i g h f 9 7 i g i f 9 Xm l o v i h $uta9 v C 9 EC c { yw `7}X'8t3 |zxd s 6@$h'tUq&hQtUY p g r i g h f 7 3 Y i g i f 7 3 Xnl om v i h $uta9 d k i A5 uW8tG h 7E C Cc 9 E 3a i g if 7 3 Y A 5 G i g hf 97 a 3 7 9cE C V 3E V H G 5 3 3 cE 3 C c C PE C7 f 3 C yF'UtT56jpbQtU$8t4$b0T50R(UBC@F'D6XDgFBc`9RD8tt8F8y8tF'@86e r d x C Gc C H GE C 5 C a 3 77 H c S 3 9 r i gf t!Rt4FBc`9R@'8'tIyB3(868`c0hd i x gf d q g A 5 G x C v 37 E 3a r r9 q i gf d 7 5 9c Y 5 Sa 9 A 9E C V A C H G P H G CE E C 97 5 3 8W4hD8tyw6u6hutsChRUphe88B3@U8Db`YD8B3X'DWU8TSRQDI'F4'DBA@8642  ) %    "     10('&!$#!

v 6 r v i d if d 7 C 9 7 9E C7 c S D &hyUt`PtU`CF'}8Df d i g f '' i g f 9 i g f 9 i gf 9 pth0T57 thuT57 b5@7 hd k3 7 5'UCi v s g r i g i f 9 p d { ~ v { CP A DA83i o { #g8pb5@7 o l z}Dtx k 5C &'8c d g s i gf td hd d g v ibfdt8y`7y`c8UX!8D'UtX'tt8UyU`CF'8z6868'tU4q g c 3 5 9 H S7CE C c q7C EC P 5 3Y 7 9EC7 C c S 3 c 5C PC r q EC P 5 3Y c 3 5 7C 3 A 7 9EC7 C c qc 9 C { q { x S 3 HG r tCtF't6U}t8U6DU`CF'8p'`78UzD$6868`cR}q HRD63`R@TE$g v s i g f G 5 9cGE 9 HRD56`3R@W hd G 9cGE g E 63 g i gf bd kst'@B78UxBHIb8D86`3RDDU$T5}`7R@6U54 r r C r c 9 C 3c 9 Ga q 7 5 9c G S C 9 H GE Cc 9 r k7 v H f 3E V C v 37 ECc 5 S 3Y 5C G ''`C86XD!6('D6U8'v C EC PC 3 r C P3 f G 7G AC 5 C A CE G 7c 5 9Y C 3Y E 9E S 3 A 5 G 7 9EC E 9E S 3 C 4'tU4R}ttRqt!RU8zDFt'`CUgz6U'`CFD68tUBCX'8!'`CFD6p8 |t8||Ixpt b d 'Dth'p6Uut`8'F6$tgz'8Ut@t6h4tFz'Ft8'U r { FheDthtRRFtz($6TU'z6t6uWpz { e8tG A5 { x { x 5 C 9 5 G P A 5 G v H G C E G A 5 G x 9 H 3 P 9 7 v E C c C c c G c C c t e'8y`78tQ8t$y`HRFtz8e858`PttT5y@'8RDt3 4xQ{ i thf d x xf d i{ Rm 6ze f i { x x { h m 6 { hf m i p e p h|  xR{ i m{ m{ hf x d xd mp  p g i gf o 6r m zphd Fm l k 3c 7C H V 97 9 p!Ut`98Tv$U`Y8y p g i g i gf o 6r m 'hf { ephd Fm l kic 9 C 3c HGE Cc 9 C `76UxR6UT58c E 3a S 3 5 C A C P G C f H C 9 i gf d 9 v S77 Gf i g A 5 G i gf d 5 C C c C f Y 5 G V CE Y 9 6h6888'UtD'D&BHU4B7uph8T5D@Rh}bf { g8th'U4FUDU8tD'F'`7DA C c 9 9 HGE Cc 9 C c EC 97 5 3 2 r7c 5 9Y C 3Y E 9E S 3 C c CE A 5 x CEC 8858`PtR8UtT58'DBA@864''`CUgz8U'BCXD68FtG { 8tG { F'8y

 5 3 5C v 3 5C V 7 f 9 k CE S 9 88'68'8Dy8DTfD!FD6`
-1.5 -0.2 -0.5 -1.5 -0.5 0.5 1.5 0.5 1.5 -1 -1 0 1 0 1 -1 -0.15 0 -0.1 1 -0.05 2 0 3 0.05 4 0.1 5 0.15 6 0.2 7

C H S V E H S 5 GcYCE 3c 7 5 9cG 9 3E V V k CE S 9 `78D}tTG868tzUUFg}88B3@RTv6RFDDtXD6`

fseries( 63,x) fseries(163,x) rect(x) fseries( 1,x) fseries( 7,x) fseries(63,x) rect(x)

 6
si x v t YRW x v t e E Q x v t YRW y x v t h V h q e h g f e G V I ER aR a IR YRW V G T E GR I Q I G E D wBcSX7BcrH'6ibBcwSXb@@Bwuds6rpi6dXcSbS`SX@'BU'S9PHFC 

iv t'Ff

{ F
1 ) 8 8 8 9 

A B @

p Q{ i f & d 
5 3 6 3 7 ) 5 

kcG c Y S7 7c 5 9Y C 3Y A 5 3c E 'RD8{ '`CUgz8UW8Fc

5 3 64 1 2

ACY G V7 H G S C CcCEY 9 UUID`BHIDUUF'`7DA

i 'u ~F tD q t $ f ~~v g d k g a3 7C HG |RU8SIQP c G i gf d 5 9c Y 5 Sa 9 A 9E C V G a 3 C H G P C c E C 97 5 3 Rph6`3U8D`YD6`3F'qeIn8TSRR$8'DBA@8642

) 0

Ip}tpIxxtp|Dt|


i thf

'

$ 

$ 

 &  

CEC F'8y

i tf
% 

! # ! "

p g 6r { hd Xm l t { i g f o d k A 5G qCcG S 5 3Y H V v 3Y C c 7Cc 3 5C 8tDtR68 6U$U`C86U8}U8'DA { { ttF'863 k C PEC7 f i { {x 6z thf { i { { hf { i x v t p x p

   

k 9 &`c}

i tf
  

i gf p x hd

{ { xR{ 9 R { { i { { f z i { { f x ii g if 97 { i g if 7 3 Y { x pb0T5|zx&hQtUtbf
      

px xQ{

k 3 '7tUYWA85G$5@74E6haCRTGH8S@E6jaC8zcv63FaG}}78S6`396Pf63C8T59 { 8T58@BcXyUC 9 3 v 3 E c 97 S C 9E tq|g

r i 7 G c7 Ga 7 G c7 G H c G 3E C 3c A 5 Cc { A 5 G { x r ut'R}R4RyI'BC(RgF'U@$8'zn8gt}q r 7c 9 3 V H G c G i gf d 3 'T5tD`HR}R}bc H G S C 9 7 9E C E 9E S 3 C c a 3 v S7 C c q i d ~ vf 5 3 7 S 3 S 9c 5 3 Y 3 H G 9 i gf d r RDDU|`7}U`CF'8'`CFD6(8RD(8'tqhW648688T5t8U!`7R4B7h0a}q r d A5G v c tW8t}RG d f d i vf A 5 G 7c 9 3 V E 9E Cc 9 c G i gf d gf 3c 7 C E C P 5 3 t0zi thuthd 8T5tDu6`3F'T5&R}t0zubxi hd &UtF't6UY 7 9E C7 E 9E S 3 C c q 7 S 3 S 9c 5 3 Y C 9 C Y 9 V A 5 G i d ~ vf 9 A C A 5 S 3 f 9 i gf d r UBCX'4'`CFD6p!8D8686T56U`7}UUU`C8n8tqh5UD8D6D`74b|a} k7G E H 'Iy'`C8V 97 CE G 7 9 c 977CY 3E V HG 5 97 9 ACECc 5 S 3Y 5C 7 H V v G C C H HGCE 9 qEC PC 3 v@Ftyt8T588T5UU8FDRD6`$T5UF'D6U8'yU`C8Ujta`9R'FT56'tU4Q(

) 

1 2

6 pX i d p x d f  ki i v E 3 f CEC Ut (6jagF'8y xQ{ p i i i d f 9 7 { i d f 7 3 Y { x x i gf R t u5@zi R t QtU|hf m | hd k7 9EC7 97 A 5 G 7 3Y 3c 9 YG f 9 c Cc H7 5 GEc 5 GY C 'U`CF'T5W8ttUT5URDB78zyIG@8tt'e 5 9cY 5 Sa c 3 3c G7 3c 7 5 9c H 3 VE Cc 9 c 9 3 V A 5 G k CE S 9 6`3U8Dwt8}86`3RTGtD@'tT5tT5tqgRW8t D!FD6`
% q

A B

' 6 5 3

ln ll l

o 

h r

A B 

llk j m

h i

g P1

A B

h i 

A B

h i

g P1

A B c

A f

e Hd 5 3

r7 5 9cG S C HG 9 9E 3 C '86`3RDDU0ID5BX68c 7 C 9 G v 9 C 3c 9 5 9c G S C 9cE C7 9 3 7 k E C A G CE C c E 3a C 9 YE C UF7`cI7$'563`cRGDSDU56`3IDUe8T5X'8T5R8U'DR'F$88j`7UF'8 p i 'tFf i f y F d  y k3 7 h $i p ( y `{ h i v y B{  y `{ 4 k&6`39FE'7Iyct5RTG8W8tUBCX'`YFU6UID!8zgy8Df 5 c C7 G Gc H f A 5 G 7 9EC7 9EcC v 3C G a 3 v S7 C c f c S p Q{ p p i f y `{ {   F & d  y k 3c cYC V7CE 9 v S7 A 5 y 8 @yUUqUF$`c}uD$8tG F tB8TV`c8S f H 9 H
p 3 1 A B

-5 -4 -3 -2 -1 0 1 2 3 4

saw(x) trig.pts.8 trig.pts.64

10

12

i g 7 3 i g 7 3 i g f 7 3 Y ' pthm 'f tUY p6m Qf tUY upthQtU m d hd i gf k 9 7 9E C7 C 97 3 Y C r d g v E 3a i g df g i gf d C Gc q H V v G C E C c 3 5 '`7yU`CF'8T5tU8 $}8jpe$huhRtpt`C8tDU4'8t8 i d g v E 3 i g f 9 i g f 9 i g f 9 46haf ' pthu5@7 pthu5@7 huT57 d } k7 S 7C 9 7 9EC7 C 9 '8Ut`PtyU`CF'@!8T57 C8'8}'t}FtDD7FzUD`7te`c}6xUUUUDDUR$ p e`8Tv!`79 c 5C iC PG CE G S f EcC v v 9c 5 G 9 E 3 qi ACcYC V C 7G x H V 97 9 f 7 9EC7 C 97 3Y C c 9Gc f 3 5C q HG 9EcC v v 7 EC 9C d A 9EC V 9 U`Y8y$UBCX'8T5tU8T5Rz86C '8yutB`HR'`YFzUD'8`cUt6`3F'qe`c} Y`D6`39F'qVWDe`788'8Ut'e6h bRtt`C8tD 9 A E C C f 3c 9 c A 5 Cc C 5 G Y C r d g v E 3a i gf d C Gc q H V v G p g ri g if 97 i gf 6@&hT5Dphd d { o l !z k 9 &`c} p { i g if 97 { pb0T5z k7 9EC C 9 'UC`F'@7}85T7!8T59DA856qUF6U48U 4'8uUp}F}ha 8tD0'8zt48 3 V7 CEE 3 Y C c c C C 5 C c k i g f d i gf d 9 A 5 G E C c 3 C c 5 p { i g if 7 3 &hQtUY { x p x k H V 97 9 7 9EC7 E 9E S 3 C c CY 5C &`8TvB7U`CF'4'`CFD8!8z!U8'8 v z { p g ri g if 7 3 Y i gf 8&h'tUqhd o l d x { p g ri gf 8hd o l d p x k 5 C c i gf d i g f d 9 c G c q 9Ec C v v 7 9 i gf 9 c G c C PE C7 f &'8h}Fy`7}IDDt`YFU`7hd ta4RD!F'8
~

|
{ } t ) ) ) 3 3 x

{ |` z

|Rg

y d

v wu t

p d p it'Ff g ri g if 7 3Y i gf o 8pbUDbd Fm l d d i d f73Y d f i R t UDi t d y { x k ii gf U@hd ACPGCf HC E3 U'D'q0`HU446haf i 7G q H V v G C 5 G 7G { 9 r7c 9 3 V CE 3 v A 5 G CE 3 R&tBC8tUtyR$x 8T5tG tT5tqF8n8t!X6v c G i gf d 9 C CE G 5 9c Y 5 Sa 5 9c H 3 VE Cc 9 C c C G v 5 G Y R!phe`c}UX6Re6`3U8D6`3RTGtD@'tT58zRtt'C R'F'8T54! C7GCEY 9 C 7
) s

i t'Ff

3 )

{
)

pX i R t T5qi d i d f 97 d f p  X i i d f73Y d f R t 'tUqi d 

3 3

tx {

i t Ff


i Df


i t'Ff

B e 

i gf h0

'BCXD68T5R864!U8'( E 9E S 3 C c 9Gc f 3 C CY 5C


@ B e 

6 g 8r phf ig 0 l d r iD f l kUTE9RD8jF8E G V vE 3a7 5 G l w i Df w

vwt 

t F6 f e 

i Df

kC V3 C 5Cc q tD68!4'8D i t'Ff

9 H d V Cc7 C c E 3a w 9 9E 8T5}R`3`Hyxr $'86j $8T5`cFy



6 B e   B e   @

i Df

c f

e d

r 

e 

#6

i i f d Df 80 { i gf { { phd kCY5C r d tU8'(|ut { i A 5 G q d i CEC D f 8tp & F'8y d i6UFf r 6 x l t D f i f i d r o { 8 0 l d t i f o d g 6r { phd o l t { i gf r d A 9EC te8B3X'DV 7RDU`Y8y&Ug h0 hW8zD 8`3F'DzBc(bx 6`3U8D$`YD8`3F'DG G 9 q i d gf i gf d C 5 C A A 9E C V 9 i gf 5 9c Y 5 Sa 9 A 9E C V E C 97 5 3 H G Y7 a 3 C 5 G Y C G v q 5 9c G S C 9 7 9E C7 H V v 3Y C c v 3Ea 9cE Gc 'DBA@86UY `CR'Rt8tDU4G RQq!6`3RDDUT50U`CF'U`C86U|8z6F}8T5Xt8

t|&tt$pt

r 9 G A S 3 9 A E C V C c c G 5 9cY 5 Sa C c a 3 7C H G P C c Yc G 7 5 9cY 5 Sa 97 G '7U`CFEt85D8Df`YD6`39F'q84R6`3U8Db8zRU8TSRR8Iv 88B3@U8Db(`7RDf C8c0I3yUC8TSRRPy8c'86'8t8tD'RUtF't6U}U`CF'88|RDF'8 a 7 H G C 5 C qE C c 3 C c 5 G c E Cc7 Ga 7C E C P 5 3Y 7 9E C7 C 5 3 c G c C PE C7 f ' ( d kd g cG A5 DtxgRW8tG ' igthfuT597 ih0T57 &b0T57 d h0g i g df g f 9 i gf 9 k 9 7 9EC7 C 97 9 A 5 3 V7CEE 3Y C '`7}U`CF'8T585D88DUF@6U8c ' m m | m d k Dv yRv hyIU8'8 g 7G i gf d 7 G C Y 5 C
t s t

 6
c c h g B e " s 

c c @

f e 

s 

f e 

e 

r r 6 D r 8 0 8 d l t i f i f l l d r r 6 D r t 0 8 d l t i f i f l l d d r D f D l t ph0 i i f i gf k vE 3a7 5 GEc C7E C P 9 C c 3c 9 9cE C7 5 i i f i $6hF8t@F'tT58tT58T5F'8$ D f D D f vE 3a7 5 GEc G 7 G i gf 5 9c Y 5 Sa c G i 6hF8tzWRD$hu 8B3@U8DbRDy D f i gf qi f bx UD i gf C7 3 V V S phd DDDD
 u 

|xt0 g
x d ! s w h H g

d !

Be

9B

'

V 3E V 9 H 3a C c C P G C q 9 G V vE 3a7 5 GEc C c 9 5 97 3c i 6FD85(Q`3`Hth}8z}tD 48UTERD8jF88ghta86`!@D f



i D f { D phf `{ 'UTvn8tUUq i f ig q7C 9c i A 5Grrr r i Df i g CY 5 3 5 9c 9c 5CEC 9 r phf U886`3RTG'F' q i f i gf 9c 9 7 Y 5 C S CE r h0 85@jta8$U8'8DUF4p i Df i x gf 9c 9 7 C 9c CY G V r 6ebx 8T5jta8TvR URDD}q i D f m xD x hf m h'D x i f i g i gf 5C '8c i Df m iphf m 8tiD ' g A5G f i gf 9 q A 5 G x 7c 5 Gc7 5 3 Y E 3a 9E G C r phQD taR8tet86U6h F`cFt'8T59 q r 7 vE 3a7 5 GE 'w6hF8tc C8T57$A85G8T57tUY 7UBCX'7!8T57$8tC8T5t3U80I}U86t`3RDt!Xt4F}RTG8 9 C 9 3 9EC C 9 A 5 G 97 Y C c a 3 7C S HG 5 G CE G 3 c c7 H C p g6@D fT5qghd l }uD f ri g 97 i f i A A 3 A 5 G H G CE i gf D6$8tuR'F4h0 9a p g ri g 73Y i f 8D f'tUqghd l uiD f k 5 P A 5 H G C i gf &'CtUC$8tGuR'FE4h0 9a iD f iD Ff 5'8xHR'FE4hf0 9a Cc GC i g GCE 5 c v v r F`c`9HR'FA8tGgFEUCw8} k7 9cE 'U`CF'C i gf 9 9E h0 8T5`cF

tRp}t
% v F v U 

@ c

B e d

r i i gf d i gf df 7 C 9 H GE Cc 9 H f S 3 A C c H GE C 5 C CE 3 v H C f 7 H 3 tti buhhUBPI6UT5`C8D6D8z$B`HR@'8'tF6uBHU4D`At8 5 9cG S C 5C c q 3 6`3RDDU'88 c u6F}`C8t@6U54`7R hd 8&8B3@RTGFtQ$UD8D8D0R3 v 3Ea H f GE Cc 9 9 i gf A 5 G q 5 9c 9E G P A C A 5 S 3 f a B7htaD DBA8!TG86h|`78'8 I|5 C v C c G c 7 G 9 v C E 3 C H G E C c 5 E 9 E S 3 9 i gf d 9 7 H 3 H S vE 3a 9 c 5 C a 3 c ''R@y`7'X6U8 I6U}s7 '`CFD8 d it Ff r D r 6 d l l t phd i f i gf k 9 Gc f 3 C i7 H f 9E G P C CEa E 3a 5 9c GE C 97 5 3 Y C S A 9 f E C c C 3 &5I864U`C8tTGFtRUFb6h6`3R'D`A88U8$`c}'8Utc 5 C r 7c 5 C 5 3 V v 3Y Y 5 C S CEa a 3 v SEcYC V7 G 7 G 5 9cY 5 Sa G 7C77CE V C 5 9c G S 'RtG t'88D6UU8'8UXb8RDUUDz40R6`3U8DUUFDDU4'6`3RDD
p

 @

i tthf

c w

d r l g 8r i f l g8r i f l D i gf hd ki CE S CC f H V v G C 5 977 S G r U6XD6U7$BC8tUG@8t!}q

  7  4

! d#

m mx i f i gf x t |D q phd k v c S 3 f G 9EcC v v 7 5 9cY 5 Sa C c C G v Dy88Dt!`YFUw@$6`3U8D8Rt!C a x i f v D q $gg v bd s i gf v

 7 

k HV vGC HV 9 r t`C8tDUBC8v8}

k 7 9 G V vE 3a7 5 G H V v G C C v 3 'FTERD8jF8E `C8tDU68

" {

i 

~ ` 

 

 $

e d d

s 

"

i 

Ub

i 

t&  d i f iv r m D l t ! r m 6 d l 0hf i f k$'F63U8(s7 RQUXt tF'8 Q UD' hD8zy`7DW8B3@RDUT59 m D v CE C c H G P C7E G C PE C7 f r C c 9 5 9c G S C i f `CBH8Ui&d aR36`3U8Dtht`'Fzth68`76`3RDD T59 9 q f 5 9 c Y 5 S a C c 9 5 9 c G S 5 9cY 5 Sa C 6`3U8D8 i tthf ri g f di f } @6 d l bf ig i 'Dhf i f m D i Df D Df i f i i Df i f #D X k&'C8cqU}FW8t(h86`3U8DbpR'X8t`Ptt6U@R'0IGUUqy 5 q i g f d A 5 G i gf d 7 5 9c Y 5 Sa H G CE C c C H 3 P 5 3 Y C7 G Y H 9 Y C V7 G 7 ri i f ! f i f i f l 6 x 8 d k HE 9 9 `FtTG`HTv8 i i f Df D ri gf i f 8 hx 6 d l kE U63 itthf v i gf i gf h0 p hd i t'Ff ri gf i f 8 hx 6 d l b0 i gf k A 5 G d a 3 C c 9 Gc f 3 5 C c C q 5 9c G S C v 3E 8tRth6`ttz8T5R86'848D e8B3@RDU6X
s %

% @

 

t `

k7 9cEC V 3E V C v 3 UBC@F'D6XD!68 p i gf w i gf phDb Tv`9H k7c 9 C cG c 5 9c V v S77G 9 C c 5 '@`76U`c9 ID6`38D@ReBA`H}8z$ v g r i g f i tthf s 6@hD l r Dv 7 G i f x 9 Y H V CEf i gf a 3 7 9cE C V 3E V C c 9 A Cc7 R}@ g$8T5URTG8'Fh!phD RU`CF'D8FD8T5$UUC ECc 9 CE G C 5 9c G S C 9 C P 3 f G H V v G C C H S V E H S 5 GcYCE C c 9EC 97 5 3 'T5Ft4gR6`3IDUT5tRDtg`C8U@B78DG888tUUF885X'DBA@8642 rH G vE 3a hR8jT59 `UFDD`7R`3UD'R'Fzg8 tXD6U!`78}RD6U@886tUD`76`38'`7DA HCE S V 9 HC f c 5C vcGCEc C r C7E S 3Y 9 c a3 C V 3Y7 C c A 5 3 C f 9 5 977 SY 9 H`8G RU`CF'D8FDV @Bc`c}t8B38tRTG8`H`H}W48tt8'$`78`RDtG H Sa 7 G 7 9cE C V 3E 7 9 9 C7 3 H A 5 G c7 Ga H V 9 C A 5 G q C7 5 C7 97 H G 5 H G 977 HY C c 9 H G c G 5 9cY 5 Sa G c 3 5 9 5 9cY 5 Sa 9 c qE C PC 3 r H V v G7 a R'`YRTGU85`HR!Re6`3U8D!8`7W 6`3U8DF B78z&U'tU4R}$8T5`98t4R3 5 9c GE C 97 5 3 Y C c 9 C7 S H f GE C 97 5 3 Y a 3 C f 3c 5 9c Y 5 Sa w Y G 9 C c A 5 H G 7 C 8B3@R'D`A86U}8zT5}8!`C8t'D`A86U&RD!8B3@U8DbhyR@E 88D`HRDz}e

t8||thp| &te |
z "  ' 3 t d t

 c

ighD c785'DDDeIDyD # D tF'86$8Dv yRyRDzUBY@t3 f 3 C V V G c G i f C PEC7 f 3 A 5 G q x 7G cG c C 9c x D x g vm s i g f i6Rhf x ix f 9 i f 6D T57 D q 8 g hD k 5 9cY 5 Sa C H S V E H S 5 GcY r &6`3U8D!@B78D(tTG868tUUC q 7 7 5 C V V G c G E C 97 5 3 rH G S A H C7 9 5 9c Y 5 Sa 5 977 S G RG 8'qDtDyRD}'D`A86e2 hRDtaUy`76`3U8DbG@8t! r
v t

7 vE 3a7 5 GEc 9 C c A 5 G 7 5 9c S 9Ec 9 A 5 977 S G k CE S 9 6hF8t@}TEU8$8!86`38DTfF`7DtTG8!|XD6`

0 0.2 0.4 0.6 0.8 1 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 0 0.2 0.4 0.6 0.8 1 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2

rC 5GE C 9 5 G tt8t`c8WT59 `86!X'U68`7R4`76`3U8D6hF8t@'`CFD6p}8ta'DDtD|6`8T5UF'@tT59 H 5 3 3E C 5 3 5 3 H G 9 5 9c Y 5 Sa vE 3a7 5 GEc E 9E S 3 C c 9 5 C V V G c 9 v 9c7 CE Cc 9 c C v 37 c G c c7 C S7 H S vE 3a7 5 GEc E 9E S 3 C c a 3 CE Sc G 5 9Ec C v v 7 C 8T58U8yIDy@Utt8BA883 u6hF8t@}'`CFD88|RFDRDBYXUD8 r&6`3U8DbeRDR6`38'8Ug`YD8B3X'D8zDt'4t8t`c8T5F'U685 5 9cY 5 Sa c G c a 3 5 97 5 Cc C 9 A 9E C V G C 5 C A 5 G Y C q C 5 GE C 9 5 G 9 3E C 5 3 7BU`Y8y8B3@U8Db6hew'8t8tg`cR8T5t68`364 6`3U8DW`YD8`3F'De6ha 9 9 5 9c Y 5 Sa G E 3a G E C c 3 5 G 9 c G 9 3 H E 3 5 9c Y 5 Sa 9 A 9E C V G E 3 7c 5 9Y C 3Y E 9E S 3 a3 cC7 CcCEY 9 A G 7 S C PG 7 9EC7 C 97 C 97 3Y A 5 G H V v 3Y E S t'`CUgz8U'`CFD6uR|U@UF'`7De8t64UBCX'!8T5R 8T5Ug8t$U`C86UD
~

ptp|

s

|&
{

c

r w 9 `c} y i hd f $e Dw |D f i k 9 Gc f 3 A 5 G iE C A G CE C c E 3a C 9 YE C f 9 c vE 3a7 5 GEc E 9E S 3 5 G Y C &T5R86$8}'DR'F!8z46h`7UF'8UC`786hF8t@}'`CFD8t'4 { i i gf e ehDw hf w ig k 7 H G PE Cc 9 a 3 C c C 5 C A C 'RG RQF'@tT5Rth'p6U}`(tU&tzt88zDe r'`cTv`9ytD}UU8'8DUBC8D'tDU8D `7R'F8}`7RD8B3@R8g8T5D8'8UC 7 9 H C PG 3c 7CY 5C S C7 H f G 5C c 3 f C c HGCE C c 3c HG 5 9cGE C c 9 A 5Cc 3c 7 S 3 H G 5 G 7 G A CE C 97 5 3 Y C f 5 G Y 7 A 5 9Ea 7 9 A 5 G 5 9c Y 5 Sa w C c 7 G C v 37 5 86tt`3RDtReUF'D`A86Uqn'D8'BCXb(`cW8t6`3U8Dh8(6}!8
   ~

3 e 

"

3 u

r gf w f gf wi gf Cc 3 5 E C cE Sa r i hDi 0 i !b@h0 t8}'8zFDbp rrr 5 'U6`39 c H 3 P 5 3 Y 9 9 c C P E C 7 f 8TStt6UG `7`78tF'8 i 6pe @hd l f d g ri g f wi gf k 9 H C c 9 Gc q 77 C 57 S 3E 3 9E E 3a '`cTv9`88T5tDU886F8t`F46jA E G C E 9 A H G S 7 S 9 C Y 5 C Ft6UX`7DRD8$`c}U8'} i f d g r gf i gf 6 8i hD bd l ki U U f RRF'T58$T5 67 ~ HGPECc 9 C c 9 C v 3 E 3 v C 3C C H G C 9 C E C 9 c f g 5 9cY 5 Sa C H S V ACY H V 9 6ha'FE6U8c8TSRRP!cRTGDAUv@'ctT5C8 i hf 8B3@U8Db`78DUUIG8zB7DA C8e8T58S8tG ~U !RGQX'CtT598T58686T56U`7he8T5DR}q c 97 A 5 i f H PE c C c 9 7 S 3 S 9c 5 3 Y 9 i gf d 9 v S77 G r v g v g8hDw ri gf l v s k 5 9cG S C a3 9 H C c 9 Gc r qtW6`3IDU0IBcwv`9885t4q
 0k o "

v g v v g g

s bw i gf

``HRD8'U86U4 HG ScVCY53Y r

v t
e6 6

p6

p6

B e d

U  e

e r

"

c

2  

2 

{ iv tthf i i df Y 97 f eu hD8T5Fi i d x i i df &0 h|d d { f ii i df df 9 f @eu h|huT57 i i x i 0 kYC tU7 E C V 7 5 9 A GE 5 G c E C c GE cE C 9 ACc 3 S H G vE 3 5 Y 5 C S CEa H V v G7 C 'D8tTGDRtD'8R$F'(T5U8`BHI68 d U8'8DUF8T5`98t8c a3 7 vECc 9 9 c C 9E CE 5 GY C q 9G v 3 R}'@$T5`78!`cFUF$t'!4p&T5R6DA 8z46hTvR`C8tDU!88T5ttG C c E 3a C 9c a 3 H V v G C C c 9 { di g i tthf idi g f 9 d f & T57 i &i x { r o{ l i &i 0 d f d { r o { i &i x t l d f d d r D f l h0 i i gf k vE 3a7 5 GEc C7E C P $6hF8tz!F'tT59 C c 9 vE 88T563 a E C V A 5 G 5 9 c G S C 9 7 r i g f C Y 5 C A 5 G i 7C 5 C A HCc H V v 3 h'qg8t! g6`3IDU8T5( Uphx U8'8g8tD f U8zD`UU`C86UY if C Y 5 C S C7 C c CE 3a CE C c i bx U8'8U@W8$F8jzF'8z4 BHI8jv D f R86W48S H G E 3a i 35 C c { i t hf i F0q m Fm f o { i t x t D f id f d i k 9Gcf &5I863 C q 5 9c G S C 5 vE 3a7 5 GEc E 9E S 3 C c 9 vE 3aE C r CE S 9 5 3 7 7 4|qt6`3IDU63 6hF8t'`CFD6p$8e8T56h'u DnXD6T5Q8zRG i F0D f f q i d i t D f k 9G v3 A g Cc &T5R6D!8$T59 d HGPECc 9 cG H V v G7 f 5C cqi ~ f HGPECc 9 C c C 97c S 3 3EC 9 i t|`7RQX'tT5uR8T5`98tzt'8U FRQF'@tT5484D`A86F'U`7D f B7IDuU`c8D'D8tTGDRbf 9 cG c qi 9 5 S g EC V 7 5 9 AGE f A C 9 H A 5 G f Hc 9Ec7 9 i gf 9 H C7E C P 5 3 U`cTv`988tD`U`YF`7(h0 taBU@F't642 r d A 9EC V 9 i xtn8`3F'D`c}D f R3 a 5 97 5Cc C 9 A 9EC V C c 9 9 i 8B3@8'6U`YD8`3F'Dy8z|`7U`Y8yuD f vE 3a7 5 GEc G 9 7 H S7 CE 9 G v 3 A g C c 6hF8tzgT5|@Bc8zUFT5R6D}859 B7IQF'T54RG 8T5`98tz0R}`c8UF8D f H GPECc 9 c H V v G7 a3 H S7CE C c i i gf 5 G E 3a c G c Cc 3 C 9 c v 3E ph0 t46hRD@t85 4}`786X y i t hf i t f hd y i hd f i t Dw D f i Df i Df Df i ki UD f ig phf A5G i 8tD f i gf 9 r 9 G v 3 vE 3a7 5 GEc C c 9 5 9c H 3 P 5 3 Y G 9 9 G v 3 h0 `c&T5R6DA @6jX8t8T56`38TStt6UW`75I6DA g C c 9 cY S A 3E V G cG c 3 5 q7 5 9c H 3 P 5 3 a 3 7 9cEC V 3E V C c EC f vC v W8zT5U86XDeRDR88WC 486`38TStt6UY IgU`CF'q6FD8'q''C i g wi gf phf bx { i tthf i i gf wi if ehDe hx hf ig kH G 5 97 C 9c Cc CE Y 9 A C c 9 Gc f 3 3c H G PE Cc 9 c G i gf H G 5 97 H V v G 7 E C 97 5 3 jID6`|Tv4UX'`7D|85I86} `7RRF'5uR&bx RD6`(G 8T5`98tzu'DBA@8642

p 

5 n   j  5 6 ' 3 T o ww mdU1 6 R ( & k E ! E 3 E H  ) " l  H T '  375 c5 3 ' c5 c S pE v   w #c wS pE   E 5 5 E 5 H   ' Q I   5 B 3 ' u p' 2i6 E  i 3 fg E 5 E 5   E E 0 H h a cwS H   9`c R F cU u  R#6 5 " " ' 20  E    5  5 E !  w E  9 fe  E  3 E d pE @ 
p @

c 6

 )

" #e E ' 

 0 v 5  3 pE  E y 5 @ 5 c5 ' 20

cS H 3 5   3 c 5

 R E cS  R   '

 3 pE  H V pE

1p

5 3

5 F DC A @ 8 " pE  3 #Ewyxe6 5 5 TV e w v pE  3u  9 4 " y2ph 3 E ts r q i g 5 a fe W d 5 a Y W  E  9 Fc F 4 " b` XV E  9UI4 ' H S T Q E P  R ' ! E F DC A @ 8 6 5 H I F c 7 GEB97 4

&    " #7( '% E

  "    $#b!  E

 e  E E  p4E !  3 p' 21 d  E E 0 )     F c FpE @

d

d 

d 

a '

z  #

x  X
-W

v yu h w v ut sr ByTIT(

 T'

c d

5 4

q gpE

-0.4

-0.2

0.2

0.4

0.6

0.8

1.2

0.2

0.4

0.6

0.8

0 1 0 1 -8 -6

-4

-2

0 0 2

W 4 y = sinc(x) 6 8

H 

c0

HE

H  " p

E '

 $ @

'

F F

5  3

E '  E

5 4 (

2VV2I

0 " T TV pE 5 5   S 5 E E  V F 3  E  c ' '  R   V e X U p' 3  5  h E '9  q ' c E ! E 4 ' E @ @ e  32  E   E  E    

p {

Xj g}#yB}'py}4d ! h s

g p 1$

d ds 

E '

'

p {

s ws  g3}|#yB}'py}4d ! h l d dj s  1$ E ! E 4 ' E  E

 2

p e

5 4

5 " t d  5 # ! gs 3 t  @  w E   9e H  pE  pE 5 d z  E  3  #E " 1' yBy}  } ' is  p c wS i 7 ' 

'!

D C A F D 5 " 3IG'|1B3'GVd   4 3 pE ' 3 E 1 T'  5  E 5 "   T'   F  5 D C A F DC 5 E w R c c3  'GI1dBw'!B|B#ev 3 '  0c  E  5   E 5 ' F c T E @ ' c   5 r1 3 E w R r4

E ' E

 e R    

e 0

    V pE e  E  H I  d ' g

" ! F c 5 HT E' I  c   5 y 5 "

r

B1g$i%eg%11R%g

E '

E '

5 3

E '

y " y 5 wc5 c

E

    e 3

E '

 E   Q e T'T

53 e  p! 3   5 E E  TV  z

n   5 " Q P '  Pc Q P n " @ R   @  3eF c  Rc R 5  3 g$e wq E 7c  #rE ~ ' E  5 z  H  cwS H 3  i rh  P Rkc R  E  F h  E c3  y 4   5  5  " B d {
c w p

7 p

E '

c

E ' E E '

r  V  ~

y 5

c S pE  5 3g 3 5

5wcT 5 5   '

4d } ` | {

d d d d dj

xs p { s  s  s | 1 xs s  s  s R 1 xs xs 's  s s  s 1| 1 s p { 'a!#By'B} d l h } p 's  1| s

  4 5 5 E  h 'a'  F E ! 3 h ! Bu yr H  E 2 d g E r d  E  " a  'r  g 5 E ! E 4  5  E' E @  3iR 6 iH E ' wS  c sc T  # " E z  5 E  53V H E R 3 ' r  $B e 6 #  w r p { g g c e H P "  z #  w ipE cS pE @ 3V e  e  x 4 5   z g 4  e
F

'31'$'y1

'

d d d d

s  w st $   s s 3 $ wyi s s

j $ s $ s $

1 5 3 s i V   4

5 4

p {

g~#yB}'py}4d ! h   s d s yi 1$
c s H I 0   V  E  3 TV 5 p

c w

S  '!w'y2'e'wy|y 4 1 e  I   CF A A C 5 " 

E '

s  5 4

5 3

 2 1 e ' "  "

4 " ! Bs Bu 2 Xj r 5 H T E H s  3

s c

53 ys s   ~e 5 7

E '

S h ' E  e!c5 "  

E '

@ '  

5 3

# dj V 5 #U9s  2 H 4 s

" a

w R tXj d 2 w t j h dj d j

 E  T ' s   5 3  E s  

   

 u
@ r

E '

5  3V pE 9   u E !
p

5 5 w Iw wc5 E ' R  ee 

E ' R E

8  FF  5  B!!A  c3  5 3 5 pE @ 

E E }'

" B

E ! E 4 ' E  E    dR 

  4

5    

c3 T E 5  5  3 5 pE @ 0 |

" 4 4 5 4

g 

E '

c3 5 S

  

 " B E ~ E $   5 h  s r 3 a 

E '

g  r  d g  ) 

xs  Xs 

5  4 3e

  

" s s  { s  5 5 3d 3 e' d d j
dj

p { 3|#By'B41h  } } p } d s p { 3#BBIB41h } } p } d s

xs xs  { s  e s 1 xs xs  s  1

@

5  pT c  H

" p | {

 i c dV '  j  E d d c 5 'p { d h  h

  R  

   

p @

dj

w ' w ' ' w  '

vj

dj

 

j vj

dj

vj

 

vj

q '

  i

9RV E   E ' ' !   E  ' E c2 3V E  5 

 5 "  c c    3 4 ' I V " 3 c3 4 5 5 E 5   4  5 z y  #  w  v R c4 d s  g 1$

p {

g} } p } 3|#yB'y4d ! h j s

V R d ds 

E E y#
p

5 4

p {

g} } p } #yB'y4d ! h Xj s

g p 1$

4 H Iu

E '

's)

5 4
" #

ee " H 'y

) ' &(e7'&y$ %

E '

3 E E 5  E@E  z R # c  pE c ' ph T g    4 5 j   5  l n h y @E  @ XV T'  3 H E

  0 4 "

E  " B ' 63 E 5 5   g H T  T e e 3 E I V c B E ' 3'  @ 3 pE R r 5  5 H E 5  5  5  5 #9R  E  3 U  3 H  E E @  c3 F   3  "  5 H  5 H E 5 e  9 3 '   c 3 h y E  3   E  E R R p'rr e    @i ' 5 c5  E     '  4 r' 5 c3 e ' 5 x " a 5  E E h H
p

"  g  g  R R  E   g ! 3 p y 5 r  e 6 #  w 4 5 E  z  g w g c e  e F #  w g  'g " pE cS pE c 5 z a  4   z E  e 6  c e  ' F #    pE cwS E H ' R 5 6q" p

 5 $

5   3i5

E ' R

8  FF  fB!!A

Stage 1

z  #

x pE

Stage 2

Stage 3

 '
d

Q C

d b xB b

d A A yd eW b C b

d C Q b W b A A yd xB b d d d C b

5 3

5 h z E

E $

5  3V

E '

5 3
h g g g fg

 E  z E  ! d b

'

@ I   E T y 3 5

w u u u u vt

C W p p 1$ 1iA frqp
p p rqp p p rqp

1iA W C IW C iyW C

1iA W C " "" s'W C DBW C

" "" 1$ IW C C 1$ iyW

d eW b

5  d b E z E 3 E  pE ReW c T yer2h T  n E   " nE  4 5   h E E   ' pE d  3  E r#' r  y e c d pE '  E P "    E ! E pE e R 6#c S pE   $R pE  # 5 c5 e c d e  H z    E E    g @ e   E ! E 4 c '  ' E @w E  E F y 5 "

` a

g$

g

X Y$

B1g$i%11R
W

T U

V " # Ug   !

53 3 ' 3 E T E ' r g  Ug 5 E 5 5  V  r 5 wc5 r TV wS I 3 E '  4 5  4 5 H  

E '

 T'
3 4 5 6 0 1 2

c d

 V
4 5 6 0

a p pE

 q

c w

S w

 h  " # c'  ' 3  0  "  2 B e c d    5  2 sC

7

E 5 '  IwF r  5 pE @  p

I R P SQ HBu G F tr

xs 7 W 1$ 1iA

g p

DB C

i   v str w A1 9@8 ! yTyu 2   w  3} ! W E H   T' c d E e x 7 3m 3 E " E ~ E 4 E e g5 E ! E 4 6 '  ' E c2 3 5 4 5    E E  5   5 5 E    E '    3 '  5 ~ rh d e '   5 E 3 4'%2
0 1

-0.5

0.5

0 0 1

cos(x)

-0.5

0.5

0 0 1

cos(y)

-0.5

0.5

0 0 1

cos(x)*cos(y)

E '

E E

5 H   5 3 pR  c4 " 'a g   c5 e c d  E  R 
6

E '

5  5  3 E  i c 3' E $0 c # ' H z

5 4

w

V E 
5

6 e

 e

n   B R  4 g  4 "
0 1 2

p p 'pE

5 5 E " 9R  E  pE  R  v T ' c 3 '  ' u H 5    H 5 5    Q 3% E ~ pE  w " wc5  u  c c e T' S E   E 'H # T' E  c F E 5 5 c F ' !  e5 6 R ' E      "  4 5   g g 3 c0 cz  olg  e pE S TV R  E  5 d|ek) @iUc ge# R pE  w  44 g 4 ) ) j f d h f d H

I3#1'Uh!''!s

5    5  " 3 E '  pE  E e e B pE H V '3 E  c   H  4 5  E h H E R  3 y 5 i  p' w I z E H E ! E 6 R R  E i g  5  5 3  F r 3 5  c rR E ' @ T' c d a E ! e pE   wH c3Iw E 0 x     5 5   

d b xB b d

d yd eW b d b

d b eW b d d xB b d d b
r

5 3

E '

h

d b

d b

 HIu E  E ~ E pE e V  5 E E E B @ I  E ! 3  '

p {
H '

} } p } |yBBIB4d

s 7

5 3 z E

U!Y

5 3 d

0.05

0.15

0.25

0.35

0.45

0.1

0.2

0.3

0.4

0.5

DCT DFT Original

E '

C ap b

E '

"  1 e  I 

 

5 3

 I

 e 

 E  I"h H
" #

% 'h

 "    H B e c d F  e p' pE E  H E     5  h e c d E f e E E ~ Fe   E 5 c5 e c d @V E  E    R R     #' r 5 pE  E  H E E r 5 c5 T' c d 3 h E    5   g 7c  c e c d$ E e pE qTV pE E  u  #F 4  z 5     

" # R  E d pE 9  h n c"   d 5 wS 3 E !v' 0 5 c5 V e 6 '   0 # 5    E 5 4 z  F ' ' V uc@3  ' a E !  E   E  55 h h z g '  # E R I 5  e   3 ' i  " H c5 6 3 e   w F c3  5 E 5  5 3 P 3 5 eB  3 pE e c d e B p@E  3 k33!IX ~ 5  5   E  5  D C F ) " S 5 } D F oC '!BBBy 5 c5 T 5 5 $ E e pE   I c5 E e 4 5

h '

 5 pE  pE R5 c5 3 7 E 76ic h  e P H 0   5     X B 5 r e c d  E  E 3 E 1 $ E !g

"  1 e c d   ##   '  z z h n  w pE r  ' d) E 

'@'|

4 5  E z 5 5 5  E " 3V ' #   3 3 3 e  w R '  5 w E p' 5 T' E ! E 3 3 ' # dS  3 E  5 5 5 E  " p {}  E  5 4 5 5 F D 5 P y 5  h ' #G 0 b E ~ E E 76P 3 h 3`3 x
w

p { p p p { p

p { p { p p

p { p { p p p

p
h g g fg

u u u vt

H  9F pE r9   E PR H  5 hz E zy

 #rE 2 ! ' E  z  "   5 5 c  y p' 5 c d@' T pE E 5 E p { E 5 H Rv h d C rz E xIXI'p Q a x b h w w v H

'31'$

t 1u

3 3 q sro!s

(

5  y p p " 3 5 'pE @  5 5  5 "1   E   E 5 6  c 0 X cz   6` 3 y Rd@ 5  5 5 c z  E ' y 5 E 9 s e cz  3 h 9    #c S pE pc5 3 H  5     E E E 5 4 r e  T ' I ' ! p' 5 #B'# !'pxiy  6 e  E  3  h DF CF o C n  W 5 5 53u ! W d ! 9 pE 4 g h e cz  m 3 g g 5 " 3 c3 H '   E  E   4 " 3 I ' ' 5 5  5 4 5 H E E E

l p

6B'   5 E  n iu 3R e  3 ' 5   U E   T y V c pE 3 #' f e w 5 E E 5  VH cz  e 6 p5 c5 c

i d   5  3  ' T' wS  e 'uy V ' E  E  R r   c' '  R  E    E 5 di'wF " # e3} !  e  5 " } i E E '  c pE w E  h ' } ! W i E a e E  pE p' pE "
rp &

###P9'y'p { } } } p }
5 H '  3 e  p

cs

5 3

W  

 T'  ' 
 

5 5 c3 3

'

( (
c

p { } ! W 3} ( W i { i i} { 'p ! W 3} ( W { i S

i3} ! W 9 i 3} ! W v
5 c3

 e

F c p

E '

 e

E E

 T

 2

 e  I 
200

H '

 e
150

p a p BpE
50

h' 9    ' F

e pE uc h  " #p71  W   i  W e i  W 1 71  W  1  E R 3 3 ' 6  E 3  3    } ! W e c d 3  5 5 5 5  5 i 5  3 T' '  r  I c4 ! # E 5 E   e ' vT E 9R  E 5  Q  " E H 5 E 3 p' 5 ' a " e V  E  3 '    5 E h
8q #&

-100 -50 100 150 200 50 0

100

function gradient

250

300

5 P0903A.DAT 4 3 2

9000 P0903B.DAT 8000 7000 6000 5000

1 4000 0 3000 -1 2000 -2 1000 -3 -4 -5 0 50 100 150 200 250 300 0 -1000 -2000 0 50 100 150 200 250 300

61

Quantized Degrees-of-Freedom in a Continuous Signal

We have now encountered several theorems expressing the idea that even though a signal is continuous and dense in time (i.e. the value of the signal is dened at each real-valued moment in time), nevertheless a nite and countable set of discrete numbers suces to describe it completely, and thus to reconstruct it, provided that its frequency bandwidth is limited. Such theorems may seem counter-intuitive at rst: How could a nite sequence of numbers, at discrete intervals, capture exhaustively the continuous and uncountable stream of numbers that represent all the values taken by a signal over some interval of time? In general terms, the reason is that bandlimited continuous functions are not as free to vary as they might at rst seem. Consequently, specifying their values at only certain points, suces to determine their values at all other points. Three examples that we have already seen are: Nyquists Sampling Theorem: If a signal f (x) is strictly bandlimited so that it contains no frequency components higher than W , i.e. its Fourier Transform F (k) satises the condition F (k) = 0 for |k| > W (1) then f (x) is completely determined just by sampling its values at a rate of at least 2W . The signal f (x) can be exactly recovered by using each sampled value to x the amplitude of a sinc(x) function, sinc(x) = sin(x) x (2)

whose width is scaled by the bandwidth parameter W and whose location corresponds to each of the sample points. The continuous signal f (x) can be perfectly recovered from its discrete samples fn ( n ) just by adding all of those displaced sinc(x) functions W together, with their amplitudes equal to the samples taken: f (x) =
n

fn

n W

sin(W x n) (W x n)

(3)

Thus we see that any signal that is limited in its bandwidth to W , during some duration T has at most 2W T degrees-of-freedom. It can be completely specied by just 2W T real numbers (Nyquist, 1911; R V Hartley, 1928). Logans Theorem: If a signal f (x) is strictly bandlimited to one octave or less, so that the highest frequency component it contains is no greater than twice the lowest frequency component it contains kmax 2kmin i.e. F (k) the Fourier Transform of f (x) obeys F (|k| > kmax = 2kmin ) = 0 and F (|k| < kmin ) = 0 (6) and if it is also true that the signal f (x) contains no complex zeroes in common with its Hilbert Transform (too complicated to explain here, but this constraint serves to 62 (5) (4)

exclude families of signals which are merely amplitude-modulated versions of each other), then the original signal f (x) can be perfectly recovered (up to an amplitude scale constant) merely from knowledge of the set {xi } of zero-crossings of f (x) alone: {xi } such that f (xi ) = 0 Comments: (1) This is a very complicated, surprising, and recent result (W F Logan, 1977). (2) Only an existence theorem has been proven. There is so far no stable constructive algorithm for actually making this work i.e. no known procedure that can actually recover f (x) in all cases, within a scale factor, from the mere knowledge of its zerocrossings f (x) = 0; only the existence of such algorithms is proven. (3) The Hilbert Transform constraint (where the Hilbert Transform of a signal is obtained by convolving it with a hyperbola, h(x) = 1/x, or equivalently by shifting the phase of the positive frequency components of the signal f (x) by +/2 and shifting the phase of its negative frequency components by /2), serves to exclude ensembles of signals such as a(x) sin(x) where a(x) is a purely positive function a(x) > 0. Clearly a(x) modulates the amplitudes of such signals, but it could not change any of their zero-crossings, which would always still occur at x = 0, , 2 , 3 , ..., and so such signals could not be uniquely represented by their zero-crossings. (4) It is very dicult to see how to generalize Logans Theorem to two-dimensional signals (such as images). In part this is because the zero-crossings of two-dimensional functions are non-denumerable (uncountable): they form continuous snakes, rather than a discrete and countable set of points. Also, it is not clear whether the one-octave bandlimiting constraint should be isotropic (the same in all directions), in which case the projection of the signals spectrum onto either frequency axis is really low-pass rather than bandpass; or anisotropic, in which case the projection onto both frequency axes may be strictly bandpass but the dierent directions are treated dierently. (5) Logans Theorem has been proposed as a signicant part of a brain theory by David Marr and Tomaso Poggio, for how the brains visual cortex processes and interprets retinal image information. The zero-crossings of bandpass-ltered retinal images constitute edge information within the image. The Information Diagram: The Similarity Theorem of Fourier Analysis asserts that if a function becomes narrower in one domain by a factor a, it necessarily becomes broader by the same factor a in the other domain: f (x) F (k) (8) (7)

k 1 (9) f (ax) | |F a a The Hungarian Nobel-Laureate Dennis Gabor took this principle further with great insight and with implications that are still revolutionizing the eld of signal processing (based upon wavelets), by noting that an Information Diagram representation of signals in a plane dened by the axes of time and frequency is fundamentally quantized. There is an irreducible, minimal, area that any signal can possibly occupy in this plane. Its uncertainty (or spread) in frequency, times its uncertainty (or duration) in time, has an inescapable lower bound. 63

10
10.1

Gabor-Heisenberg-Weyl Uncertainty Relation. Logons.

The Uncertainty Principle

If we dene the eective support of a function f (x) by its normalized variance, or the normalized second-moment:
+

(x)2 =

f (x)f (x)(x )2 dx
+

(10)

f (x)f (x)dx

where is the mean value, or normalized rst-moment, of the function:

+

xf (x)f (x)dx (11) f (x)f (x)dx

and if we similarly dene the eective support of the Fourier Transform F (k) of the function by its normalized variance in the Fourier domain:
+

(k) =

F (k)F (k)(k k0 )2 dk
+

(12)

F (k)F (k)dk

where k0 is the mean value, or normalized rst-moment, of the Fourier transform F (k):
+

kF (k)F (k)dk (13) F (k)F (k)dk

k0 =

then it can be proven (by Schwartz Inequality arguments) that there exists a fundamental lower bound on the product of these two spreads, regardless of the function f (x): (x)(k)
1 4

(14)

This is the famous Gabor-Heisenberg-Weyl Uncertainty Principle. Mathematically it is exactly identical to the uncertainty relation in quantum physics, where (x) would be interpreted as the position of an electron or other particle, and (k) would be interpreted as its momentum or deBroglie wavelength. We see that this is not just a property of nature, but more abstractly a property of all functions and their Fourier Transforms. It is thus a still further respect in which the information in continuous signals is quantized, since the minimal area they can occupy in the Information Diagram has an irreducible lower bound.

10.2

Gabor Logons

Dennis Gabor named such minimal areas logons from the Greek word for information, or order: lgos. He thus established that the Information Diagram for any continuous signal o can only contain a xed number of information quanta. Each such quantum constitutes an independent datum, and their total number within a region of the Information Diagram represents the number of independent degrees-of-freedom enjoyed by the signal. 64

The unique family of signals that actually achieve the lower bound in the Gabor-HeisenbergWeyl Uncertainty Relation are the complex exponentials multiplied by Gaussians. These are sometimes referred to as Gabor wavelets:

f (x) = e(xx0 )

2 /a2

eik0 (xx0 )

(15)

localized at epoch x0 , modulated by frequency k0 , and with size or spread constant a. It is noteworthy that such wavelets have Fourier Transforms F (k) with exactly the same functional form, but with their parameters merely interchanged or inverted:

F (k) = e(kk0 )

2 a2

eix0 (kk0 )

(16)

Note that in the case of a wavelet (or wave-packet) centered on x 0 = 0, its Fourier Transform is simply a Gaussian centered at the modulation frequency k0 , and whose size is 1/a, the reciprocal of the wavelets space constant. Because of the optimality of such wavelets under the Uncertainty Principle, Gabor (1946) proposed using them as an expansion basis to represent signals. In particular, he wanted them to be used in broadcast telecommunications for encoding continuous-time information. He called them the elementary functions for a signal. Unfortunately, because such functions are mutually non-orthogonal, it is very dicult to obtain the actual coecients needed as weights on the elementary functions in order to expand a given signal in this basis. The rst constructive method for nding such Gabor coecients was developed in 1981 by the Dutch physicist Martin Bastiaans, using a dual basis and a complicated non-local innite series. The following diagrams show the behaviour of Gabor elementary functions both as complex wavelets, their separate real and imaginary parts, and their Fourier transforms. When a family of such functions are parameterized to be self-similar, i.e. they are dilates and translates of each other so that they all have a common template (mother and daughter), then they constitute a (non-orthogonal) wavelet basis. Today it is known that an innite class of wavelets exist which can be used as the expansion basis for signals. Because of the self-similarity property, this amounts to representing or analyzing a signal at dierent scales. This general eld of investigation is called multi-resolution analysis.

65

Gabor wavelets are complex-valued functions, so for each value of x we have a phasor in the complex plane (top row). Its phase evolves as a function of x while its magnitude grows and decays according to a Gaussian envelope. Thus a Gabor wavelet is a kind of localised helix. The dierence between the three columns is that the wavelet has been multiplied by a complex constant, which amounts to a phase rotation. The second row shows the projection of its real and imaginary parts (solid and dotted curves). The third row shows its Fourier transform for each of these phase rotations. The fourth row shows its Fourier power spectrum which is simply a Gaussian centred at the wavelets frequency and with width reciprocal to that of the wavelets envelope.

66

The rst three rows show the real part of various Gabor wavelets. In the rst column, these all have the same Gaussian envelope, but dierent frequencies. In the second column, the frequencies correspond to those of the rst column but the width of each of the Gaussian envelopes is inversely proportional to the wavelets frequency, so this set of wavelets form a self-similar set (i.e. all are simple dilations of each other). The bottom row shows Fourier power spectra of the corresponding complex wavelets.

67

2D Gabor Wavelet: Real Part

2D Fourier Transform

Z 0.5

0.5 Po sit ion 0 in De 0 e D in e gre

Z 0.5

0.5 s

10 Sp

ati

al

gre

es

-0.

5 -0.5 Po

io sit

Fre 0 qu en

cy

(C

lF PD 10 -10 atia ) Sp

0 cy (C n ue req

10 ) PD

Figure 1: The real part of a 2-D Gabor wavelet, and its 2-D Fourier transform.

10.3

Generalization to Two Dimensional Signals

An eective strategy for extracting both coherent and incoherent image structure is the computation of two-dimensional Gabor coecients for the image. This family of 2-D lters were originally proposed as a framework for understanding the orientation-selective and spatial-frequency-selective receptive eld properties of neurons in the brains visual cortex, and as useful operators for practical image analysis problems. These 2-D lters are conjointly optimal in providing the maximum possible resolution both for information about the spatial frequency and orientation of image structure (in a sense what), simultaneously with information about 2-D position (where). The 2-D Gabor lter family uniquely achieves the theoretical lower bound for joint uncertainty over these four variables, as dictated by the inescapable Uncertainty Principle when generalized to four-dimensional space. These properties are particularly useful for texture analysis because of the 2-D spectral specicity of texture as well as its variation with 2-D spatial position. A rapid method for obtaining the required coecients on these elementary expansion functions for the purpose of representing any image completely by its 2-D Gabor Transform, despite the nonorthogonality of the expansion basis, is possible through the use of a relaxation neural network. A large and growing literature now exists on the ecient use of this non-orthogonal expansion basis and its applications. Two-dimensional Gabor lters over the image domain (x, y) have the functional form

f (x, y) = e[(xx0 )

2 /2 +(yy

0)

2 / 2

] ei[u0 (xx0 )+v0 (yy0 )]

(17)

where (x0 , y0 ) specify position in the image, (, ) specify eective width and length, and
2 (u0 , v0 ) specify modulation, which has spatial frequency 0 = u2 + v0 and direction 0 = 0 arctan(v0 /u0 ). (A further degree-of-freedom not included above is the relative orientation of the elliptic Gaussian envelope, which creates cross-terms in xy.) The 2-D Fourier transform

68

F (u, v) of a 2-D Gabor lter has exactly the same functional form, with parameters just interchanged or inverted:

F (u, v) = e[(uu0 )

2 2 +(vv

0)

22

] ei[x0 (uu0 )+y0 (vv0 )]

(18)

The real part of one member of the 2-D Gabor lter family, centered at the origin (x 0 , y0 ) = (0, 0) and with unity aspect ratio / = 1 is shown in the gure, together with its 2-D Fourier transform F (u, v). 2-D Gabor functions can form a complete self-similar 2-D wavelet expansion basis, with the requirements of orthogonality and strictly compact support relaxed, by appropriate parameterization for dilation, rotation, and translation. If we take (x, y) to be a chosen generic 2-D Gabor wavelet, then we can generate from this one member a complete selfsimilar family of 2-D wavelets through the generating function: mpq (x, y) = 22m (x , y ) (19)

where the substituted variables (x , y ) incorporate dilations in size by 2m , translations in position (p, q), and rotations through orientation : y = 2m [x sin() + y cos()] q x = 2m [x cos() + y sin()] p (20) (21)

It is noteworthy that as consequences of the similarity theorem, shift theorem, and modulation theorem of 2-D Fourier analysis, together with the rotation isomorphism of the 2-D Fourier transform, all of these eects of the generating function applied to a 2-D Gabor mother wavelet (x, y) = f (x, y) have corresponding identical or reciprocal eects on its 2-D Fourier transform F (u, v). These properties of self-similarity can be exploited when constructing ecient, compact, multi-scale codes for image structure.

10.4

Grand Unication of Domains: an Entente Cordiale

Until now we have viewed the space domain and the Fourier domain as somehow opposite, and incompatible, domains of representation. (Their variables are reciprocals; and the Uncertainty Principle declares that improving the resolution in either domain must reduce it in the other.) But we now can see that the Gabor domain of representation actually embraces and unies both of these other two domains. To compute the representation of a signal or of data in the Gabor domain, we nd its expansion in terms of elementary functions having the form

f (x) = eik0 x e(xx0 )

2 /a2

(22)

The single parameter a (the space-constant in the Gaussian term) actually builds a continuous bridge between the two domains: if the parameter a is made very large, then the second exponential above approaches 1.0, and so in the limit our expansion basis becomes
a

lim f (x) = eik0 x

(23)

the ordinary Fourier basis. If the frequency parameter k0 and the size parameter a are instead made very small, the Gaussian term becomes the approximation to a delta function at location xo , and so our expansion basis implements pure space-domain sampling:
k0 ,a0

lim f (x) = (x x0 )

(24)

Hence the Gabor expansion basis contains both domains at once. It allows us to make a continuous deformation that selects a representation lying anywhere on a one-parameter continuum between two domains that were hitherto distinct and mutually unapproachable. A new Entente Cordiale, indeed. 69

Reconstruction of Lena: 25, 100, 500, and 10,000 Two-Dimensional Gabor Wavelets

Figure 2: Illustration of the completeness of 2-D Gabor wavelets as basis functions.

70

11

Kolmogorov Complexity and Minimal Description Length

An idea of fundamental importance is the measure known as Kolmogorov complexity: the complexity of a string of data is dened as the length of the shortest binary program for computing the string. Thus the complexity is the datas minimal description length. It is an amazing fact that the Kolmogorov complexity K of a string is approximately equal to the entropy H of the distribution from which the string is a randomly drawn sequence. Thus Kolmogorov descriptive complexity is intimately connected with information theory, and indeed K denes the ultimate data compression. Reducing the data to a program that generates it exactly is obviously a way of compressing it; and running that program is a way of decompressing it. Any set of data can be generated by a computer program, even if (in the worst case) that program simply consists of data statements. The length of such a program denes its algorithmic complexity. It is important to draw a clear distinction between the notions of computational complexity (measured by program execution time), and algorithmic complexity (measured by program length). Kolmogorov complexity is concerned with nding descriptions which minimize the latter. Little is known about how (in analogy with the optimal properties of Gabors elementary logons in the 2D Information Plane) one might try to minimize simultaneously along both of these orthogonal axes that form a Complexity Plane. Most sequences of length n (where most considers all possible permutations of n bits) have Kolmogorov complexity K close to n. The complexity of a truly random binary sequence is as long as the sequence itself. However, it is not clear how to be certain of discovering that a given string has a much lower complexity than its length. It might be clear that the string 0101010101010101010101010101010101010101010101010101010101010101 has a complexity much less than 32 bits; indeed, its complexity is the length of the program: Print 32 "01"s. But consider the string 0110101000001001111001100110011111110011101111001100100100001000 which looks random and passes most tests for randomness. How could you discover that this sequence is in fact just the binary expansion for the irrational number 2 1, and that therefore it can be specied extremely concisely? Fractals are examples of entities that look very complex but in fact are generated by very simple programs (i.e. iterations of a mapping). Therefore, the Kolmogorov complexity of fractals is nearly zero. A sequence x1 , x2 , x3 , ..., xn of length n is said to be algorithmically random if its Kolmogorov complexity is at least n (i.e. the shortest possible program that can generate the sequence is a listing of the sequence itself): K(x1 x2 x3 ...xn |n) n (25)

An innite string is dened to be incompressible if its Kolmogorov complexity, in the limit as the string gets arbitrarily long, approaches the length n of the string itself: K(x1 x2 x3 ...xn |n) =1 n n lim (26)

An interesting theorem, called the Strong Law of Large Numbers for Incompressible Sequences, asserts that the proportions of 0s and 1s in any incompressible string must be nearly equal! Moreover, any incompressible sequence must satisfy all computable statistical 71

tests for randomness. (Otherwise, identifying the statistical test for randomness that the string failed would reduce the descriptive complexity of the string, which contradicts its incompressibility.) Therefore the algorithmic test for randomness is the ultimate test, since it includes within it all other computable tests for randomness.

12

A Short Bibliography

Cover, T M and Thomas, J A (1991) Elements of Information Theory. New York: Wiley. Blahut, R E (1987) Principles and Practice of Information Theory. New York: AddisonWesley. Bracewell, R (1995) The Fourier Transform and its Applications. New York: McGrawHill. Fourier analysis: besides the above, an excellent on-line tutorial is available at: http://users.ox.ac.uk/ball0597/Fourier/ McEliece, R J (1977) The Theory of Information and Coding: A Mathematical Framework for Communication. Addison-Wesley. Reprinted (1984) by Cambridge University Press in Encyclopedia of Mathematics. McMullen, C W (1968) Communication Theory Principles. New York: MacMillan. Shannon, C and Weaver, W (1949) Mathematical Theory of Communication. Urbana: University of Illinois Press. Wiener, N (1948) Time Series. Cambridge: MIT Press.

72