Wavelets and Filter Banks Gilbert Strang Massachusetts Institute of Technology Truong Nguyen University of Wisconsin ADAHASAHAH ASAD IS eo 2 Wellesley-Cambridge Press Box 812060 Wellesley MA 02181Contents Preface ix Guide to the Book — xvii Chapter1 Introduction Ww 12 1.3 1.4 15 Overview and Notation 1 Lowpass Filter = Moving Average 7 Highpass Fitter = Moving Difference 12 Filter Bank = Lowpass and Highpass 15 Scaling Function and Wavelets 22 1.6 Wavelet Transforms by Multiresolution 28 Chapter 2 Filters 2.1. Signals, Samples, and Time-Invariance 37 2.2 Ideal Fitters, Shannon Sampling, Sinc Wavelets 45 2.3 Lowpass and Highpass Filter Design 53 2.4 Fourier Analysis 61 25 26 Bases andFrames 69 ‘Time, Frequency, and Scale 80 Chapter3 Downsampling and Upsampling 34 3.2 3.3 34 Matrices for Downsampling and Upsampling 87 ‘Subsampling in the Frequency Domain = 91 Sampling Operations in the 2-Domain 96 Filters interchanged with Samplers 100 Chapter 4 Filter Banks 41 42 4.3 44 45 Perfect Reconstruction 103 The Polyphase Matrix 4 Efficient Fitter Banks = 122 Polyphase for Upsampling and Reconstruction 128 Lattice Structure 134vi Wavelets and Filter Banks Chapter Orthogonal Filter Banks 5.1 5.2 5.3 5.4 5.5 Paraunitary Matrices 144 Orthonormal Filter Banks 147 Halfband Filters 153 Spectral Factorization 157 Maxflat (Daubechies) Filters 164 Chapter6 Multiresolution 61 62 6.3 6.4 65 The Idea of Multiresolution 174 Wavelets from Filters 186 Computing the Scaling Function by Recursion —_193 Infinite Product Formula 201 Biorthogonal Wavelets 208 Chapter7 Wavelet Theory TA 72 73 14 75 Accuracy of Approximation 221 The Cascade Algorithm for the Dilation Equation 234 ‘Smoothness of Scaling Functions and Wavelets 242 Splines and Semiorthogonal Wavelets 250 Muttifitters and Muttiwavelets 259 Chapter8 Finite Length Signals 8.4 82 83 8.4 8.5 Circular Convolution andthe DFT 263 ‘Symmetric Extension for Symmetric Filters 272 Cosine Bases andthe DCT 276 Smooth Local Cosine Bases 282 Boundary Filters and Wavelets 289 Chapter9 M-Channel Filter Banks 91 9.2 9.3 9.4 95 Freedom versus Structure 299 Polyphase Form: MChannels = 304 Perfect Reconstruction, Linear Phase, Orthogonality 316 Cosine-modulated Filter Banks 325 Multidimensional Fitters and Wavelets 331Contents Chapter 10 Design Methods 10.1 Distortions in Image Compression 337 10.2 Design Methods—General Perspective 343, 10.3 Design of Perfect Reconstruction Filter Banks 347 10.4 Design of Two-Channel Filter Banks 352 10.5 Design of Cosine-modulated Filter Banks 356 Chapter 11 Applications 11.1 Digitized Fingerprints andthe FBI = 362 11.2 Image and Video Compression 365 41.3 Speech, Audio, and ECG Compression 383 11.4 Shrinkage, Denoising, and Feature Detection 387 11.5 Communication Applications and Adaptive Systems 391 11.6 Wavelet Integrals for Differential Equations 394 Glossary 403 Appendix 1 Wavelets (American Scientist) 433 Appendix 2 Wavelets and Dilation Equations (SIAM Review) 440 MATLAB and the Wavelet Toolbox 453 References 475 Index 485Preface This book is about filter banks and wavelets. Those are new ways to see and represent a signal. They are alternatives to the Fourier transform, using short wavelets instead of long waves. We will explain the advantages (and disadvantages!) of the new methods. The final decision will depend on the application itself, the actual signal, and its bandwidth. To design and understand wavelets we still use Fourier techniques — the connections be- tween time and frequency. This idea remains at the center of signal processing. Wavelets are “alternatives” rather than replacements. The classical transforms will survive very well. But other ideas have come quickly forward, to be understood and applied. In a word, the new transforms are much more Jocal. An event stays connected to the time when it occurs. Instead of transforming a pure “time description” into a pure “frequency de- scription”, the new methods find a good compromise —a time-frequency description. This is like a musical score, with specified frequencies at specified times. Remember that the Heisen- berg uncertainty principle stands in the way of perfection! We lose accuracy in time when we gain accuracy in frequency. The musician cannot and would not change frequencies instantly. But our eyes and ears succeed to give location as well as frequency, and the new wavelet trans- forms have the same purpose. The extreme case is an instantaneous impulse, with all frequencies in equal amounts. Its Fourier transform has constant magnitude over the whole spectrum. By contrast, a wavelet trans- form will involve only a small fraction of the wavelets — those that overlap the impulse. Fig- ure 0.1 shows a sum of two extreme cases — an impulse and a pure wave 2 cos wn. The Fourier transform spreads the impulse while it concentrates the wave (at frequencies w and —w of eon + e-in) The wavelet transform in the third figure is large at the time of the impulse and also large at the frequency of the wave. Purpose of the Book There are already good books on this subject. The ideas and applications are beautiful, and the word has spread. The bibliography lists many of those books, which have special strengths. Our purpose is different. We believe that a textbook is needed. Our text explains filter banks and wavelets from the beginning —in several ways and at least two languages. The examples and exercises come from our courses at M.I.T. and Wisconsin, which brought students from all over engineering and science. Whether you are working individually or in class, we hope this book clarifies the central ideas. Implicit in that goal is the recognition that we cannot describe every filter and prove every theorem. The book concentrates on the underpinning of the subject, which is now stable. There is a special “glossary” to organize and define the terms that are constantly used, some from signal processing and others from mathematics. The central idea is a perfect reconstruction filter bank, with properties and purposes selected from this list:x Wavelets and Filter Banks 86-4) Figure 0.1: Impulse plus sinusoid, in time and frequency and time-frequency (time-scale). Properties: Orthogonality, symmetry, short length, good attenuation. Purposes: Audio/video compression, echo cancellation, radar, image analysis, communi- cations, medical imaging, ... . Design and analysis techniques: Time domain, frequency domain, z-transform. If the reader is willing, we would like to develop specific examples in this preface. Tradi- tionally, the opening pages thank those who helped to create the book. Our debt to friends will soon be very gratefully acknowledged. But first, we go directly to transforms. Transforms Start with the basic idea and its purpose. The transform of a signal (a vector) is a new repre- sentation of that signal. The components x(0), x(1), x(2), x(3) of a four-dimensional signal are replaced by four other numbers. Those numbers y(0), (1), ¥(2), (3) are combinations of the x's. Our transforms are linear, so these are linear combinations — for example sums and differ- ences: yO) = x0) +x(1) ¥(2) x(2) + x(3) yx) = x) —x(1) ¥(3) (2) — x(3). What is the purpose of the y’s? And can we get back to the x’s? The second question is easier and we answer it first. This transform can be inverted. If you add the equations for. ‘¥(0) and y(1), you get 2x(0). Subtracting those equations yields 2x(1). The inverse transform uses addition and subtraction (like the original!), and then division by 2: x0) = 0.570) + y(1)) #2) = 0.5(¥(2)+y(3)) x1) = 0.5) -y()) 23) = 0.52) —y(3)). wuPreface xi ‘The y’s allow perfect reconstruction of the x’s, by the inverse transform. Looking ahead for a brief moment, we divide transforms into three groups: (a) _Lessless (orthogonal) transforms (orthogonal and unitary matrices) (b) Invertible (biorthogonal) transforms (invertible matrices) (©) Lossy transforms (not invertible) A lossless unitary transform is like a rotation. The transformed signal has the same length as the original. This is true of the Fourier transform and all its real versions (DCT = discrete cosine transform, DST = discrete sine transform, HT = Hartley transform). The same signal is measured along new perpendicular axes. For biorthogonal transforms, lengths and angles may change. The new axes are not nec- essarily perpendicular, but no information is lost. Perfect reconstruction is still available. We just invert. Orthogonal wavelets give orthogonal matrices and unitary transforms, biorthogo- nal wavelets give invertible matrices and perfect reconstruction. These transforms don’t remove any information (or any noise), they just move it around — aiming to separate out the noise and decorrelate the signal. The irreversible step is to destroy small components, as we do below in “compression.” Then invertibility is lost. Matrices will appear very early, and we make no apology. A matrix displays the details of the transform. (The key texts in signal processing are amazingly empty of matrices. With the importance of systems like MATLAB, this must change.) Our book maintains a time-domain de- scription by matrices, in parallel with the frequency-domain description by functions of @. When the inputs x(n) and outputs y(n) are seen as vectors, the transform from x to y is executed by a matrix: yO) 1 10 #0 x(0) y(1) . 1-1 0 0 x(1) Oe x(2) ¥@Q) oS x3) The sum.x(0) +x(1) comes from the first row. The difference x(0)—x(1) comes from the second tow. Readers may recognize this as the 2-point DFT, and now comes its inverse. The matrix that recovers the x’s from the y’s is changed only by the factor 3: x(0) 1 1 0 0 yO) x(1) 1] 1 -1 0 0 yl) x(2) 2] 0 Oo. 1 y(2) x(3) 0 0 1 -l (3) If you multiply these matrices, the result is the 4 x 4 identity matrix. Small note When the first matrix is divided by V2, it becomes an orthogonal matrix. The tows of length 1+ 1 = J/2 will become unit vectors. The inner products between rows remain at zero. The transform becomes unitary (lossless) when the perpendicular axes —— the four rows — are correctly normalized. The inverse is the transpose. The length squared of the transform is now Oey ey : ey . (Oey, This simplifies to x(0)? +2(1)? +2(2)? +x(3)*. Thus [ly ll? = (|x|? and the transform is unitary. For real matrices we also use the word orthogonal.xii Wavelets and Filter Banks Purpose of the Transform One important purpose is to see patterns in the y’s that are not so clear in the x’s. This means that the computer should see the pattern. What it sees best is Jarge versus small. The computer will notice nothing special about the four numbers xOQV=1.2 x)= 10 x(2)=-1.0 4(3) = -1.2. Your eye notices various things in Figure 0.2 (I hope). Maybe the small movement and then the jump. Maybe the antisymmetry. To see this signal in the y fepresentation, compute sums and differences: yO) =2.2 y(1)=0.2 y2)=-22 y3)=0.2. 22 12 Transform 1.0 Original y@) signal x(n) . 02 02 4 a | 12 eee Figure 0.2: The x's transform to the y’s by sums and differences. First point: y(1) and y(3) are much smaller than y() and y(2). The differences are an order of magnitude smaller than the sums. Our transform has almost lined up the signal in the y(0)-y(2) plane. Those two components of y do most of the work of four. It is true that we need all four Js toreconstruct perfectly all four x’s. But if we change the small numbers y(1) and y(3) — just cancel them! — the compressed signal y, is JO) =2.2 y(I)=O y,2)=-22 9,3) =0. ‘Those numbers 0.2 were below our threshold. In the compressed ¥, they are gone. Figure 0.3 shows the signal x, reconstructed from ¥,~—the small difference between x(0) and (1) is lost. For comparison Me compute the 4-point DFT, expecting and seeing the imaginary number j=v-l: #0) fee 12 0 eC ye 10 |_| 220 +5 Ca) lee lie ate -1.0 | ~ 04 ¥3) ie ie ie -12 2.201 i) That zero in reflects the antisymmetry. To see it in the sum-difference transform, we must go to the next level.Preface xiii Reconstructed xe(n) —Ld = Figure 0.3: y, and x, are close to y and x, when small becomes zero. Multilevel Transforms by Recursion A key idea for wavelets is the concept of “scale.” Sums and differences of neighbors are at the finest scale, We move to a larger picture by taking sums and differences again, This is recur- sion—the same transform at a new scale. It leads to a multiresolution of the original signal (0), x(1), (2), x(3). Averages and details will appear at different scales. The wavelet formulation keeps the differences y(1) and y(3) at the finest level, and iterates onty on y(0) and y(2). Iteration means sum and difference of the transform: 2) =y(0)+59(2)=0 and z(2) = yO) —y(2) = 4.4. At this level there is extra compression. One component 2(2) now does most of the work of the original four. The wavelet transform z(0), y(1), z(2), y(3) is in Figure 0.4. Haar Transform Second Level} 44 Compressed | +4 22. oo 0 20) yl) 22) ¥) %(O) Yel) 2e(2) ¥-(3) Figure 0.4: z(0) and (2) are the sum and difference of y(0) and y(2). Compress to (2). The key point is the multilevel construction, which is clear in a flow-graph: > sumz(0) sums y(0), »(2) — i. i + difference 2(2) differences y(1), ¥(3) This two-step wavelet transform is executed by a product of two matrices: 1 1 1 1 0 0 1 1 1 1 1 1 -1 0 OF) ) 0. U°# 1 -1 o 01 ot famed ea 1-1 -! 1 0 oOo 1 -1 0 0 1-1xiv Wavelets and Filter Banks That is invertible! To draw the flow-graph of the inverse, just reverse the arrows. To invert the matrix on the right, multiply the inverses of the matrices on the left (in reverse order of course). The wavelet transform becomes unitary as before, when we divide sums and differences by V2. You will see this number appearing throughout the book, to compensate for scale changes. Perhaps one more transform could be mentioned. It is the Walsh-Hadamard transform, which iterates also on the differences y(1) and. '¥(3). Instead of the “logarithmic tree” for wavelets, we have a complete binary tree for Walsh-Hadamard: sum z(0) = 0 sums y(0) and y(2) difference z(2) = 4.4 sum z(1) = 0.4 differences y(1), y(3) difference z(3) = 0 This also counts as a wavelet packet (those include every binary tree). You would not want to miss the “Hadamard matrix” for this transform, which is exceptional. All entries are 1 and —1 and all rows are orthogonal: 1 1 1 1 1 1 1 -1 1 +1 tt ~1 -1 1 When divided twice by /2 this matrix is orthogonal and also symmetric. The original x’s can be reconstructed from the y’s or the w’s or the wavelet outputs y(1), ¥(3), (0), 2(2). From com- Pressed outputs we get approximate (but good) reconstruction. Don’t forget the disadvantage compared to wavelets! All sixteen entries are nonzero. The complete tree takes more computation than the wavelet tree, For signals of length n (a power of two), the Hadamard matrix has n? nonzeros and the wavelet matrix has only n+n log, n. Those are the costs without recursion, jumping from the original x’s directly to the final z’s. When the computations are recursive, as they always should be, we have a Product of very Sparse matrices. The flow-graph gives 2n log, n for the complete Walsh-Hadamard transform (exactly matching the Fast Fourier Transform). This is good but wavelets are slightly better. ‘The transform to z(0), y(1), z(2), ¥(3) computes the sum and difference of n — 1 pairs. This needs only 2n — 2 calls to memory. The wavelet transform achieves the Holy Grail of complexity theory (or simplicity theory). The transform is an O(n) computation. But does it separate the true signal from noise? Does it allow compression of 4 :1 or 8:1 or 20:1? This sum-difference transform is analyzed further in Chapter 1 — it is such a good example — but we admit here that it is too simple. Better filters are needed, and they lead to better wavelets (still with O(n) operations). That is the subject of our book,Preface xv Applications The choice of waves or wavelets, Fourier analysis or filter analysis, depends on the signal. It must. Signals coming from different applications have different characteristics, It is helpful to see a broad picture: Audio: Use many subband filters or a windowed (short time) Fourier transform. Speech: The time variation is irregular and requires nonuniform intervals, Images: Finite length signals need special treatment to reduce blocking. Symmetric extension goes with symmetric filters. Video: Use motion prediction (optical flow of images) or space-time filtering. The best choice for medical imaging is not clear, The legal questions arising from lossy com- Pression are not clear either. Identification from synthetic aperture radar (SAR images) is an enormous problem. So is de-noising, which is at the heart of signal representation. Ultimately we are trying to choose a good basis, The problem is to represent typical signals with a small number of conveniently computable functions. The traditional bases (Fourier, Bessel, Legendre, ...) come from differential equa- tions. Wavelets do not come from differential equations! One reason is, those equations don’t include dilation. A dilation equation p(t) = S>2h(k)p (21 — k) involves two time scales. Its solution $(¢) is nonzero only ona finite interval. Then $(2t) is nonzero on half of that interval, The basis is localized. It is quickly (and recursively) computed from the numbers Atk). Those are the coefficients in the corresponding lowpass filter. This relation of filters to functions is the heart of the book. Acknowledgments Many friends have created this theory. A lot of them also helped us to write about it. The bibli- ography points to their original work (we wish it were possible to be complete — that is better attempted electronically). Our personal thanks must begin with Vasily Strela and Chris Heil. They read most of the words and discussed all the ideas. Their unselfish help is deeply appreci- ated. Other friends will see, at specific places in the book, exactly where their ideas and sugges- tions made a difference. We are enormously grateful to Kevin Amaratunga, Ross Barmish, Christopher Brislawn, Charles Chui, Albert Cohen, Adriannus Djohan, Dave Donoho, Jeff Geronimo, Doug Hardin, Peter Heller, Tom Hopper, Angela Kunoth, Seng Luan Lee, Michael Lightstone, Eric Majani, Stéphane Mallat, Henrique Malvar, Ricardo de Queiroz, Jianhong Shen, Wim Sweldens, Pankaj Topiwala, Steffen Trautmann, Chi Wah Kok, Victor Wickerhauser, and Zhifeng Zhang. There is no way to thank you enough for such generous help. ‘We want to recognize separately the outstanding leadership of four authors, P. P. Vaidya- nathan and Martin Vetterli in signal processing and Ingrid Daubechies and Yves Meyer in math- ematics. The two subjects are thoroughly mixed, thanks to these four! Their work has been a model in every way, personal (to us) as well as scientific (for all). Our writing of the book was shared in a natural way — words mostly from the first author, filter designs and applications from the second author, ideas from both, Vasily Strela and Robert Becker helped to get Versions 1.0 to 9.9 into ‘TEX, and Martin Stock made it more beautiful. ‘The whole project grew out of our courses and workshops (which will continue). Those were tremendous sources of inspiration, thanks to the Participants,xvi Wavelets and Filter Banks This subject is not just theory. The ideas have to be implemented, and MATLAB is the out- standing way to do that. It is a pleasure to have this book closely linked to the Wavelet Toolbox offered by MathWorks. Exercises that use this Toolbox are in the book; the reader can see the wavelets and the multiresolution they produce. A more extensive Wavelet Manual will come from the same sources: Wellesley-Cambridge Press and MathWorks. For correspondence about this whole subject, and for comments and corrections of any kind, the authors thank the readers. Especially we thank Jill and Thuy Duong for their patience and support. It is finished at last! We hope you enjoy this book. Gilbert Stang and Truong Nguyen M.LT. and Wisconsin, November 1995 [email protected] and [email protected] http: //saigon.ece.wisc.edu/~waveweb/Tutorials/book. htmlGuide to the Book This book has a two-part subject. One part is discrete, the other is continuous. In discrete time we develop the idea and applications of filter banks. In continuous time we have scaling fune- tions $(t) and wavelets w(t). By a natural limiting process, iteration of the lowpass filter leads to the scaling function. One highpass filter then produces a wavelet. Our goal is to make this con- nection clear. We find the conditions on the discrete coefficients that lead to good filter banks and good wavelets. Historically and mathematically, the filters come first. Perfect reconstruction filter banks were developed in the early 1980’s. The excitement around wavelets started later (and grew quickly). This excitement was not universal — designers of filter banks naturally asked what was new. Part of the answer is precisely in that process of iteration. For a filter to behave well in practice, when it is combined with subsampling and repeated five times, it must have an extra property — not built into earlier designs. This property expresses itself in the frequency domain by a sufficient number of “zeros at 7”. Then the frequency band can be successfully separated into five octaves. The underlying problem is to choose a good basis. We want to represent a signal well, by a small number of basic signals. These can be sinusoids and they can be wavelets. On a dis- crete grid, » = 2 is the highest frequency at which a signal can oscillate. Those oscillations x(n) = e™ (—1)" are stopped by the lowpass filter with a “zero at 2”. The highpass fil- ter lets fast oscillations through, and the synthesis filters can reconstruct the exact input. But compression may come between analysis and synthesis. Frequencies that are barely represented will be intentionally lost. That mostly means high frequencies but the filter bank is impartial — it keeps the basis functions that are important to the specific signal. We want to show when, and why, filter banks and wavelets are effective in reconstruction and signal representation and compression, Filter Banks Some readers will begin this book with Chapter 1. Others will jump forward to a topic of partic- ular interest. This brief guide is for both, especially to tell the first readers what is coming and the second group where to look. We are pointing to places where preparation and explanation come together, to design and study new structures. For filter banks, that place is Section 4.1. There we identify the two conditions for perfect re- construction (in the absence of lossy compression). One condition removes distortion, the other condition removes aliasing. The anti-distortion condition applies to the products FoHo and FA, along the channels of the filter bank. Then the anti-aliasing condition controls how those prod- ucts can be separated into the four filters. The design of a perfect reconstruction filter bank is a choice of Foy and then a factorization. To understand the conditions on distortion and aliasing, we apply the techniques of multirate filtering. Those techniques are explained in Chapters 1-3, with examples throughout. Of coursexviii Wavelets and Filter Banks filters are to be defined! But we can go forward even now, to illustrate a filter bank that gives perfect reconstruction. The analysis bank is on the left. It has a lowpass filter Ho, and a highpass filter Hy, and dec- imation by (| 2) — which removes the odd-numbered components after filtering. The analysis bank yields two “half-length” outputs. Then the synthesis bank on the right begins with the up- sampling operation (2) which inserts zeros in those odd components: Ho 12 oe 12} Fo a, Hizb» 4i2H Fy ‘Two-channel filter bank; Separate the input into frequency bands (filler and downsample). Then reassemble (upsample and filter). input x — L— output & The gap in the center indicates where the subband signals are compressed or enhanced. The applications of this structure are extremely widespread. We believe that any reader interested in signal processing (and image processing) will find that filter bank analysis is extremely useful. The filters Ho, Hi, Fo, and F, are linear and time-invariant. The operators (| 2) and (¢ 2) are not time-invariant. These multirate operations are responsible for aliasing and for imaging — they create undesirable and extraneous signals that the filters must cancel. To understand how that happens, and to design good filters, we use the tools developed in Chapters 1-3: especially transformation to the frequency domain and z-domain. We try to explain the analysis of multirate filtering, with (| 2) and (¢ 2), clearly and memorably. The theory and the design of filter banks and wavelets will dominate Chapters 4-6. This is the heart of the book. The structure of an orthogonal bank is very special, and the next figure shows how the filters are related. For length 4 all filters use the four coefficients a, b, c,d that Daubechies derived: of! abd Hi we t2H debe 4-e.b,-af-] 2 oe [21H -ab,-c.4d ‘The form of an orthogonal filter bank with four coefficients. x(a-3) How did she choose a, b, c,d? Part of the answer will have to wait, but here is the essential idea. The product along the top channel gives a particular “halfband filter”: (@, b, c, d) * (d, ¢, b, a) = (-1, 0, 9, 16, 9, 0, -1)/16. This convolution is a multiplication of two polynomials, when a, b, c, d are the coefficients: (at bz 427 +z) d + ee"! + bz? +.az> (-14. 927 + 1627 + 92-4 - 2/16, The four coefficients are pleasant to calculate. The serious job in Chapter 4 is to explain what is special about that 6"-degree polynomial in which z~! and z~ are missing.Guide to the Book xix A filter bank also gives perfect reconstruction if it is biorthogonal. This design is less re- stricted. The product FoHy must skip the same odd powers of z~!, but Fo does not have to be the transpose (the flip) of Ho. Here are specific numbers for the filter coefficients — not the only choice and maybe not the best. They show how the filters Fy and F; on the synthesis side are related to the analysis filters H, and Hp (by alternating signs): ~ tel A biorthogonal filter bank: Perfect reconstruction with 3 delays. 16 x(n 3) For filters, we stop here. This time Fo(z) = 1 + 2z~! + z~?. Multiplied by Ho(z) it gives the same important 6"-degree polynomial as before. To understand why the zero coefficients are necessary in that polynomial, and why —;4, and % are desirable, I am afraid that you have to tead the book! Our discussion went this far (further than we intended) so as to make a basic and encouraging point: The construction of new filter banks need not be complicated. This subject is accessible to new ideas and experiments. Wavelets Wavelets are localized waves. Instead of oscillating forever, they drop to zero. They come from the iteration of filters (with rescaling). The link between discrete-time filters and continuous- time wavelets is in the limit of a logarithmic filter tree: _~ Scaling function (t) pe ‘ Wavelet w(t) f(t) da Hy da Ay Scaling function and wavelets from iteration of the lowpass filter. Scaling functions and wavelets have remarkable properties. They inherit orthogonality, or biorthogonality, from the filter bank. Because of the repeated rescaling that produces them, wavelets decompose a signal into details at all scales. The wavelet w(t) and its shifts w(t—k) are at unit scale. The wavelets w(2/t) and w(2/t — k) are at scale 2~/. The biorthogonal functions ¢(t) and &(r) come from iterating the synthesis bank. Wavelets produce a natural multiresolution of every image, including the all-important edges. Where the low frequency part of the Fourier transform is often a blur, the output from the lowpass channel is a useful compression,xx Wavelets and Filter Banks Sections 5.5 and 6.2 study the particular wavelets created by Ingrid Daubechies, They are orthogonal, with the advantages and limitations that this property brings. Sections 4.1 and 6.5 study biorthogonal alternatives, which come from different factorizations of the same polyno- mial (as above). This polynomial corresponds to a “maxflat halfband filter”, and we hope you will like the connections. More than that, we hope you enjoy the whole book. This subject is a beautiful combination of mathematical analysis and signal processing applications. The analysis and the applications are based on designs that give perfect reconstruction. To explain both sides of this subject, we need words from mathematics and words from digital signal processing. The Glossary at the end is a dictionary of their meanings. Above all we need ideas from both sides, and from a tremendous tange of application areas. It is to the understanding of filters and wavelets, and the growth of successful applications, that this book is dedicated. Summary of the Theory ‘There are four conditions that play a central part in this book. Because of their importance we highlight them here. They apply directly to the coefficients in the filter banks — and the conse- quences are felt (after iteration!) in the scaling functions and wavelets. Here are the four condi- tions — some might say in decreasing order of importance: PR Condition Perfect reconstruction. ‘The synthesis bank inverts the analysis bank, with £ delays. Biorthogonal banks with no aliasing and no distortion. Condition O Orthogonality. The analysis bank is inverted by its transpose. The wavelets are orthogonal to all their dilates and translates. Condition Ay Accuracy of order p for approximation by scaling functions $(t — k). p vanishing moments in the wavelets, pth order decay of wavelet coefficients for smooth f(r). Condition E Eigenvalue condition on the cascade algorithm. Determines convergence to #(t) and smoothness of wavelets. Equivalent to stability of the wavelet basis. One step further and this Guide is ended. The four fundamental conditions will be stated explicitly for a two-channel filter bank, with the sections in which they appear. We continue to use the polynomials Ho(z), 4) (z), Fo(z), and F,(z), whose coefficients come directly from the filters. By convention, these are polynomials in z~', and the lowpass analysis filter is Tepresented. by Ho(z) = h(0) + A(I)z7! + --- + #(N)z~%. Here are the conditions that give filters and wavelets with good properties: 1. Perfect Reconstruction (PR condition in Section 4.1) Fo(2) Ho(z) + Fi(z)Hi(z) = 22"© and Fo(z)Ho(—2) + Fi(z)Hy(—z) = 0. ‘The second equation gives the anti-aliasing choices Fo(z) = Hi(—z) and F\(z) = —Ho(—z).Guide to the Book xxi 2, Orthogonality (Condition O in Section 5.3) The filter coefficients are reversed by Fo(z) = 2% Ho(z~') and Fy(z) = 2" Hy(z-'). Then perfect reconstruction depends on the “double-shift orthogonality” of the lowpass coefficients Atk): DAW hk + 2n) = 6(n). In terms of the polynomials this is Ho(z) Ho(2~!) + Ho(—z)Ho(—27') = 2. 3. Accuracy of order p (Condition Ay in Sections 5.5 and 7.1) The lowpass filter has a zero of order p at z = —1: 1t+27! Hoe) = (FS )’ oe. 4, Convergence and Stability (Condition E in Section 7.2) The transition matrix T has 4 = | as simple eigenvalue and all other {A(7)| < 1. Final note: The sixth degree polynomial in the examples above has four zeros at z = —1: M1492 + 16273 4 924 = 28 = 2 (Hae = 2), ‘These zeros give flat responses near @ = 7 and also w = 0. The absence of z~! and z~5 is the key to perfect reconstruction. Polynomials of higher degree, also with zeros at z = —1 and also with only one odd power, factor into Fo(z) Ho(z) to give the best filters for iteration. In the limit of the iterations, these filters give good wavelets.Filters and Matrices h=(b©), A(N)) — Causal FIR lowpass filter (impulse response) with }7A(k) = 1 H_ Toeplitz matrix with h(k) on the kth diagonal: Hj = h(i — j) H(o) = Dh(k) et (frequency response in reduced notation) H(el®) = ak) eH? (signal processing notation) H(z) = Ch(k)z* Transfer function in the z-domain with z = e/® y = Hx = h * x corresponds to Y(z) = H(z)X(z) Convolution Rule S Delay gives (Sh)(k) =h(k— 1) Time Invariance is SH = HS s-! Advance gives (S~'h)(k) = A(k-+ 1) Multiply transform by z = e~/” (12) Downsampling operator (| 2)h = even = (h(0), h(2), (4), « (L2)S71h = Rods = (A(1), 4(3), A(5), - even phase of h ) Odd phase of h H(z) = [Heven(2) Hoda(2)] = [La(2K)z“* = hk +12] Polyphase representation (t2) Upsampling operator (+2) = [»(0) 0 v(1) 0 v(2) 0 ---] has z-transform V (2?) (42)()2)x = [x(0) 0 x(2) 0 x(4) OJ has z-transform 3[X (z) + X(—z)] = Xeven(2) H(z7') — Transpose filter with coefficients h(—k): anticausal matrix H” zNH(z-!) Flip to h(N — k) produces (h(N), ..., (2), (1), h(0)) H(-z) Alternating sign (—1)*h(k) produces (h(0), —A(1), 4(2), —h(3), ...) 2" H(—z7!) Alternating flip (—1)‘h(N — k) produces (A(N), ... , (—1)%A(0)) h(k) =h(N —k) Symmetric filter (W symmetry for odd N and H symmetry for even N). Filter Banks L=(}2)C = (42)V2Ho — Lowpass analysis channel: Filter and downsample B=({2)D =({2)V2H; — Highpass analysis channel: Filter has )° hy(k) = 0 J2Fo(42) — Lowpass synthesis channel: Upsample and filter with Fo(z) = H,(—z) V2F\(t2) — Highpass synthesis channel: Upsample and filter with F(z) = —Ho(—z) F(z) Ho(z) — Fo(—z) Ho(—z) = Po(z) — Po(—z) = 2z~£ No distortion: @ delays (odd é) —| Aoeven(z) — Ho,oaa(z) —| Foeven(z) Fi,even(z) Polyphase 4,0) =| Hieven(2) Hioaa(2) | aed vo =| Fooda2) Fi.oaa(2) | matrices _ | Ao(z) Ho(—z) _ Fo(z) Fi(z) Modulation a) =| A(z) Ai (-z) | o Fa=[ Fo(—z) -Fi(~z) | matrices F,(z)Hp(z) =1 Perfect Reconstruction with no delays 22-¢ 0 Fn (@)Hn(2) = [ oe ] Perfect Reconstruction with ¢ delays (odd £) AY (z-')Hp(@) =1 H(z) is paraunitary and the filter bank is orthogonal Dho(n)ho(n + 2k) = 6(k) and Ho(z)Ho(z~!) + Ho(—z)Ho(—z-!) =2 — Orthogonal lowpass X(-Dtk"ho(k) = 0 forO 2h(k)p (2t — k) w(t) from the wavelet equation w(t) = > 2h (k)p(2t — k) wye(t) = 24? w(2/t — k): Normalized wavelet on [kAt, (k + N)At] Scale parameter j for stepsize At = 2-/ (At = 2/ in [D] and MATLAB Toolbox) Shift-invariant subspaces Vj C Vj41 with Vj ® Wj = Visi: f(t) © Vo f(2/t) € Vy {¢(t—k)} An orthonormal basis (or only a Riesz basis) for Vo {w(t —k)} An orthonormal basis (or only a Riesz basis) for Wo {2#?¢(/1-k)} and {2/w(2/t—k)} Bases for V; and Wy. Joint basis for Vj41 Orthogonal multiresolution Orthonormal bases with V; perpendicular to W; Semiorthogonal multiresolution _ Riesz bases with V; perpendicular to W; Biorthogonal multiresolution _Biorthogonal bases with V; 1 W, and W; 1 Vj 9(t) and &(t) from the analysis filters ho and hy generate Vo ® Wo = V1 $(t) and w(t) from the synthesis filters fy and f, generate Vo ® Wo = Vi (@¢ —k), Ot - ) = 6(k— 1) Biorthogonal (dual) bases for Vo and %o (w(t —k), Bt — £)) = 6(k—1) _ Biorthogonal (dual) bases for Wo and Wo il ap = (£0), Fat) Coefficients in f;(t) = Dy aje$jx(¢) = projection of f(t) onto V; (f(t), Dje(t)) Wavelet coefficients in f(t) = LV opwp.(t) il Dix jp = So hol — 2k) ajyi.¢ and bj, = Doi ( — 2k) aj41,2 Mallat Fast Wavelet Transform Oj 41.0 = Dfoll — 2k) ajx + VF (E — 2k) bj, Fast Inverse Wavelet TransformDilation Equation — Solution and Smoothness $(@) = H($) $($) Fourier transform of the dilation equation $(o) = 11%, H(S) Fourier transform of the scaling function $(¢) GP) =P 2) (2t—k) Cascade algorithm with ¢(t) = box function M=({2)2H Cascade matrix with double-shifted rows: M = /2L m(0) and (1) N by N submatrices of M Entries 2h(2i — j) and 2h(2i — j +1) m(0) = — Eigenvector with A = 1 gives @ = (6(0), (1), ...) at the integers ) m(t)) m(t2) --- m(t) All products uniformly bounded <» Bounded recursion for o(t) O61 +) = m(4)® (2 ++) Recursion for ©(t) = (¢(1), O(¢ +1), ...) at dyad T=({2)2HH? Transition matrix from autocorrelation h *h? corresponding to |H (w)| Ta =a __ Eigenvector of T gives the inner products a(k) = (p(t), o(t +k)) A(z) = Da(k)z* and A(w) = Y [w+ 2k)? Buler-Frobenius polynomial from h(k A(@) = 1 and A(z) =1anda=6 Orthonormal basis {#(t — k)} 0 > At=2! t Aw, At fixed Figure 1.1: Time-frequency squares for Fourier decompositions become rectangles for wavelets. Short time intervals are natural for high frequencies Its vertical placement gives frequency, its horizontal placement indicates time. A musical score is almost a wavelet decomposition — except that it has fractional jumps in frequency. There are notes between middle C and high C, while wavelets jump by octaves. The shortest note I have seen is a 32nd note, corresponding to level j = 5 (because 2~> = 4). Wavelets often stop there too, in practice. But in principle the scale level j goes to infinity. That article on wavelets was written for a nontechnical audience, but it aims to explain the essential ideas. In the wavelet decomposition, all instruments play the same tune! They have different amplitudes and they play at different speeds and different times. The basses contribute the coarsest signals bo, w(t — k), starting at integer times f = k and overlapping. The cellos play an identical tune but twice as fast. They contribute b,,w(2r — k), starting at half-integer times t = k/2 and again overlapping. The violas and violins add details at levels j = 2 and j = 3. Those details are wavelets w(4r) and w(8r) and their translates. It is the orthogonality of all these tunes, and especially the localization of each tune into a short time interval, that makes it possible to decompose the symphony efficiently. Similarly it is orthogonality (or biorthogonality) and localization that make wavelet decom- positions attractive for other signals. Frequency Domain and Notation To see a filter as a multiplication, we must take Fourier transforms. This will be the discrete-time Fourier transform, since the vectors x(n) and h(n) and y(n) are discrete. The time index n goes from —oo to oo. (A vector with zero components at all negative times is called causal.) The transform of x has two reasonable notations. They both stand for the same transform, which we denote by X: Xe”) = Ye x(n ei" (signal processing notation) Xo) = So x(n)en™ (reduced notation).1.1 Overview and Notation 5 You see two differences. On the right side, one uses j and the other uses i. Both represent /—1. On the left side, the standard signal processing notation uses e/® while the reduced notation writes only w. (Each has advantages and we are prepared to print the book both ways!) The standard notation allows a direction conversion of the Fourier transform to the z-trans- form. The transform is still X but the variable becomes z: X@ = Dox) 2". We simply replace e/” by z, extending the formal definition of X from e/® on the unit circle to z in the whole complex plane. (Remember: e/” has magnitude 1.) The Fourier transform will dominate the first part of the book, but the z-transform appears more frequently in the end. The reduced notation has the advantage of outstanding simplicity. There will be many, many occasions to write X(w) and X(w + 2) or to write X(e/”) and X(e/°*”), The first occasion is the most important right now, and we want to express the action of a filter both ways: Convolution by h in time becomes multiplication by H in frequency: Y(el) = H(e!®) X(e!”) _ in signal processing notation Y@) = H(w) X(@) in reduced notation. This is the transform of y(n) = }°h(k) x(n — k). Exercise 10 will ask you to verify this funda- mental fact. It is the “convolution rule”. In the z-domain it becomes Y(z) = H(z) X(z). The only inputs to the proof are the definition of convolution and of the transform. Decision on notation. Simplicity often wins. We keep the freedom to write X (w) rather than X(e/®), In reduced notation, the frequency response is H(w): H(o) = Do ane”. This is the response at frequency « to a unit input at that frequency. When the input at each frequency is X(w) = 1, the output at each frequency is H(«). Those inputs are coming from an impulse (all X (@) are equal). Then the frequency response is the transform of the impulse response: When the input is a unit impulse x(0) = 1, the output is y(n) = h(n). When the input is a unit impulse X (w) = 1, the output is Y(w) = H(o). Please note the difference between the vector x = (...,0,0,1,0,0,...) in discrete time and the function X(w) = 1 in continuous frequency. If the impulse is delayed to time ¢ = 1, the input is (...,0,0,0, 1,0, ...). The output is equally delayed to y(n) = h(n — 1). This is time- invariance delay of the input just produces a delay of the output. In the frequency domain, the delayed impulse is X(w) = e~*. The corresponding output is ¥(w) =e H(@). Ine! notation this is ¥ (e/) = e~/°H (e/®). Itis the transform of the delayed impuse response h(n — 1).Chapter 1 Introduction Convolution by Hand A good way to compute y = fx is to arrange it as an ordinary multiplication — but don’t carry digits from one column to the next: The coefficients x(n) = 4, 2, 3 add to X(0) = 9. The sum of h(n) = 2, 5, 1 is H(O) = 8. x(2) x(1) x(0) 3 24=x hQ) A) h(O) 1 52=h Qe OD) © 6 48 3B) 2 W 15 10 20 @ 6) @ Sect yA) y3) yQ) yA) yO) 3.17 20 248 = y Then notice that the sum for y = h «x is H(0)X (0) = 72. Another check is at@ = a. The alternating sum 4 — 2 + 3 gives X(z) = 5S. Similarly H (x) = —2. Then necessarily Y (7) = (5)(—2). This agrees with the alternating sum 3 — 17+ 20—24+8=—10. Problem Set 1.1 . Suppose the only nonzero components of x and A are x(0) = 1, x(1) = 3, and h(O) = 4, h(1) = 4. Compute the outputs y(n). Verify in the frequency domain that ¥(w) H(w)X (w). . What are the components /() for the filter to become a simple advance? For any input vec- tor x(n), the output is y(n) = x(n + 1). Find H(o) for this filter and verify that ¥(o) = H(w)X(@). . If the input filter vector x and the vector h are both causal, explain why the output y is also causal (meaning that y(n) = 0 for negative m). If h is causal and if x(n) = 0 for all n < 8 what can you conclude about y(n)? . If the output vector y is causal whenever the input x is causal, explain why the vector hk must be causal. . If -=(...,0,0, 1,0, 0, ...) is the unit impulse at time zero, show that convolution with any vector v leaves that vector unchanged. Translate this statement v * - = v into the frequency domain. . What are the seven components of ft h, if h(0) = h(1) = h(2) = h(3) = 1 (all other h(n) = 0)? Use the long multiplication format in the text. . The long multiplication format corresponds to thinking of h and x as h() + 10A(1) + 100h(2) + --- and x(0) + 10x(1) + 100x(2) + ---. The product begins with h(0)x(0) + 10(__) + 100(___). This is the convolution rule Y(z) = H (z)X (z) with z = 10. . Verify the convolution rule Y = HX in the important special case when A(1) = 1 and all other h(n) = 0. Thus, h = (...,0,0,0, 1,0,...). 1. Whatisy = hex =h* (...,2(-1),4),4(D),...)? . What is H (e/), also written H(w)? . In this special case the filter H represents a___. . Verify that y(n) e™ agrees with H (e/”) X (e/”). kwon1.2 Lowpass Filter = Moving Average fg 9. Repeating the previous exercise k times shows that ¥ = HX is still correct when H is a k-step delay: h(k) = 1 and all other h(n) = 0. The frequency response for this delay is An arbitrary vector h is a linear combination of delays (and also advances!). By linearity, HX is true in general. 10. (Important) A direct approach to the convolution rule ¥Y = HX. What is the coefficient of 2" in (ACK) <~/) (Sx(€) 2“)? Show that your answer agrees with 7 h(k) x(n — k). ‘The exponent —n appears in H times X when k +/ equals —__. 11. The autocorrelation p = h * h” is the convolution of h with its time reversal (or transpose): h(n) = h(—n). Express the kth component p(k) as a sum of terms. 1.2 Lowpass Filter = Moving Average We go forward with this introduction by studying the simplest lowpass filter. Its output at time t = nis the average of the input x() at that time and the input x(n — 1) at the previous time: y(n) = 4x(n) + $x(n — 1). (1.4) The filter coefficients are h() = 4 and h(1) = 4. Equation (1.4) fits the standard form Lh’) x(n — k), with only two terms k = 0 and k = 1 in the sum. This is a convolution y =h xx. Itis amoving average, because the output averages the current component x() with the previous one. Old components drop away as the average moves forward with the time. Suppose the input is the unit impulse x = (..., 0,0, 1,0,0,...). Then there are only two nonzero components in the output. The input vector has x(0) = 1; all other input components are zero. The output vector has y(0) = 3, from equation (1.4) with n = 0. It also has y(1) = 5, from equation (1.4) with n = 1. All other outputs, or moving averages, are zero. Thus the impulse response is the vector y = (....0,0, 5, 5.0,...). Its components agree with the filter coefficients h(n) as they should. ‘We want to see this filter as a linear time-invariant operator. It is a combination of two special operators, the “identity” which yields output = input and the “delay” whose output is the input one time earlier: averaging filter = } (identity) + } (delay). Every linear operator acting on the signal vector x can be represented by a matrix. Since the vectors are infinite, so are the matrices. Infinitely many components in x and y mean infinitely many entries in the filter matrix H. The matrix has a special structure which you see immediately iny = Hx: yD) $ x(-1) yO) = eo x(0) Ht y) x(1) The numbers + on the main diagonal come from 5 (identity). The numbers } on the subdiagonal come from $ (delay). Substitute the unit impulse for x, with x(0) = 1 as its only nonzero com- ponent. Matrix multiplication produces the impulse response y. This vector has components +8 Chapter 1 Introduction and 3 The response y is the filter vector #, in the middle column of the matrix, when x is the unit impulse. This is a first occasion to see a filter as a constant-diagonal matrix. The coefficient h(0) appears constantly down the main diagonal. It represents (0) times the identity matrix and it yields h(0).x(n). The coefficient h(1) appears down the first subdiagonal, to represent (1) times a delay and to yield A(1) x(n — 1). If there is a coefficient h(2) down the next diagonal, it multi- plies a two-step delay and yields h(2).x(n — 2). The total output y(7) is the sum of these special outputs: yn) = AO) x(n) +h) x(n — 2) + hQ)x(n — 2) +++ Law xn —k). E The reader notices that h(—1) is not being allowed. We are dealing with causal filters; the output cannot come earlier than the input. This makes h(n) = 0 for negative n, and it makes the filter matrix lower triangular. Our example has only a finite number (two) of nonzero filter coefficients h(n). The filter has afinite impulse response. It is an “FIR filter”. For large n, the coefficients (11) that give distant responses to the unit impulse are all zero. To repeat: A causal FIR filter has (1) = 0 for all negative n and for large positive n. The matrix is banded and lower triangular. Only a finite number of coefficients h(0), A(1), ..., A(N) can be nonzero. The filter has N + 1 “taps”. The matrix has bandwidth N; all other diagonals contain zeros. We concentrate almost exclusively on causal FIR filters. Frequency Response To find the frequency response to the filter, we change the input vector. Instead of an impulse, which combines all frequencies, the vector x will have pure frequency «. Its components are x(n) =e", (1.5) This vector extends over all time, —oo < n < ox, Its special feature is that the output vector y is a multiple (depending on w) of the input vector x. A linear time-invariant operator, which is represented by a constant-diagonal matrix, has a pure frequency response to a pure frequency input. For our moving average this response is x(n) + $x(n = 1) eine 4d glinDe = (3+3e Mem. (1.6) y(n) You see in parentheses the frequency response function H(w) = 4 + 5e~*®. Notice that H = $+ 4 = 1 when = 0. At zero frequency (direct current) the signal x = (...,1, 1, 1,...) comes out unchanged as y = (..., 1, 1, 1,...). The low frequencies, near @ = 0, have H(w) near 1. Thus the name “lowpass filter”. There will be a similar H(«) for other filters. When the filter coefficients are h(n), the re- sponse to x(n) = e!® is y(n) = h(i? + (eV + ---, The input is multiplied by the frequency response H(w): Hw) = h(0)+h(e" +h(2) e714 Lame. (7)1.2 Lowpass Filter = Moving Average 9 This function is always periodic: H(@ +27) = H(w). When we add 27 to the frequency , we add 2777 to the angle ne. The cosine, sine and complex exponential e~!"® = cos nw —i sin no are not changed. Note that the response function H () involves complex numbers. We strongly prefer e’” or e/® (o its rectangular form cos w +i sin w. Separate formulas for the real and imaginary parts are much more complicated than a single formula for H(@). But a single graph cannot so easily represent this complex function. One way to do it is to plot the magnitude | (@)| separately from the phase angle ¢(w), recalling that H(w) =|H(o)|e@. Our example has H(w) = $+ $e7!. We factor out e~!*/? to leave a symmetric quantity $ (ec? + e-'/), This quantity is a perfect cosine: = (cos) e-tel2 H(w) = (cos 5 ) eo, (1.8) This displays the magnitude and also the phase: |H(o)| = cos $ and $() = 8. (1.9) Figure 1.2: The magnitude |H (w)| = cos (#) and the phase of H(w). Figure 1.2 shows the plot of the magnitude | (q)| against the frequency @. The cosine of $ drops to zero at @ = . This high frequency is wiped out, when the filter takes a moving average. In the time domain, w@ = x corresponds to the input vector x(n) = e'™" with components (—1)". This input vector is caer oy The moving average of these components is constantly zero! This confirms H (zr) = 0 as the correct response to the frequency w = 7. This is a lowpass filter. The lowest frequency @ = 0, which is the DC term (direct current), is exactly preserved because cos 0 = 1. The input vector (..., 1, 1, 1, ...) is equal to its moving average. The phase $(@) is the angle from the horizontal when the complex number H (@) is plotted in the complex plane. For this particular response, those points H(w) = 3 + fete lie along a circle. The constant term } is the center. Then $e~' produces a circle of radius $ around that center. Figure 1.3 shows this “Nyquist diagram”.10 Chapter 1 Introduction -io Figure 1.3: Nyquist diagram of the points H(o) in the complex plane. The graph of # (o) = —$ is a straight Line. This isan example of “Yinear phase”, an impor- eo property that some special filters possess. It reflects the fact that the filter coefficients + 3 and } are symmetric. Reverse the order of coefficients and nothing changes. Note that if the filter coefficients were symmetric around zero, so that h(—1) = h(1) and h(—2) = h(2), the frequency response would be real: H() h(O) +h(1) (e'® +e!) +--+ (symmetric coefficients) = h(O)+A(1) 2cos w) +--- (real response function) In this case the phase angle is $ = 0. The response has “zero phase”. Similarly, coefficients that are antisymmetric around zero produce a pure imaginary H(@). The phase is ¥ or —3. If h(-n) = —h(n) then h(O) = Hw) = hil) (-e® +e!) +h) (—e% + e-%) +++. (antisymmetry) —2i [A(1) sin w + h(2) sin 2w + +++] (imaginary H (w)) Zero phase is ruled out for a causal filter. The coefficients can be symmetric or antisymmetric, but not around n = 0. Causal filters have linear phase when their coefficients are symmetric or antisymmetric around the central coefficient: hk = WN-k (symmetry) Linear phase h(k) = —h(N—k) (antisymmetry) Moving the center of the symmetry from 0 to N/2 produces a factor e~'"/? in H (w). This means a linear term —N«/2 in the phase. The magnitude | H (w)| is an even (symmetric) func- tion because H (—w) is the complex conjugate of H(w). The graph of |H(w)| for0 < @ < 7 displays complete information. The exercises ask you to compute |H(w)|? = $(1 + cos «). Then a trigonometric identity produces our special form co: |H(w)P = 4¢ 1 + 008 «) = eos? S so that |H(@)| = |eos 2. For other filters, | H ()|? is a cosine series but its square root |H (@)| has no simple formula.1.2 Lowpass Filter = Moving Average L 3. 6. 10. i. 12. . Obtain the same result from |H(o) .. What constant-diagonal matrix represents the anticausal filter H’ with coefficients h'(0) = 4 Problem Set 1.2 The magnitude squared is H(«) times its complex conjugate H(@). Show that |H(w)?> = (1 +cos @) for the moving average filter. = (real part)?+ (imaginary part)’, with H(@) = $ +} (cos @ — isin w) Why is the formula | (w)| = cos ¥ wrong beyond the frequency w = 7? Draw the graph of |H(q)| from —27 to 2m. . In the complex plane, draw tan ¢ (c) as the ratio of the imaginary part Im H () to the real part Re H(«). Simplify this ratio to — tan 8. Solution. The phase $(o) is the angle from the horizontal, so the tangent of $ («) is the ratio of imaginary part to real part: Im H(w) int) ReH@) Again trigonometric identities make this unusually simple: in £ cos tang(w) = — o o ae =o tans and o@)=-5- ne .. Find the frequency response H () and its magnitude and phase, for the 3-point moving aver~ age filter with h(0) = h(1) = h() phase? . Does it have zero phase, constant phase, or linear What infinite matrix H represents the filter with h(0) = A(1) = A(2) vectors x(n) for which the output is Hx = 0. °) Find two input Solution. x = -2,—1,-1, 2, -1,-1,...) and x’ = delay of x. . What is the phase (c) of a symmetric filter with h(k) = h(8 — k)? 1 and h'(—1) = 42 What matrix represents the symmetric filter F = H’H and what is the frequency response of F? . Find causal filters whose magnitude responses are |H (c)| = | cosco| and |H («)| = (cos $)* Are they unique? Iterate the averaging filter four times to get K = H*, What is K (w) and what is the impulse response (1)? Why is an antisymmetric filter, with &(k) = —h(N — k), never lowpass? Consider an averaging filter with four coefficients 4(0) = h(1) = A(2) = h(3) = § and the input x(n) = (—1)". What is the output y(n) = x(n) * h(n)? Without computing H(w), explain why h(n) is lowpass. . Let H(z) be a lowpass filter with N + | coefficients and let G(z) = z-" H(z"). Find g() in terms of (72). What is the phase of G(w) if H is symmetric? Is G(z) lowpass or highpass?ae Chapter 1 Introduction 1.3 Highpass Filter = Moving Difference A lowpass filter takes “averages”. It smoothes out the bumps in the signal. A bump is a high- frequency component, which the lowpass filter reduces or removes. The response is small or zero near the highest discrete-time frequency @ = 7. A highpass filter takes “differences”. It picks out the bumps in the signal. The smooth parts are low-frequency components, which the highpass filter reduces or removes. Now the frequency response is small or zero for frequencies near @ = 0 (which is direct current and has no bumps). The lowpass filter that outputs the moving average $ (x(n) +.x(n — 1)) has a twin (or mirror) filter. This is the highpass filter that computes moving differences: Highpass: y(n) = }x(n) — $x(— 1). (1.10) The new filter coefficients are 4(0) = 4 and h(1) = —}. Equation (1.10) is a convolution y = hx, and the vector h for this highpass filter is, (a) This h is exactly the response to the impulse x = (..., 0,0, 1,0, 0,...). At time zero, the dif- ference 4 (x(0) — x(—1)) is $(1 — 0). The next difference 4 (x(1) — x(0)) is $(0 — 1). Those numbers 4 and —4 are the coefficients in h. This unit impulse at time zero we will denote by 6. In the language of convolution, we are saying that 6 = (...,0,0, 1,0, 0,...) always yields hx 6=h. (1.12) In the language of matrices, multiplying any matrix times the special column vector 6 always picks out the zeroth column of the matrix. The matrix for our highpass filter does have } and —5 in its zeroth column: yD -4 x(-1) yO) j= 3 LOVE Ie (1.13) Baas yl) x(1) This is again a constant-diagonal matrix because the filter is again time-invariant. (We define and discuss time-invariance in the next chapter.) The main diagonal entries produce 4 (identity). The subdiagonal entries give —4 (delay). The filter as a whole is highpass filter = 4 (identity) — 5 (delay) = $1— 38. (1.14) This is another causal FIR filter with two taps, but its frequency response is completely different from the lowpass function Ho(@) = § (1 +e).1.3 Highpass Filter = Moving Difference 13 Frequency Response ino At frequency «, the input vector is x(n) =e". The highpass output is eit? — 1 ginDo i) gine = Hoe”. (1.15) y(n) This quantity Hy(@) = 4 — 4 e~ is the highpass response. We want to graph it and compare it with Ho(c) — the lowpass response. We are introducing the subscripts 0 and | for lowpass and highpass. As before, we take out a factor e Hy(w) = 4 (iw? fe e71#/2) eto in (§) je? (1.16) i#/2, This leaves a sine not a cosine: The magnitude is | H;(~)| = |sin $|. Since the sine function can be negative, we must take its absolute value. Figure 1.4: The magnitude and phase of the highpass filter | — } e~". The graph of |H;(«)| is in Figure 1.4a. It shows zero response to direct current, because sin = 0. It shows unit response at the highest frequency @ = sz, because sin} = 1. In the time domain, these numbers come from taking differences of the lowest and highest frequency inputs: -1,1,1,1,1,...) and xin = (-.,1,-1,1,-1,1,...). Xow = The response to Xjoy is Oxjoy. The response to xpigh iS 1 Xhigh- The first vector has no bumps and the second vector is all bumps. The phase factor in H,() isa little tricky. When the magnitude |sin $| is removed, we are left with a sign change at @ = 0: ie? for -1<0<0 eit) — +iel#? for O0 |-[5 |= (2D > metal This matrix represents the whole analysis bank. Itexecutes the lowpass channel and the highpass channel (both decimated). All rows are unit vectors (because of the division by V2). Those row vectors are mutually orthogonal. At the same time, the columns are also orthogonal unit vectors. The combined square matrix is invertible. The inverse is the transpose: (s]-bleal to 4 Multiplying in either order yields the identity matrix. The second matrix [L’ B"] represents the synthesis bank. This is an orthogonal filter bank, because inverse = transpose. We pause to summarize what you have seen in this example. Pelee The channels L = ({,2) C and B = (| 2) D of an orthogonal filter bank are represented in the time domain by a combined orthogonal matrix: [2 a|[ ; |-eeeete=r. The synthesis bank is the transpose of the analysis bank. When one follows the other we have perfect reconstruction. For causality we add a delay. That summary is a foretaste of later chapters. We will construct longer and better filters, but the underlying problem will be close to this one. There is an analysis bank and a synthesis bank. ‘When they are transposes as well as inverses, the whole filter bank is called orthogonal. When they are inverses but not necessarily transposes, the filter bank is biorthogonal. In the biorthogonal case, the coefficients in the synthesis bank are different from e(n) and d(n) in the analysis bank. Our Haar filter bank has orthogonal filters. This chapter pursues it further, all the way to Haar wavelets. Block Form of a Filter Bank The best way to represent a two-channel filter bank is by a block diagram (Figure 1.5). The input is a vector x. The blocks are linear operators and the output is two half-length vectors: Lx = (| 2) Cx and Bx = (| 2) Dx. For this special filter bank, we want to display all vectors in detail. Later we could supply only the essential information: the coefficients e(m) and d(n). There will be several ways to dis- play those coefficients — the filter bank form, the modulation form, and the polyphase form. We18 Chapter 1 Introduction Fe 2 vy =G 2Cx=Lx pL \2 v,=4 2)Dx=Bx Figure 1.5: Schematic of the analysis half of a two-channel filter bank. always need the coefficients! Here we use the direct filter bank form, and write the components of each output vector: 1} x0) + x(-bD 1} x0) - x(-b =B| x) + x) Br="Bl x2) — x(1) etree eee x(3) x4) -— x(3) The odd-numbered components of Cx involved x(1) + x(0) and x(3) + x(2). Those are gone in Lx. Similarly the odd-numbered components of Dx are gone in Bx. We are left with two half-length signals. Nevertheless we have enough information to recover the full-length input x—and you see how. To recover the zeroth component, add x(0) + x(—1) to the difference x(0) — x(—1). That gives 2x(0). Since our sums and differences are already divided by V2 in the analysis bank, we need another V2 in synthesis: 1 (x) +x(-1 x(0) —x(-1 For x(—1), use the same components of Lx and Bx, and subtract: Cie ge eh. v2 v2 Addition and subtraction are simple for this example, but the synthesis steps must be organized in a way that extends to other examples. At the end of the reconstruction, we want to reach these two vectors wo and wi: x(0) + x(-1) —x(0) + x(-l _ 1] x@ + x(-1) _1] x@ = xp *o=5] xa) + xa) | MO" =F] 2a) + xa oY — x) x(2) + x(1) x(2) Notice how the signs in w; are adjusted so that wy +, recovers the input vector. Actually we are getting x(n — 1) instead of x(n). The total effect of the whole filter bank is a delay. The input is x(n) and the output is the delayed x(n — 1). The sum wo + w is almost, but not quite, the original x. A delay is built in because all filters are causal. We analyzed x into low and high frequencies. Now we synthesize to reach wo and w; and recover x.1.4 Filter Bank = Lowpass and Highpass 19 The Synthesis Bank A well-organized synthesis bank is the inverse of the analysis bank. The analysis bank had two steps, filtering and downsampling. The synthesis bank also has two steps, upsampling and fil- tering. Notice how the order is reversed —as it always is for inverses. The first step is to bring back full-length vectors. The downsampling operation (J 2) is not invertible, but upsampling is as close as we can come. The odd-numbered components are re- turned as zeros by upsampling. Applied to a half-length vector v, upsampling inserts zeros: v(0) 0) 0 Upsampling —(t2)| va) |=] vay |. (1.22) v(2) 0 ¥(2) Upsampling is denoted by (f 2). To understand it, look at the result of downsampling to get v = (}2)y and upsampling to get w = (t2)(1 2)y: yO) yO) yd) y(0) 0 y=] y2) Q2y=} yQ) ADLDy=] y2) |. (1.23) y3) yA) 0 yA) v4) The odd-numbered components of y are replaced by zeros. We will see that (2) is the transpose of ({ 2). Fortunately, transposes come in reverse order exactly as inverses do. So synthesis can be the transpose of analysis — apart from our ever-present delay. Small note: Also (+ 2) is a right-inverse of (J 2). If we put in zeros and remove them, we recover y. Thus (J 2)(t 2) = 1. The order ( 2)({ 2) that we actually use is displayed above, and it inserts zeros. Properly speaking, (+ 2) is the “pseudoinverse” of (| 2)— which has no inverse. The vectors reached by upsampling have zeros in their odd components: x(0) + x(-1) x(0) — x(-1) 0 0 *Q) ee ead gy Ae (1.24) x(4) + x(3) x(4) — x(3) The second step in the synthesis bank, after upsampling, is filtering. The two vectors up and u, are the inputs to the two filters. The vectors wo and w, are the desired outputs. Schematically, the structure of the synthesis bank is in Figure 1.6. Normally we would construct the synthesis filters F and G based on the analysis filters C and D. That will be our procedure in the rest of the book. For this example, when we know the20 Chapter 1 Introduction dacx 4t2 RFE oN > en dapx —f2 Figure 1.6: The synthesis half of a filter bank: upsample, filter, and add. desired outputs, we proceed more directly. The filter F that produces wo from uo is an addition filter: x(0) + x(-1) x(0) + x(-1) 1 0 1 | 2 + 2-1 F filters =] (2) ie togive 5) x2) +x(1) | = wo. x(2) + x(1) This is the output we want. It comes from a time-invariant causal filter F. There is no separate treatment of even and odd components! When the input to F has components u(n), the output has components 1 Fu(n) = (a(n) + uta — DV). (1.25) The filter coefficients are G(0) = f(1) = J5. The second synthesis filter G is a subtraction filter. Its coefficients are = arbitrary input vector w, the output has components a (-u(n) + u(n — 1)). Notice especially how G acts on wu; = (¢2)( 2) Dx: and J5. For an x(0) — x(-1) —x0) + x(-l) 1 0 TOs eG) G filters x(2) — x(1) togive | -x2) + xU) | =m. v2 — x(1) 0 x(2) The filter gives the right result, again without treating even and odd components differently. We caution that the highpass coefficients are in the order ~ J, ~. When you transpose D, the order of coefficients is reversed. Then a delay makes G a wusal filter (and the same for F). The only other caution concerns the factors They are present in all four filters. A less perfect symmetry would have 4 in one bank and 1 in the other bank. This is exactly like the two- point discrete Fourier transform, where we often allow 3 and | in the matrix and its inverse. The 1 . “ ‘ 1 inverse matrices with 1 and 4 11y' it sj -1 Te | = [ a | orthogonal matrix with 1/2. An orthogonal (or unitary) matrix has orthogonal rows and orthogonal columns normalized to be unit vectors. The inverse is the transpose (or conjugate transponse). It is an accident for this il] aie —1.4 Filter Bank = Lowpass and Highpass 21 example that the matrix is real and symmetric. You can’t see that it was transposed and conju- gated. Important. The connection between this Haar filter bank and the 2-point DFT is no accident. The simplest M-band filter bank comes from an M-point DFT. It is called a uniform DFT filter bank. We are seeing the case M = 2, written as an ordinary filter bank (and made causal by a delay). We summarize this section with a schematic of the whole filter bank (Figure 1.7). w a x(n — 1) {2} 42} 5}—{] Figure 1.7: The analysis bank followed by the synthesis bank. x(n CR} fr} TF This allows us to indicate the symbols y, u, v, w for the outputs at the four stages. For a two-channel bank, we mark those vectors in the lowpass and highpass channels by subscripts 0 and 1, When there are M channels, we need subscripts 0, 1,..., M— 1. From the filter bank, the next step is to wavelets. Problem Set 1.4 1. Write down the matrix (|, 2) that executes downsampling: (J 2) x(n) = x(2n). 2. Write down the transpose matrix (t 2) = (| 2)". Multiply the matrices (f 2) (| 2) and (12) (42). Describe the output from (2) y(n). 3. Describe the output from (|, 2)? x(n) and (+2)? y(n). 4. Send the signal with x(0) = x(1) = x(2) = | through the whole filter bank, and give the output at every step. 5. Puteachrow[ -1 1 J/V2o0fB=({2)Daftertherow[ 1 1 J/V20fL = (12)€. In this order we see a block transform. What is the inverse transform? 6. Show how to delay an anticausal (= upper triangular) filter matrix so it becomes causal. If the coefficients (0), ...,4(N) are on the diagonals of the anticausal matrix, what are the diago- nals after the delay? Give the two frequency responses, anticausal and causal. 7. The 4-channel bank analogous to Haar is based on the 4-point DFT matrix F. Find Fy and the four analysis filters and the outputs after downsampling by ({ 4). 8. Suppose a matrix has the property that @"@ = I. Show that the columns of Q are mutually orthogonal unit vectors. Does it follow that QQ" = 1? 9. Ina transmultiplexer, the synthesis bank comes before the analysis bank. Compute LL" and LB" and BB" to verify that the Haar transmultiplexer still gives perfect reconstruction: i + pr yf EE’ 1B" ]}_[ 1 0 [ B It ue |=! ger pat |= 0 7 |: 10. Compute the subband outputs yo(n) and v, (2) for the average-difference filter bank with input x(n) =0, 1, -1, 2, 5, 1, 7, 0. Reconstruct the signal by feeding y, (1) into the synthe- sis bank in Figure 1.6. Verify that the output is x(n — 1). Hl. If Ho(z) = Land Hy (z) ! (no filtering) write the entries of [ : lt Be22 Chapter 1 Introduction 1.5 Scaling Function and Wavelets Corresponding to the lowpass filter, with 4(0) = } and h(1) = 5, there is a continuous-time scaling function #(t). Corresponding to the highpass filter, with coefficients } and —4, there is a wavelet w(t). We now describe the dilation equation that produces @(t) and the wavelet equation for w(t). You will see how the filter coefficients (the ¢’s and d’s) enter these equations. Two time scales also appear. This joint appearance of ¢ and 2r is the novelty of the dilation equation. It is also the source of difficulty! We have a “two-scale difference equation”. Later we develop the back- ground of these equations and a general method for solving them. Here we quickly recognize the box function as $(t). Then we construct the wavelet w(t) and use it. The dilation equation for the scaling function $(t) is Ny $(t) = V2 ek) ot — bh). (1.26) =o In terms of the original lowpass coefficients /(k), the extra factor is 2: a b(t) = 27h) o2t—b). (1.27) t= This involves a function @(s) in continuous time, and a set of coefficients e(k) or h(k) from discrete time. The presence of ¢ and 2r is the key. Without the 2 we would have an ordinary constant-coefficient equation (look for exponential solutions e*’). With two time scales, there are major changes: 1. There may or may not be a solution (1). 2. The solution is zero outside the interval 0 < t < N. 3. The solution seldom has an elementary formula. 4. The solution is not likely to be a smooth function. Formally, we can find an expression for the Fourier transform of (r). Itis an infinite product (Section 6.4). The inverse Fourier transform yields $(r) as a “distribution” — not necessarily continuous, possibly involving delta functions. (Those are impulses. Their integrals are jumps. More cautious people call them steps.) In our Haar example, the solution (r) lies just outside the class of continuous functions — it has a jump. For the coefficients 24(0) = 1 and 2h(1) = 1, the dilation equation is b(t) = (2t) + b(2t — 1). (1.28) The graph of (1) is compressed by 2, to give the graph of @(21). When that is shifted to the right by 4, it becomes the graph of o(2t — 1). We ask the two compressed graphs to combine into the original graph. Figure 1.8 shows that this occurs when (t) is the box function: Pio eta? 0 otherwise. vo-|1.5 Scaling Function and Wavelets 23 ot) Figure 1.8: The box function with dilation and translation. The graphs of $(2r) and @(2t — 1) are half-size boxes. Their sum is the full-size box #(t). As planned, we wrote down the solution rather than deriving it. The dilation equation is linear, so any multiple of the box function is also a solution. It is convenient to normalize so that the integral of (t) from —o0 to co equals one. Note that a solution #(t) covering unit area is only possible when the coefficients in the dilation equation add up to 2. Theorem 1.1 vf (t) dt = | then 2h(0) + 2h) +--+ 2h(N) = 2. 0 Proof: The graph of ¢(2r) and every (2r — k) is compressed to area 3: 0 0 2f g(r —k)dt =f ou)du=1 — (setu = 24-8). (1.29) So integrating both sides of the dilation equation $(t) = 2 1 h(k)p(2t — k) gives 1 = hO) + (1) +--+ h(N). This is our lowpass filter convention. Important. For the filter, then the filter bank, and finally the dilation equation, the normaliza- tion is different. This is clear from the actual numbers in our lowpass filter. The sum of coeffi- cients is 1 or V2 or 2: h: § and } in the single filter, adding to 1 ¢: Jz and J; in the filter bank, adding to v2 2h: 1 and 1 in the dilation equation, adding to 2. The single lowpass filter has sum H (0) = 1. That preserves the zero frequency DC term: out- put Y(0) = input X (0). The filter bank has a factor V2 to account for the downsampling step. There are only half as many components and half as many terms in the energy, when > (y(1))? is replaced by > (y(2n))?. To compensate we must multiply y(2n) by V2. That gives the nor- malization C(0) = v2 in place of H(0) = 1. For the dilation equation, we have just seen why rescaling the time requires renormalizing the coefficients by 2 —to preserve area. There are certainly filters in which (0) is not exactly 1. If H(@) stays very near | over an interval around w = 0, this still deserves the name “lowpass filter”. Such filters do not lead to24 Chapter 1 Introduction wavelets! The requirements for wavelets are very strict, at @ = 0 and w = x. Only a subset of special filters can pass the test, which starts with H(0) = 1 and H(z) = 0. The box function is like the averaging filter — it smoothes the input. Convolution with the box function gives a moving average in continuous time, just as the filter coefficients h = did in discrete time: 1 a x(0) +x(-1) xC) +x) h CO 2) a : a ‘ Olt) #x(t) = [ x(s) ds = average over moving interval. (1.30) fa There is a similar convolution with w(r). Instead of picking up the smooth low frequency part of the function, the wavelet will lead to the high-frequency details. The coefficients for the Haar wavelet are 1 and —1. The Wavelet Equation The equation for the wavelet involves the highpass coefficients d(k). It is a direct equation that gives w(r) immediately and explicitly from $(¢): w(t) = V2 d(k) or —&). a3) In terms of the original coefficients (k), the factor V2 becomes 2: Wavelet equation — w(t) = 2 Y*hy(k) dt — k). (1.32) Inour example, #(¢) is a box function and its dilations (2 —k) are half-boxes. Then the wavelet is a difference of half-boxes: w(t) = @(2t) — @(2t — 1). (1.33) Explicitly, w(t) = 1 forO oo. This still qualifies as a wavelet. It is a localized pulse that decreases to zero and has integral zero. Ingrid Daubechies showed in 1988 that compact support was possible for other wavelets than Haar’s. We will tell that story more completely in Chapter 6. From Haar onwards, one property that most designers hoped for was orthogonality. This means: w(r) is orthogonal to all its dilations and translations. The wavelet basis, containing all these functions w(2/t — k), is an orthogonal basis. For the first few Haar wavelets that is easy to verify: inner product = JS, wOw20) dt 0 inner product = f° w(t)wQr-Idr = 0 inner product = f°, w(2r)wQr-Idt = 0. In the first integral, w(t) = 1 in Figure 1.9 where w(2r) is positive and then negative. The integral is zero. Similarly w(t) = —1 on the second half-interval where w(2r — 1) is plus and minus. The second integral is therefore zero. The third integral vanishes for a different reason — the functions w(2r) and w(2t — 1) do not overlap. One is zero where the other is nonzero. So the product w(2r) w(2r — 1) is zero everywhere. The pattern continues for all translations by & and dilations by 2/. Haar wavelets at the same scaling level (same j) do not overlap. They are orthogonal in the strictest way. When Haar wavelets at different levels j and J do overlap, the coarse one is constant where the fine one goes up and down, All integrals are zero, giving an orthogonal basis w jx (1) = w (2/t — k): 20 inner product = | w (2/t—k) w(2/t— K)dt 0. (1.34) A perfect basis is not only orthogonal but orthonormal. The functions have length I. Like a unit vector, the inner product < w(t), w(t) > is normalized to 1: ce length squared = [ (w(t)? dt =1. This is true for the Haar wavelet. To make it true for the dilations of that wavelet, we multiply by 2//? otherwise the compressed graphs cover less area. The same factor will apply to other wavelets, and we record it now:26 Chapter 1 Introduction Theorem 1.2 The rescaled Haar wavelets w(t) = 2//w (2/t — k) form an orthonormal basis: J wya(t) wyx(t)dt = 6(j — J) 6k — K). (1.35) Ee This Kronecker delta symbol equals zero except when j = J and k = K. In that case the integral of (wy « (t))” equals one. We need two indices j and k because there are two operations (dilation and translation) in constructing the basis. Important note. For these Haar wavelets, orthogonality was verified by direct integration. For future wavelets, this integration is not desirable and not possible. We will not have elementary formulas for #(t) and w(t). Instead, we will know the coefficients (Kk) in the dilation equation and d(k) in the wavelet equation. All information about @(t) and w(t)— their support inter- val, their orthogonality, their smoothness, and their vanishing moments — will be determined by and from the c’s and d’s. Second note. We have not said which space of functions has the wavelets w x(t) as a basis. Actually there are many choices. The space starts with finite combinations g(t) = )> bj, w x(t). Those functions are piecewise constant on binary intervals — length 1/2/ and endpoints m/2/. Other functions f (f) are in the space if they are limits of these piecewise constant g’s: IFO —gn()|| > 0 for some sequence g,(t). The choice of function space is decided by the choice of the norm || f — g,||. A function f(t) might be a limit of piecewise constants in the maximum norm but not in an integral norm, or vice versa. The most frequent choice is the L? norm, where the superscript 2 signals that we integrate the square: 20 1/2 IFO-sOl= (f IFO —gOP a) : 00 This choice of the L? norm is popular for four major reasons: 1. The norm is directly connected to the inner product: By definition IFIP By the Schwarz inequality [fMIIgOll = |< Ff, g@ >|. 2. Minimizing the L? norm leads to linear equations. This is familiar from ordinary least squares problems. The word “squares” introduces the L? norm. When the functions g(r) are restricted to a subspace, the closest one to f(t) is g(t) = projection of f(t) onto the subspace. Projection leads to right angles and linear equations. 3. The Fourier transform preserves the L? norm and the inner product < f, g >. This is the Parseval identity: we = f" sora = i [Flo)? do = | Al" (1.36) . (1.37)1.5 Scaling Function and Wavelets 27 The reader will know that irrelevant but necessary factors of 27 should enter these equations. They depend on how we define the Fourier transforms. Since f(«) and 2(«) can be complex, we introduced absolute values in (1.36) and complex conjugates in (1.37). Of course (1.37) becomes (1.36) if g(t) = f(r). 4. For an orthonormal basis (like the Haar wavelet basis), the L? norm ||7bjxwjx(t) |” equals the sum of squares of the coefficients bj: [oon wx) dr doin bse fou wyx (dt Yn)’: For all these reasons, and more, the L? norm is our choice. But we must mention that wave- lets provide an unusually convenient basis for other norms and other function spaces. This makes them popular in functional analysis and harmonic analysis, which deal with the properties of functions. If we change the exponent from 2 to p, the space L? contains all functions for which (FFI)! is finite. This norm ||f||,, is not quite equal to C (3° |djx|”)'””, as it was for p = 2. Butif the bj, are wavelet coefficients, rather than Fourier coefficients, this sum lies be- tween fixed bounds A, ||f|, and B, | fl. Thus the absolute values |bj.| still indicate which functions have finite norm and belong to L?. In shorthand: The wavelets are an unconditional basis for L” (no condition on the signs of the bj,). The complex exponentials are conditional; phase information is needed on the b j.. We recognize that these last comments are “pure mathematics”. The reader is invited to start learning that language too — if desired and not already achieved. It is rewarding and not difficult. The wavelet transform that connects f(t) to its wavelet coefficients b jx is absolutely central — to theory and also to applications. We turn to the practical problem: How to compute the coefficients bj, quickly? This has a very good answer for wavelet transforms. There is a recursive Fast Wavelet Transform compa- rable in speed and stability to the FFT for Fourier transforms. Problem Set 1.5 «If w(¢) has unit norm, so that f (w(1))°dt = 1, show that the funetion wj4(0) = 2/?w(2it—k) also has unit norm. 2. What combination of the half-boxes (21) and (21 — 1) is closest in L? (least squares) to f(t) =? forO -/ f(t) 2!? w (tk) dt. (1.45) For Haar, the wavelets are piecewise constant. The original wavelet w(t) = woo(t) is +1 on the interval [0, }) and —1 on the interval [5, 1). The basis function wx is 2//? on a subinterval of length $2-4 and —2// on the next subinterval. There are four wavelets at level j = 2, when we start on the unit interval [0, 1). There are 2/ wavelets at level j. To compute all the inner product integrals at levels j = 0, j = 1, and j = 2, we can integrate F(t) over all eight subintervals of length +. This gives eight numbers. How do those numbers produce bo and bx and boy and ayy? The answer is beautiful. Those eight numbers are exactly the level 3 averages a3, a31, 432, ++, @37. We act on them exactly as in discrete time: filter and downsample! The numbers at level j — 1 come directly from the numbers at level j. This is because the functions at level j — 1 come from the functions at level j. The box function is a sum of half-boxes and the wavelet is a difference of half-boxes: 1 #0) = 62 + O21) gives 4)-14(0 = [ja + j240100] (1.46) 1 WO) = 62 — G2 =D gives wj-14) = [Gal Pia]. 47) Please look at these equations, They are the dilation equation and wavelet equation. The scaling function @;(t) = 2//? @(2/t — k) is constant on the interval of length 2~/ starting at ¢ = k2-/, When we take its inner product with f(r), we are integrating f(t) over this subinterval. Multiply the two equations by f(r) and integrate:1.6 Wavelet Transforms by Multiresolution 33 Scaling coefficient: aj. = Jy(aj.rx + 4j.2041) ; (1.48) Wavelet coefficient: by1e = A5(aj2e ~ aj2441)- The coefficients follow the pyramid algorithm. Notice again the time-reversal in which 2k + 1 appears. Our coefficients aj, =< f,@jx(t) > and bj, =< f, wje(t) > are inner products, and a filter (a convolution > h(k) x(n — k)) has this reversal. The main point is the pyramid. This beautiful connection between wavelets and filter banks was discovered by Stéphane Mallat. The pyramid algorithm is also called the Mallat algorithm (don’t pronounce the ‘t’). It is a tree of butterflies in the Preface, it is a tree of filters in Figure 1.11, and a third form of the tree shows the trunk of averages a, and the branches of differences Dyk: i NS ak NS aK NS 0K bik be bo Wavelet coefficients at level j + 1 are differences of scaling coefficients at level j. This pyramid is also an equality of functions. A function at fine resolution j is equal to a combination of “average plus detail” at coarse resolution j — 1: Sain byl) = aye b) AKO + Dobe WHO. (1.49) 7 T Multiresolution in Continuous Time The equality of functions in (1.49) is also an equality of function spaces. On the left side is a combination of ¢’s at level j. Let V; denote the space of all such combinations. On the right side is a combination of ¢’s at level j — 1. This is a function in the scaling space V;_1. Also on the right is a combination of w’s at level j — 1. This is a function in the wavelet space W)—1. The key statement of multiresolution is Vj =Vj-1 ® Wa (1.50) The symbol + for vector spaces means that every function in V; is a sum of functions in V;—1 and W;_1, as in equation (1.49). The symbol ® for “direct sum” means that those smaller spaces meet only in the zero function. This is guaranteed when the two subspaces are orthogonal. In that case the direct sum ® becomes an “orthogonal sum”. For emphasis we restate the definition of the subspaces: Vj = all combinations 7 ajx@ x(t) of scaling functions at level j k W, = all combinations ‘7 b,wj«(t) of wavelets at level j. k V; is spanned by translates of @ (2/r) and W; is spanned by translates of w (2/r). The time scale is 2~/. At each level, all inner products are zero. We have two orthogonal bases for V;, either the @’s at level j or the @’s and w’s at level j — 1.34 Chapter 1 Introduction Notice how this multiresolution grows to three levels or more: V3 = V2 © Wz = Vi © Wi @ Wz = Vo @ Wo @ Wi © Wo. (51) On the left side are all piecewise constant functions on intervals of length e On the right side is the same space of functions, differently expressed. The functions in Vo are constant on [0, 1). The functions in Wo, Wi, W are combinations of wavelets. The function f(r) in V3 has a piece F;(t) in each wavelet subspace W; (plus Vo): LO = Yo aoedoe() + D7 bor woe() + D7 dw) + 2 brawl. (1.52) i i i 7 Note to the reader. The parallels between a filter tree in discrete time and multiresolution in continuous time are almost perfect. The filter bank separates lower and lower frequencies, as we iterate. Multiresolution uses longer and longer wavelets, as we climb the pyramid. We are speaking of the analysis half, where inputs are separated by scale. Discrete time Continuous time filter bank tree multiresolution downsampling @ > & rescaling t > 2r lowpass filter averaging with $(r) highpass filter detailing with w(t) orthogonal matrices orthogonal bases analysis bank output wavelet coefficients synthesis bank output sum of wavelet series product of filter matrices fast wavelet transform The reader will understand that in writing about Haar wavelets, we are writing about all wave- lets. The pattern is fundamental, the pieces in the pattern can change. The actual e(k) and d(k) and (1) and w(t) are at our disposal. The next chapters move to filter design, where we make choices. Those choices determine the wavelet design. Some filters and wavelets are better than others. We will not allow ourselves to forget the pattern that makes all of them succeed. Caution. The Haar wavelets are orthogonal. Thus they are biorthogonal fo themselves. We are not seeing a clear difference between analysis and synthesis filters, C and D versus F and G. For the same reason we are not seeing the dual functions dO and @(r). A clue to this shadow world (or tilde world) is in the time-reversals, which involve the next sample 2k + 1 instead of the previous sample 2k — 1. This suggests filters C” and D” rather than C and D. The biorthogonal case comes in Section 6.5. It has two multiresolutions, Vist = vj ® Ww, in parallel with V;;, = V; © W;. The pyramid and the fast wavelet transform go one way in analysis (with tilde). The inverse transform goes the other way in synthesis (without tilde). Problem Set 1.6 1. (a) Show that the exact sum in equation (1.43) is 27(N — 1). (b) How many matches are needed to decide the winner in a knockout tournament with N players? The average is the winner that goes to the next round (next filter). The differ- ence is the loser that stops.1.6 Wavelet Transforms by Multiresolution 35 a x - . Write out the factorization of Haar’s A for N = 8 (following 1.42). }. For N = 8, write out the factorization of $ corresponding to (1.44). |. Draw the synthesis tree, the reverse of Mallat’s analysis tree, A similar pyramid algorithm was proposed by Burt and Adelson before wavelets were named. . Split the function f (0) into a scaling function plus a wavelet. 5 fo 5 gf) .. Split the function g(r) into its pieces in Vo, Wo, and W; (box plus up-down coarse wavelet plus two fine wavelets). . The function g(r) is in V2 (its scale is 1). The pyramid splits it first into a function in V; (two half-size boxes) plus a function in W; (two half-size wavelets). Find those pieces. Then split the piece in V; into a full-size box plus a full-size wavelet. These three pieces are in Vo and Wo and W;. Synthesize f(t) in Vi from the first two pieces. Then add the details in the third piece to synthesize f(r) in V2. 4 4 2-——_ 2 2 1 ee ee 1 ra 0 1 0 1 0 1 . Suppose H(t — }) is the unit step function with jump at ¢ = 4. Its inner products aj, with the boxes $j4(1) will be nonzero for about two-thirds of the 2/ boxes on [0, 1]. How many inner products bj, with the wavelets w;() will be nonzero? This example shows the compression of step functions by Haar wavelets.Chapter 2 Filters This chapter comes between the Haar example of Chapter 1 and the full development of filter banks (leading to wavelets). We decided to collect other definitions that belong to this circle of ideas. Here is an indication of our plan: Basic filters: Ideal filters and then FIR filter design Basic tools: Fourier methods and functional analysis Bases and frames: A matrix T has a two-sided or only a one-sided inverse Integral transforms: windows in time-frequency, wavelets in time-scale. Signal processing is an enormous subject. The input signal can arrive in many forms: continuous time, discrete time, finite time. It can be processed in many ways. Our greatest interest is a signal x(n) in discrete time that is processed by a linear time-invariant operator. /f the input is shifted in time then the output is equally shifted. These operators are filters —the fundamental actors in signal processing. Filters can be expressed in three domains: 7, w, z. In each domain the filter is a multiplica- tion: (n): Multiplication by a Toeplitz matrix with h(n) on the nth diagonal. (@): Multiplication by the frequency response H(e!”) = Sh(nje~/". (2): Multiplication by the transfer function H(z) = h(n)z~". We want to explain these three forms and the connections between them. /f we emphasize the matrix form more than usual, it is because that form is less well known. We believe that the matrix formulation must become familiar, as the teaching and practice of signal processing rely increasingly on computer systems like MATLAB. Our goal is to understand filters, through impulse responses and frequency responses and transfer functions. If you know signal processing as in the Oppenheim-Schafer text, go past the first sections. If you are learning the whole subject from scratch, these sections can help. Do not hesitate to use this chapter for reference, as you reach Chapter 4 and the heart of the book. We begin with signals.2.1 Signals, Samples, and Time-invariance 37 2.1 Signals, Samples, and Time-invariance A discrete-time signal is a sequence of numbers. The sequence could be finite or infinite. Most signals in this book are doubly infinite; the index n goes from —oo to +00. Time has no start and no finish. The signals look like x(-1) (0) x(1) x(2) eeey Eats XO) Xs F29--) OF Those components are real or complex numbers (usually real). One particular signal is of tre- mendous value. It is the unit impulse x = 6: 0, #0 1, n=0. en 6=(...,0,0,1,0,0,...) hascomponents 6(n) = | The continuous-time analogue is the “delta function” 6(t) —also called a Dirac impulse. In one case n is an integer, in the other ¢ is a real number. The standard notations aren € Zandt € R. Together with the special vector 6 goes the delayed impulse $6, where the unit component appears one sample later at 1 = 1. The whole vector is shifted by one time step. The symbol S stands for the shift or delay that has this effect on the vector 6: 0, n¥l S6=(...,0,0,0,1,0,...) has components an ={ l ea Itis worth emphasizing that a shift to the right (a delay) produces the minus sign in the expression n — 1. Itis the same in continuous time. The graph of f(t), when it is shifted one unit to the right, is the graph of f(t — 1). The delayed function at t = 1 equals the original function at r=0. When the components are shifted to the left, the impulse comes sooner (at n shift is an advance instead of a delay. The symbol is S~!. The operator “S inverse” h opposite to S: 0, n#-1 Hl 5 eae 1, n=-1. 0,1,0,0,0,...) has components 6(n + 1) = { Ofcourse S“'S6 = 6. The impulse is delayed by § and then advanced by S~!. Also SS“'5 = 6. The operator $~' is a “two-sided inverse” of S. The vector 6 could be defined, and Dirac’s delta function should be defined, by what hap- pens for inner products. The inner product (dot product) with any vector x(m) or any continuous function x(t) picks out x(0): x76 = x(n) dn) = x00) and (x(1), 8()) ai x(1)3(t) dt = x(0). The number x(0) is a sample of the function x(t). Many discrete signals come from sampling continuous signals. This analog to digital (A/D) conversion is a central part of communications38 Chapter 2. Filters technology. We sketch two continous-time signals x(t), a step and an exponential, and their discrete-time samples x(n). ell, ecb -3-2-1 01 -3-2-1 01 Unit Step One-sided exponential The samples of cost are of special importance. The continuous signal has frequency @ (often normalized as f = #). Everything depends on the sampling period T and the sampling rate f, = +. The sampling may be fast enough to catch the oscillations in cos wt, or it may be too slow. We may catch the oscillations in cos wt, or miss some. The borderline is the Nyquist rate. The sampling rate is exactly the Nyquist rate when f, = 2f and + = 2 and oT =z. This is the rate (two samples per oscillation) in the first figure. Those samples have the fastest oscillation that a discrete vector can achieve: x(n) = (—1)". To repeat: The Nyquist rate gives the highest possible frequency @T = x. alias . . ° . eietere ° ° ~2n \-2 afin t ~2n \-m a] 2n t ~2n/\-" mfQn t . . . . Sampled at Nyquist rate Twice the Nyquist rate Slower than Nyquist rate The second sampling rate is faster than Nyquist. So the digital frequency wT is less than x. In this figure, the sampling frequency f, is twice the Nyquist rate, four samples (four bullets) per oscillation. The sampled signal is x(n) = coswnT = cos 4. Therefore the samples are 1,0, -1, 0, 1,0,. The third sampling rate is slower than Nyquist. The sample after cos 0 is cos #. The sam- pling rate is 3 of the Nyquist rate (+ = 32). At this slow rate, the samples are the same 1, 0, —1, 0, 1,0, as in the second figure! We cannot tell whether the true frequency of the continuous signal is @ (as drawn with solid line) or a slower frequency $ (as drawn by dotted line). This is aliasing. The slow frequency $ is an alias for the true frequency w, because the discrete samples at this rate will be exactly the same. An extreme case of slow sampling is when @T = 2x. All the samples are cos nT = cos 2n = 1. Every sample is at the top of the wave. The digital frequency 27 looks identical to frequency zero (which is the alias of 277). When the continuous signal is a combination of many frequencies w (or f = #2), the largest ONE max Sets the Nyquist rate 2 fy;a.. The corresponding Nyquist period T has ®axT =. As2.1 Signals, Samples, and Time-invariance 39 long as the sampling period is smaller than this 7, the sampling rate is faster than the Nyquist rate. Then there is no aliasing (by a lower frequency than the true frequency). The continuous signal x(t) can be recovered from its samples x(n). The Shannon sampling formula, to achieve that recovery, is in Section 2.2. impulses and Delays in Three Domains Impulses are the building blocks for all signals. Delays are the building blocks for all filters. We will present signals and filters in three ways — with time variable n, and frequency variable w, and complex variable z. Signal in the time domain x(n) (rex (a) x(O)ex (see) Signal in the frequency domain. X(e4”) = — Yox(n)e~/°" (standard) X(@) = Yx(n)ei" (reduced) Signal in the z-domain X(z) sn x(t) zane The standard notation and reduced notation were compared in Section 1.1. We use both! Some- times the standard notation is clearer; it allows direct replacement of e/” by z. Sometimes the reduced notation is simpler, as in ¥(w) = H(w)X (w). The impulse 6 = (..., 0,0, 1,0, 0, ...) becomes the constant function “1” in the frequency domain and z-domain. The only nonzero component 6(0) = 1 is in the constant term. The delayed impulse y = (...,0,0,0, 1,0,...) looks more interesting. In the other domains this y(n) = 6(n — 1) is Y(ei”) = e-#® and ¥(z) =z). Note that y(1) = 1 multiplies the negative power of z, by the signal processing convention. Tf the impulse is advanced instead of delayed, the nonzero occurs at n = —1. The transform is z instead of z~!. This signal is no longer causal. The advance operator S~' is not a causal filter. Now we define filters in general and study delays in particular. A digital filter is a combination H = Yams" of delays § and advances S~'. The filter is completely determined by its coefficients (7). When this sequence is finite, we have an “FIR filter”. When h(n) = 0 for negative n, we have a “causal filter”. Our greatest interest is in causal FIR filters like Ne 3a Spe es 3h l/l I+ S+ S? +S? (aubechies Dy filter). nes 8 8 8 Suppose this filter acts on the impulse 6. The output is a combination of 6 and its delays $6 and S?6 and S°6: l+v3 3+v3 3-V3 1-3 5 H6=(..., 8 8 3 8 This is the “impulse response.” It is equal to the vector h of filter coefficients: The impulse response (causal and FIR) is h = (4(0), h(1), ...,4(N)). You see why H = h(n)" acting on 6 produces this output h. Each term h(n)S" produces one response h(n) at time n.40 Chapter 2. Filters As displayed, the filter is FIR and causal with N + 1 “taps”. The filter length is even when N is odd! Some authors end at h(N — 1), so the length is N — but then the power z~‘Y—" enters into a large number of formulas. We prefer to have sums from 0 to N, and scaling functions and — in all those formulas. The Daubechies filter has wavelets on the interval 0 < ¢ < N, and z length 4 because N = 3. The delay S takes x = (...,x(0),x(1), ...) intoy = (...,x(-1),x(0),...). Every linear operator like S is represented by a matrix. Since x and y = Sx have infinitely many components, the shift matrix S has infinitely many rows and columns. This causal operator becomes a lower triangular matrix: x(-1) x(-2) 0000 - 100 0 x0) |_| xD Sx = 0: 1 020 x) |> | x) 0010 x(2) x(1) The only nonzero coefficient for this filter is4(1) = 1. This coefficient goes along diagonal one. In general (in) goes on diagonal n. What does the delay do in the z-domain? The input X (z) and output Y (z) are X( +e) $a(Det tee and ¥(z) Se +O) +eeF +e The delay has multiplied X (z) by z~! to produce Y(z). This transfer function z~! is exactly the z-transform of the vector (....0, 1, 0, 0, .. .) of filter coefficients. The pattern is always Y (z) = H(z)X(z). Here is the special result for this particular filter, a delay H = S: X(e/) is multiplied by e~/” and X(z) is multiplied by z~!. Time-invariant Filters Our filters H are linear, This means in particular that “zero in produces zero out”. If x = 0 then necessarily y = 0. The output from 2x is 2Hx. The output from x + z is Hx + Hz. This has an important consequence: H is represented by a matrix. Our filters are also time-invariant (meaning shift-invariant). This leads to a special constant- diagonal property of the matrix: H(Sx) = S(Hx) : A shift of the input produces a shift of the output. Each column of H is a delay of the previous column. Each diagonal of H is constant and the nth diagonal contains h(n). Those are different statements of time-invariance. They imply that H is a combination of shift operators: Every filter has the form H = > h(n)S". The filter H = I + 48-+3S? has coefficients h = (1,4, 3). These are the entries down every column of the Toeplitz matrix.2.1 Signals, Samples, and Time-invariance 41 G0 ©: 0) 6 row 0 » 4 1 0 @ - row 1 H= 40a. OFS deal cs a diagonal number column 0 1 2 core lee O) Toeplitz matrix = constant-diagonal matrix with entries H,; = h(i — j). The numbers 1, 4, 3 also appear in every row —but the order is reversed. This is a causal FIR filter. It is time-invariant, because HS or SH (they are the same!) is § + 4S? + 38°. All columns and all diagonals shift down by one, from the delay. The difference between the row number i and the column number j is the diagonal num- ber k = i — j. The entries of H depend only on k. This is a constant-diagonal matrix and a convolution matrix and a Toeplitz matrix. Matrix Multiplication and Vector Convolution The jth column of H is S/h. We can compute Hx as a combination > x(j)(S/h) of those col- umns. We can also compute )> h(k)(S*x). Best to see the nth component Hx(n): YexGna - j) = hdoxan — i k le t nth component of delayed A nth component of delayed x The equality comes by changing j ton — k. The sums are over all integers, so the change is allowed. The finite sum to j = N would not equal the sum to k = N, unless A has period N. (Then the N by N matrix H would be a circulant matrix. This “wraparound” is the easiest way to deal with finite length signals, but not generally the best way.) Tn both formulas for Hx(n), the indices add to n. The zeroth component is row zero of H times x: ¥(0) = Hx(0) = h(N)x(—N) +++» + A()x(-1) + (Ox). (2.2) Each pair of indices on the right adds to zero— which is the index on the left. The numbers h(N), ...,h(O) are like a moving window that multiplies x. The nth component of the output has indices adding to n: Hx(n) = h(N)x(n — N) +++» +h()x(n — 1) +h(0)x(n). (2.3) This is the pattern that produces convolution: The output vector is Hx = hx = convolution of h with x.42 Chapter 2. Filters Convolution Rule Indices automatically add when they are the exponents in a polynomial. Multiply H(z)X (z) and their coefficients undergo a convolution: H(2)X(z) = (hO) FAA)! + FANE) x(—Dz + x0) +x(Dz7! +++). The coefficient of 2° is h(O)x(0) + A(1)x(—1) + ++» + A(N)x(—N). This is Hx(0). The coefficient of 2" in the product H(z)X (z) is the nth component of Hx. Multiplying polynomials means collecting terms with the same exponent. This is convolution. Example 2.1. The filter matrix has (0) = 1, A(1) = 4, and h(2) = 3. Suppose the input has x(0) = 1 and x(1) = 1. Then we multiply polynomials or we take convolution of vectors: 341 tA (+4e4432%)d+e') = L4+S2'4+72e7? 4325 341 341 (143) Cd) 5:73): ee aoe At z = | this is 8 times 2 equals 16. At z = —1 we check 0 times 0 equals 0. Example 2.2. Multiplying two filter matrices (Toeplitz matrices) is also a convolution. The product FH is another time-invariant filter, and its coefficients are inf * h. This is just like multiplying polynomials, with the shift § in place of the complex variable z~!: FH = (I + S)(I+4S + 38°) =1 +585 + 78° + 35°. Note again that the order is not important: FH = HF. This will change when there are sampling operators (J 2) and (+ 2) between the filters. Example 2.3. The convolution of (1, a, a”, ...) with (1, b, b’, ...) is (+ bet $e $e) tar! +a2 7? +++) = 14 (a tb)o!+@ +abt hz? + The power z~? in the product comes from z° times z~, and from z~! times z~!, and from z7* times z°. The sum of exponents is —2, to give z~? in the answer. This is the pattern for the indices k and n — k. Their sum is always n, to give y(n) in the convolution. The Inverse of a Time-invariant H The filter H is invertible if and only if H(w) £0 forall frequencies H(z) £0. forall |z| = |e”| = 1. 4) Then H™" is also a constant-diagonal matrix. Its frequency response is 1/H (). Invertibility is the first of many properties that become infinitely simpler by transforming convolution to H(@)X(w). The inverse of multiplication is division! We recover X(w) from Y¥(@)/H(@). The requirement is H(w) # 0.2.1 Signals, Samples, and Time-invariance 43 To emphasize: If we know a frequency wo for which H(wo) = 0, then we know an input x for which Hx = 0. That input has the pure frequency wp. It is the vector with components x(n) = e~/*", The pure frequency is selectively killed by H(w) = 0. Then H ()X (wo) = 0 and H™! fails. A moving average with equal weights h(0) = h(1) = + is not invertible. The frequency response H(w) = 5(1 + e~/”) is zero at @ = 1. The vector with components x(n) = e~/"" = (—1)" is exactly the vector that has Hx = 0. By changing to two unequal weights the system becomes invertible. Example 2.4. Suppose h(0) = 1 and h(1) = —£. The frequency response is H(w) = 1 — Be~'. If we select 8 smaller than one, then 1 4 Be~'’. Thus H(w) 4 0. The matrix H has 1 on the main diagonal and — on the diagonal below. To invert in the frequency domain, divide by H(w). To invert in the time domain, practice with a 4 by 4 matrix: 1 1 a Bol | =| p41 : (2.5) “Bb 1 BoP pt This suggests the correct diagonals 1, 8, 6, ... for the infinite matrix H~'. If H is I — BS, its inverse I + BS + B?S° + --» has the frequency response 1/H(w): 1 1— pe = 14 pei? + (Bel)? 4 (Pe 4 The most important of all infinite series (the geometric series) gives us this inverse: 146 +f? +--+. The sum is restricted to || < 1. Otherwise the series diverges. Example 2.5. What if is larger than 1? For a finite matrix we don’t notice the difference. The 4 by 4 inverse above is still correct. But the infinite series has to be written in powers of 1/8. The inverse matrix changes from causal to anticausal. Look first at the frequency response 1/H (w): iar iar iw iw 1 ees ee ee ae 06 I= pero ~ (ee/B) — This involves positive powers. We have advances instead of delays in the inverse. The difference between || < 1 and || > 1 is the difference between a zero inside the unit circle and a zero outside that circle. H(z) = 1 — Bz~! has its only zero at z = . Here is the general rule for inverses of causal FIR systems: No inverse when a zero is on the unit circle : I + S has no inverse. Causal inverse when all zeros are inside the unit circle: 1 — Bz~' has |B| < 1. Anticausal inverse when all zeros are outside |z| = 1: 1 — Bz~! has |B| > 1. A zero on the unit circle gives a particular frequency i at which H = 0. Then 1/H breaks down and the system has no inverse.44 10. ll. 12. aa 14. ae . For matrices show that SS~! Chapter 2 Filters Problem Set 2.1 . What matrix represents the inverse shift y = S~!x? In the z-domain, the input is X(z) = +++ x(0) + x(1)z-! +--+ and the output is Y(z) = —_. The advance $~' multiplies the z-transform by .. Express the filter H = $-+S~! with coefficients (—1) = 1 and A(1) = 1 inall three domains. Write the matrix H with two nonzero diagonals. Write the transfer function H(z) with two terms. Write the frequency response H(e/”) or H(w), and check that this is the transform of the impulse response h = H-. . Show that H=S+S~' is not invertible in three ways. Find a nonzero input. x such that Hx = 0. Find a frequency w that has response H(e/®) = 0. Find a number with |z| = 1 such that H(z) =0. }. What are the matrix H and coefficient vector h for the 3-term moving average Hx(n) = Lx(n) +.4( — 1) +.x(n — 2)? This is not invertible. Find two vectors x for which Hx = 0. Find two numbers with |z| = I such that H(z) = 0. Find two frequencies such that H (@) = 0. . Express this 3-term moving average in the form H = )*h(n)S". What is N? Find the output y when the input has x(0) = x(1) = 1. In the z-domain what are X(z) and Y(z), and how are they related? What is the corresponding statement in the z-domain, about 2 the transfer functions of S and . Multiply the matrix S by itself. The product H = S? corresponds to what coefficient vector h and what transfer function H (2) . Every filter > h(n)$" commutes with a delay: h(n)S"*! is HS and also SH. Why does every filter commute with every other filter? . If the continuous-time signal is (1) = cos ¢, what is the period T' that gives sampling exactly at the Nyquist rate? What samples x(n7) do you get at this rate? What samples do you get from x(t) = sins? If the sampling period is T = 1 and the continuous signal is x(t) = e7*!"/5, describe the discrete signal x(n). Is it periodic? Find two other frequencies w such that x(t) =e’ would give the same samples. If the signal x(¢) has bandwidth 3 Khz, then the sampling rate must be at least __ to avoid aliasing, Note that the sampling period is generally normalized to T = 1. Then the largest digital fre- quency is @ = xr. Our graphs of H(w) do not extend beyond x. Why is the downsampling operator (| 2).x() = x(2n) not time-invariant? Give an example with (12) Sx # S({ 2x. When are these filters invertible? Which has a causal inverse? Which has an FIR inverse? Which is allpass with |H(e!)| = 1? H(z) (sag (l= Bz") H(z) 1+ Be! +622 + BeF+--- Hy) = (@— B)/(1— Bz!) Hz) = 1-Brt+z? Determine the range of and 6 for which the LTI system with impulse response h(n) = a"; n20 . : =cC) { Bw co stable. Find the output y(n) when x(n) = (—1)". Determine the impulse response for the cascade of the two LTI systems having impulse re- sponses Ay(n) = (})" u(n) and y(n) = (})" w(x) using the convolution formula h(n) = Soa (hyn — k). Here u(n) is the unit-step sequence.2.2 Ideal Filters, Shannon Sampling, Sinc Wavelets 45 16. Let H; be a system that throws away odd-indexed samples: y(n) = x(2n). Is Hy linear and time-invariant? H, is the downsampling block and its operation is discussed in Chapter 3. x(n/2), even 0, odd n Is Hy a linear time-invariant system? Hp is the upsampling block and its operation is discussed in Chapter 3. 17. Suppose Hp inserts an extra zero between samples of the input: y(n) = 18. Which cascade of downsampling and upsampling is time-invariant and what is its impulse re- sponse? x(n) >| Hh >| etme x(n) >| Mh >| Hy > y(n) 19. Give two examples of LT! systems, two examples of linear time-varying systems, and two examples of nonlinear systems. 2.2 Ideal Filters, Shannon Sampling, Sinc Wavelets The word filter suggests that H selects a band of frequencies. It rejects another band. For @ in the passband, the frequency response is near to H(w) = 1. For @ in the stopband, the response is near to H(w) = 0. Any realizable non-ideal filter has a transition band in between, where Hw) changes from pass to stop (from near 1 to near 0). 1 ese 1 1 1 rene S a x = Ek z © 7 0 3 feaaaa 0) © 5 0 7 x sure 2.1: Ideal lowpass filter and best mean-square approximation with 20 terms. We begin with an ideal filter, which has no transition band. Its responses are exactly H(@) = land H(w) = 0. This is often called a brick wall filter, because of the step function in its graph. The response from an ideal lowpass filter is shown in Figure 2.1. This is a halfband filter, with sharp cutoff ato = 2. The response H(w) = oh(k)e~*” is 2z-periodic, and the ideal response is zero in the high frequency band from 4 to =: Ideal lowpass (0) = Yomyem = { i Os 27) What filter coefficients 4(k) produce this response? Multiply the equation for H (w) by e'"® and integrate from —7% to Te: oo. | * H(wye deo = | “(XS ade) ema, (2.8) t= G046 Chapter 2. Filters On the right side, there is an integral of e~*’e'"® for each k. The great property of complex exponentials is that this integral is zero except when k = n: x i(n-Ko 7 —ikw pina a4) — | & a i e do= [I 0 if kAn. (2.9) The function in brackets is periodic. It has the same value at —7 and z. After substituting those limits, the definite integral is zero. The complex exponentials are orthogonal. Now equation (2.8) has only one term on the right, from k = n. Integrating this constant from —z to m gives the result 277h(n). This equals the left side: _ | H(w)el"® dw = 2xh(n). (2.10) ‘The brick wall filter has H(w) = 1 on the part jl < %: : oe anhen) = f em dev =[ i | ee Qu) Ls in|, n°" 2 The coefficients in the ideal lowpass filter are samples of a sinc function: \ a sin Bee h(n) = =% 41) nodd (2.12) a 0, neven,n £0. The halfband cutoff has produced a halfband filter! The coefficient h(0) = 5 is the “DC term” = average value of H(w). All other even-numbered coefficients are h(n) = 0. When H (@) is antisymmetric around the halfband frequency @ = 4, the filter is always halfband. For odd n, the numbers /(n) alternate sign and decay slowly: nd) =n) =-4, hG) =h(-3) = a ae cid 3m Sx The series that adds up to the brick wall (= square wave = ideal lowpass response) is fe east asi H(@) = = — —*— + — SO ae = 3m Sree Ato = §, the only nonzero term is halfway down the brick wall: H ($) = 3. Most important is the behavior close 10 this jump at © = %, as shown in Figure 2.1 and below. Suppose we chop off the series after N terms: (2.13) The ripple at © = % gets narrower as N > 0 but its height approaches a constant (about 0.09). This is the Gibbs phenomenon. iW AA/\ | ‘Magnitude Response Normalized Frequeney2.2 Ideal Filters, Shannon Sampling, Sinc Wavelets 47 This Gibbs phenomenon can be a disaster numerically. The ripple represents error. It is ex- pensive to take a large number of terms and impossible to take all terms. A finite N gives the best approximation in the mean square sense — but the tall ripple remains. This “sidelobe” shows up as an echo in audio filtering and as a ghost in image processing. Practical design turns toward equiripple filters, which have many ripples of equal height. This design minimizes the maxi- mum ripple height instead of the total ripple energy. Note however that equiripple filters do not behave well in iteration. They do not lead to good wavelets. Minimax filter design is implemented by the Parks-McClellan algorithm, which computes best approximations to ideal filters. Those filters have a passband, a stopband, and a “don’t care” transition band. The ripple heights (maximum errors) decay exponentially with the filter length N. If the acceptable error is specified, there is a formula for N (see equiripple in the Glossary). An alternative is eigenfilter design. For many problems this allows a simple mean square calcu- lation, but without the big sidelobe from the Gibbs phenomenon. Historical note. It is surprising to read the original paper by Gibbs. He completely missed the Gibbs phenomenon. His correction published later was even shorter — about three important lines. This correction must have the highest signal to noise ratio in the history of science. Fourier’s Series _ by J. Willard Gibbs [Nature, vol. L1x, p. 200, December 29, 1898.] «Let us write f, x) for the sum of the first 2 terms of the series sinx — } sin2v + }sin3x — } sindx + ete. As n increases without limit, the curve defined by y = 2f,(..) approaches a limiting form, which may be thus described. Let a point move from the origin in a straight line at an angle of 45° with the axis of X to the point (xr, 7), thence vertically in a straight line to the point (7, —zr), thence obliquely in a straight line to the point (377, zr). The broken line thus described (continued indefi- nitely forwards and backwards) is the limiting form of the curve as the number of terms increases indefinitely. ... Correction [in Nature, vol. LIX, p. 606, April 27, 1899.] I should like to correct a careless error which I made in describing the limiting form of the family of curves represented in the equation y =2(sinx — sin2x-+- 4 1 sinnx) (2.14) as a zigzag line consisting of alternate inclined and vertical portions. The inclined portions were correctly given, but the vertical portions, which are bisected by the axis of X, extend beyond the points where they meet the inclined portions, their total lengths being expressed by four times the definite integral /* “du... But this limiting form of the graphs of the functions expressed by the sum is different from the graph of the function expressed by the limit of that sum. I think this distinction important, for (with exception of what relates to my unfortunate blunder described above) whatever differences of opinion have been expressed on this subject seem due, for the most part, to the fact that some writers have had in mind the limit of the graphs, and others the graph of the limit of the sum. The Gibbs phenomenon means that convergence to a brick wall is not uniform, as N in- creases. The coefficients h(n) approach zero but the sum of absolute values 1 + $+ $+ ---48 Chapter 2. Filters is infinite. These are multiplied by + sinne, so the actual series for H(w) does not blow up. But the slow 1/n decay prevents uniform convergence and allows the large sidelobe. This rip- ple always appears in the Fourier series near a jump discontinuity. Ideal Filter with Downsampling In the time domain, h(n) is on the nth diagonal of the filter matrix H. Writing a and b in place of h(1) = h(-1) and h(3) = h(—3), three rows are epee One peee Oia Oona 0 b O H= Ob Oe an aa Olcceae Ono Once s+ 0 DB 0 a O05 a 0 b 0 Those rows are not orthogonal! The dot product of the first two rows is a. Only an allpass filter has orthonormal rows (and columns). Then |H(@)| = 1 for allo. What is fundamental for this book is that the even-numbered rows of H are orthogonal. When the downsampling operator (| 2) removes half of the rows, this leaves a double shift in the re- maining rows —the rows of (| 2)H: . eee (2H = OM bt Org Osta tOn bu Out. 0b 0 a05 a 0b 0 Orthogonality is not so clear in this time domain. Moving into the frequency domain, the double shift is a multiplication by e~” and row 0 of H is orthogonal to row 2: 7 mf (row 0) - (row 2) = Hoje? Hoda = { dw = 0. /2 Similarly row 0 is orthogonal to row 4, because the integral of e is zero. This integral is over the half-period where H(w) = 1. The integral of e“® is not zero over this half-period, and row 0 of H is not orthogonal to row 1. The Ideal Highpass Filter Haar’s lowpass filter has coefficients } and 4, where the highpass filter has $ and —4. Those are clearly orthogonal. Now the ideal lowpass filter has infinitely many coefficients. We want to construct a highpass filter H), so that the rows of (J 2)H, will be orthogonal to the rows of (LDH. It is easy to make H() orthogonal to the ideal lowpass H(w). We set H; = 1 in the inter- vals where H = 0: 0 when O<|ol <4 1 when $< |o| <2. The ideal H,(w) can be | Shifted by zr, the ideal H(w) produces H,(@). In the time domain, that shift by 7 reverses the signs of the odd-numbered components (because h(k) changes to h(k)e!**): 1 1 1 hi) ==, ) =h\(-1)=+-, MB) =h\(-3)=-—., ... 2 X 3002.2 Ideal Filters, Shannon Sampling, Sinc Wavelets 49 H A |H@)| \ y! 1(@)| oe fees ° I t 1 — 0 ba Figure 2.2: Ideal magnitude frequency responses: low |H(w)| and high |H()| The graphs of |H(w)| and |H;(«)| are in Figure 2.2. We explain below why absolute values suddenly appeared. Whatever the phase, orthogonality is sure because of no overlap. We note that |H()|+|H(@)| = 1. This is not the identity that really matters. It is the sum of squares that applies not only in this case but in all orthogonal filter banks: |H@)P +|H@+zx)P = 1. (2.15) For the Haar example this is the identity cos” ¢+sin? $ = 1. The ideal case is deceptive because 1 = LP’ and 0 = 0°. The orthogonality requirement (2.15) will be established in Section 5.2. The Alternating Flip (with Odd Shift) There is an unusual point about the step from H to H,. In the time domain this usually comes from three operations on the coefficients h(n): reverse the order, alternate the signs, and shift by J (orany odd N). This takes the lowpass coefficients f(r) into an orthogonal highpass sequence: Alternating flip h(n) = (-1)"A(N —n). (2.16) For a finite sequence h(0), A(1),...,4(N) —assuming N is odd! —you immediately see the flip in the next figure and the orthogonality between rows: low —-hO)—RCA) high = -h(N) (N= 1) h(N 1) RN) (1) —h(0) Important: There is also orthogonality of double shifts, as in shift low by2—h(2)——h@3) “+ R(N-1) RN) high WN) -h(N=1) AG) —h(2) h(2)h(N) cancels —h(N )h(2). This happens for all double shifts. It does not usually happen for single shifts! A single shift of Haar to 0, 4 3 + js not orthogonal to the highpass 4 3, —3, 0. Double shifts are all we care about, because it is double-shifted rows in ({ 2)H that are orthogonal — and now the highpass rows in (| 2); complete the filter bank. The alternating flip is the key to orthogonality. In the frequency domain it has three steps: Multiply by e" to shift, take complex conjugates to flip, shift by x to alternate signs: Hw) = eH +a) (2.17)50 Chapter 2. Filters Magnitude Response in dB 0 005 O01 O15 O02 025 03 035 O04 045 O05 Normalized Freauency Theorem 2.1 The alternating flip makes the rows of (|. 2)H orthogonal to the rows of 2m. This was verified by eye, in the low and high rows above. You can verify it again for the ideal filters with infinitely many h’s: rowOofH --- b 0 (ieee aa 0 b 0 rowOofH; --- 0 -b 0 -a 05 -a 0 -b In the frequency domain we will see orthogonality in another form: H(o)H() + H(o+ 7) Hilo +7) = Hoe H(o+2) + Hot me’ H(w) = 0. (2.18) All great, but for the ideal filters there is a very strange point. There was no odd shift! From the brick wall H(w) on [—3, 3], we shifted by z to build the highpass brick wall. The wall is real, so conjugation has no effect. Still there should have been a phase shift from e'” and there wasn’t. Apparently row 0 of H is orthogonal to[—b 0 —a .5 —a 0 —b 0 -+-] without the shift. This unusual point occurs because H,(«) does not overlap H(w). We can give Hy(w) any phase we desire. It was natural to make it real. It is more consistent to include the phase shift ieee This oddity (actually it is a lack of oddity) will reappear below for ideal wavelets. The sinc wavelets should have a shift in time, from the phase factor e“ —but generally they are taken from the unshifted H,. Before turning to scaling functions and wavelets, we include a short discussion of the Sampling Theorem — to recover a band-limited function x(r) from its samples. This famous theorem appears everywhere, so we focus on a particular aspect involving (2). Shannon (Down-)Sampling Theorem Our main theme for filter banks is perfect reconstruction of all signals. With two filters this will be achieved in spite of downsampling. The Sampling Theorem restricts the input to a subspace of band-limited signals. Then one downsampled output is enough to recover the input.2.2 Ideal Filters, Shannon Sampling, Sinc Wavelets 51 The signal lies in the lower halfband |w| < 4; no higher frequencies are allowed. In an ideal filter bank, nothing comes out of the highpass channel. Full information must be in ({ 2)Hx. For a half-range of input frequencies, we only need a half-range of output samples. Suppose the output ({,2)x is an impulse 6 = (...,0,1,0,...). What was x? A first sugges- tion is x = 6, since (| 2)6 is equal to 6. But this is wrong — because 6 is not band-limited. The impulse has all frequencies in equal amounts. The correct input to yield (| 2)x = 6 has a halfband of frequencies in equal amounts. The graph of X (w) is a square wave: 2 for 0< lol <4 0 for ¥<|ol 2h(k)@(2r — k) comes from one application of the highpass filter (with downsampling) to #(¢). In the frequency domain this is aio= (5)05)=[ 0 chews, 29 The ideal filter bank cuts the frequency band in half. The upper half of the band goes through the highpass filter (discrete time). In continuous time it is a combination of wavelets. The lower half of the frequency band goes through the lowpass filter (discrete time). In continuous time this half is a combination of scaling functions ¢ — ready to be split again into wavelets and scaling functions at the next finer scale. We have an octave decomposition = logarithmic decomposition = “constant-Q decomposition” of the line of frequencies: ° w W (20) 0 T Qn 4n Note 1 Meyer smoothed out this ideal picture to produce band-limited wavelets with fast decay (IIR of course). The key is to keep > |Meyer(2"77@)|? = 1. This can be done smoothly with overlap of nearest neighbors only [D]. All band-limited wavelets have two bumps (+0) like the Shannon and Meyer wavelets. Problem Set 2.2 . Show that the inverse transform of @ in (2.23) is the sine wavelet w(t) = 2sine(2r) — sine(1). 2. Find the shifted sinc wavelet by inverse transform when the factor e“® is included in Hi(w). This is the odd shift that we normally need for orthogonality of w to ¢. 3. What are the coefficients h(n) for the ideal quarterband filter, with H(w) = 1 on [—%, 2]? What scaling function $(@) comes from the infinite product formula? Is (1) orthogonal to its translates? oe (Important) Show that H (w) has halfband symmetry (odd around its value at @ = 3) when A(n) is a halfband filter. This means (| 2)h = -. 5. Let hy p(n) denote an ideal lowpass filter with cutoff frequency «,.2.3 Lowpass and Highpass Filter Design 53 (a) Compute Hp (e!”) and normalized hy p(n) such that Hrp(e) = 1. (b) The ideal highpass filter fiz p(n) can be designed by 1-hyp(n); 2 =0, MeO = 1 ay pin); otherwise. Find Hyp(e’®). (c) Design a highpass filter with cutoff frequency at 7 — w., using hz p(n). 6. Let H(z) = (1-227! +3277 - 32-3 +2277 (w) and the group delay $'(w). 7. Let H(z) be an FIR lowpass filter of length (N + 1). Define Gy(z) = H(z) and G3(z) = H(z" ~5), Compute |H (e/)| and the phase response oN AG), Gre) = (a) What are g,(), g>(n) and g,(n) in terms of h(n)? (b) If H(z) is an even-length symmetric filter, what is the symmetry or antisymmetry of Gi (z), G2(z) and G3(z)? (©) If zo is a zero of H(z), what are the corresponding zeros of G,(z), G2(z) and G3(z)? (e) What are the relations of | H (e/”)|, |G, (e/”)|, |G2(e/”)|, and |G3(e/”)|? 8. Show that G(z) = H(z)H(z~') is a symmetric filter, What type of filter is G(z) (lowpass, bandpass, highpass) if H(z) is highpass? What is the (constant) phase of G(z)? 9. We have stated that 5, = 4, in a halfband filter. Prove this. 2.3 Lowpass and Highpass Filter Design The previous section dealt with ideal filters (necessarily IIR). This section deals with real FIR filters — often symmetric or antisymmetric (thus linear phase). We indicate the goals of filter design and we briefly discuss design methods. For ideal brick walls, the transition from H(@) = 1 to H(w) = 0 happens instantly. For FIR filters this is not possible. It is important to see an actual magnitude response graph (Figure 2.3). In normal scale, we can observe details in the passband but not in the stopband. In logarithmic scale (dB scale, plotting 20 log,y |H|), itis the other way around. The stopband details are visible in dB scale but the passband details are lost. Before moving to good lowpass filters, we review two very short and rather poor filters. This gives us a chance to emphasize the four types of linear phase filters — odd and even length, sym- metric and antisymmetric. Example 2.6. The impulse response is h = 3. 4). The frequency response is (eee Neto = (cos Z)eriv® 28) In that last form you see the magnitude |H (w)| = cos $ and the phase $(@) = —$. The mag- nitude is cos = 1 at zero frequency —this is a lowpass filter. The magnitude drops to zero at @ = 1. The phase @(«) is the angle —w/2 in the polar form re'®. This phase function — linear in w. The noninteger 1/2 reflects the fact that the coefficients in h are symmetri the “1/2” position. H@) = 3(1+e%54 Chapter 2. Filters |H(] in normal scale [Hi] in dB scale, f= ©/2n ae 08 ost o4 l \f\ f\ ANAK \f\ N\f\ -40, HI | \ \ | | \| \ | a J | WHI | ll I % O41 02 03 04 Os 8 O41 05 Figure 2.3: Magnitude response of a lowpass filter in normal and dB scale. We will compute the phase #() for other filters before analyzing its significance. Here we only mention: linear phase is a desirable property. Example 2.7. By cascading the previous example we square its matrix H and we square its frequency response. The new filter coefficients are in 0.5 : 0.25 05 0.5 0.50 0.25 090.5054] 510105110501 00511] Baw. 1) with itself. Cascading fil- The new impulse response is (+, }, +). It is the convolution of (5, + the frequency domain we mul- ters means convolution of impulse responses (time domain). In tiply responses H (w): Hew (@) = [Hoia(@)P = 41 +e) (cos? 5)e. (2.25) This squares the magnitude and doubles the phase. The new phase —e is still linear, It corre- sponds to a time shift of 1. The impulse response is a unit delay of a symmetric A. Hyew is a shift (= delay) times a symmetric matrix. Every FIR matrix can be made causal, by sufficiently many delays. A symmetric matrix (not causal) corresponds to phase = zero because Hw) is real: Hyym(@) = fe + 4+ ye1” = $1 + cose). (2.26) Was this cascade desirable? Neither H or Hyew is very impressive. The frequency responses are far from ideal. Hyey () has better attenuation in the stopband, because the cosine is squared. But it is also smaller in the passband — farther away from the ideal H = By alternating signs in the lowpass coefficients they become highpass: hy = (0.5, —0.5) and hy * hy = (0.25, —0.50, 0.25). These are still linear phase. The alternation of signs is a phase shift (a modulation) by xc. The first is antisymmetric but the second is still symmetric.2.3 Lowpass and Highpass Filter Design | It is useful to tabulate the four types of linear phase filters with real coefficients. N can be odd or even. The coefficients can satisfy h(n) =h(N — n) for symmetric, h(n) = —h(N — n) for antisymmetric. The linear phase $(@) = —@N/2 and the (real) amplitude response He(w) are seen in H (e!®) = ce~J°N/? Hp(w). The table shows that with odd N, symmetry guarantees a zero at © = 7 and antisymmetry guarantees a zero at @ = 0. The responses Hg(@) in the table have factors cos ¥ and sin ¥, Remember that the filter length (number of taps) is N + 1. Here ¢ = 1 for symmetric and c = j for antisymmetric filters. ‘Type I Type 2 Type 3 ‘Type 4 even N =2K odd N =2K +1 even N =2K odd N =2K +1 symmetric symmetric antisymmetric antisymmetric He = 4 b, cosnw | cos$ Pf b, cosnw | sinw Py |b, cosne | sing DY b, cosnw zero at @ = 0 zeros at @ = 0, 7 zero atw = 0. Now consider the ideal lowpass filter with cutoff frequency «.. This is a band-limited filter in the frequency domain, therefore its support in the time domain is infinite. It must be IIR. Its impulse response is fi7(n) = 72%, Since the time-support is infinite, one needs to approx- imate it by a finite impulse response. The sequence (1) becomes time-limited, therefore not band-limited. The magnitude response | (e/“) | typically has errors 6,, and 6, in the passband and stopband (Figure 2.4). Those bands have cutoff frequencies «, and «,. Figure 2.4: Idealized and typical magnitude responses of a lowpass filter. Given wp, @,, and N, the errors 8» and 4, cannot both be small. Let W = b/a be the rel- ative error weighting. Increasing W in the design algorithm will decrease 3, and increase 5p. Figure 2.4 shows a magnitude response plot for a lowpass filter with length 19 and wp = 0.27 and w, = 0.37. The error weighting W is large (and thus yields a small stopband error). In the sections below, several methods for the design of FIR digital filters are reviewed. These are based on windowing, minimax criteria, and weighted least squares.56 Chapter 2. Filters Design by Windowing The simplest way to truncate the ideal response h is by a rectangular window: 1, In} 1- | /1o(8) nis % The parameter f in the Kaiser window controls the attenuation of the lowpass filter. Jo(2*) is the modified Bessel function and practical designs use about 20 terms of Sf (0.5x)2k PP foes) = 14 | OP) : ra ! Minimax Criteria (Equiripple Filter) The filter with the smallest maximum error in passband and stopband is an equiripple filter. The equal heights of the ripples (and the number of ripples) assure that the error cannot be reduced — some ripple sizes will go up if others go down. A polynomial of degree N cannot have alternating signs at all ripples. The Remez algorithm to equalize the ripples was adapted to filter design by Parks and McClellan. Figure 2.5 shows the magnitude response plot of an equiripple lowpass filter. Given a fre- quency specification in terms of cutoff frequencies (wp, @s), filter length (N + 1) and relative errors (5,, 5,), the equiripple filter has the smallest maximum error in the frequency interval 0 < w < 2. The design algorithm for equiripple filters is the Remez exchange (McClellan- Parks) algorithm. The order (NV) of an equiripple filter is estimated by fae —20 logy) Vii: — 13 27 14.6Af where Af = (@, — w,)/2z is the transition band. Equiripple designs are optimal in important respects — but not optimal for iteration. The reason is that they have at most one zero at @ = 2. The sampling operators (1 2) will mix up the frequency bands that an equiripple filter so carefully separates! The Daubechies filters go to the other extreme —no ripples at all and maximum flatness at @ = 1. Then iteration of2.3 Lowpass and Highpass Filter Design 57 a ~ a = al \ 2 \ 5 x) | z i - 3 ANNAN fo. HEN Normalized Frequency Noxmalized Frequency Figure 2.5: Magnitude response of an equiripple filter: normal and dB scales. (1 2)H is very stable. But Section 5.5 will show that the transition band Aw widens from N~' for equiripple to N~? for the maxflat Daubechies filters. A compromise is certainly possible, constraining the minimax design to have a fixed number p of zeros at @ = 7. Weighted Least Squares (Eigenfilters) The eigenfilter approach chooses the filter H to minimize a (weighted) integral error poe i |D(w) — H(e!)|” (weight) do (2.28) The integral is over the passband and stopband, not the transition band. D(w) is the desired frequency response, possibly one and zero in the two bands. The weighting function is optional. The goal is to express the error as a quadratic form E = h" Ph. The unknown filter coefficients are in the vector h. The matrix P is symmetric positive definite because E > 0. If the normalization has the form h’h = 1 then the minimization is an eigenvalue problem Ph = Ayyin/t in linear algebra: T The minimumof E = PB is An(P). hhh If the normalization is changed to h” Qh = 1, the eigenvalue problem Ph = 4Qh involves both matrices. When there are several quadratic constraints h’ Q,h = 1, we go beyond an eigen- value problem —to the Quadratic Constrained Least Squares algorithm. Section 5.4 will apply this QCLS method to filter design. The applications of eigenfilters (one quadratic constraint) are very extensive, and we give two examples: lowpass filters and halfband filters. Lowpass eigenfilter design: A symmetric filter h(n) = h(2L — 1 — n) of length 2L has re- sponse 2-1 Hel®) = ame =e Pint 9S meet. n=058 Chapter 2. Filters The last sum is Hyeq(@) = h"e(@) with h = [h(O) --» h(L — 1)]. The vector e(@) has com- ponents 2 cos(L — do, ...,2cos . In the stopband, from w, to 7, where D = 0 is the desired response, the error is Eon = f H2(o)deo = i" i “ e(wje(o)Pdeoh = h™ Pop h. The entries of Psop are known integrals of cosines. In the passband from 0 to wp, we can nor- malize the desired constant response to be D = h™ ¢(0). Then the passband error involves the difference between that desired response and the attained response h” ¢(w): Epass = hh" i (c(0) — e(@))(€O) — e(@))" doh = h" Press h. 0 The entries of Ppass are known integrals of 4[1 — cos(n + $)e][1 — cos(m + 5)o]. We can weight the errors by E = ot Estop-+(1 —@) Epass. The matrix whose lowest eigenvector is the best h will be P = @Pyop + (1 — &)Ppass- Figure 2.6 shows the magnitude response of a lowpass filter of length 55 and cutoff frequencies @, = 0.2 and w, = 0.37. Here, the weight ais 0.5. . 2 5g g bbbhbiae Worn un Normalized Frequency Normalized Frequeney Figure 2.6: Magnitude response plot of a lowpass eigenfilter in normal and dB scale. Halfband and Mth-band filter design: In many digital systems, a change in the sampling rate is essential for efficiency in real time. The design of suitable filters for the rate changing operations is important. When the subsampling is by a factor of M = 2, we are led to halfband filters. When we keep every Mth sample, we need Mth-band filters. The center coefficient is h(0) = | and all coefficients at multiples of M are zero: Mth-band: h(nM) = 6(n) or in other words (| M)h = 6. (2.29) The filter banks have two channels or M channels. The design begins with a halfband filter or an Mth-band filter. This is the product filter of Section 4.1, which is factored into analysis times synthesis. We shift the filters to make them causal, for implementation. But for design we keep them centered, and simply zero out the coefficients h(nM) for n # 0. This zeros out the corresponding rows and columns of the error matrix P in weighted least squares. The optimum his the lowest eigenvector of the reduced matrix Pea, and this h is Mth—band.2.3 Lowpass and Highpass Filter Design 59 Design procedure: Given N, M, w), ©; for an Mth-band filter, find P using the eigenfilter formulation. Find Preg by deleting the rows and columns of P that correspond to zero coefficients inh. Find the eigenvector hyeq corresponding to the minimum eigenvalue of P,.g. The optimal impulse response h(n) is obtained from yea(n) by inserting the zero coefficients. cea 2 a zg 2 é 2 0 3 3 z 2 = =~ Niu =} Anna a WH unl N {nan Normalized Frequency Figure 2.7: Impulse response and magnitude response of a 4th-band FIR filter. Figure 2.7 shows the magnitude response of an 4th-band filter with @, = 0.2 and @, = 0.37. Note that wp + @; = 0.51 = 34, as required for a symmetric 4th-band filter. The length is 55 (odd length is also required). Maximally Flat Filter (Maxflat filter): The design of a maxflat filter begins with a maxflat polynomial f(y). This polynomial of degree 2p — 1 is determined by p conditions at y = 0 and at y = 1: H® (0) = 6(k) and H (1) =0 for O 0 and zero for cos <0). The p~'/? width of the transition band is established in Section 5.5, together with the approximation of the zeros of H(z). The ideal is infinitely flat. Of course it needs infinitely many coefficients. Problem Set 2.3 . Truncate the ideal lowpass filter after three nonzero coefficients. What is this windowed filter h(n)? Sketch the graph of H (w) 2. Truncate the ideal lowpass filter after 4 terms and 8 terms. Draw the frequency responses H () to see the Gibbs phenomenon. 3. Compare the graphs of the ideal brick wall filter truncated after 20 terms (rectangular window) and the Kaiser window w(n). Choose a suitable Kaiser parameter 8. What are the maximum errors in the passband? 4, Use MATLAB to design equiripple halfband filters of length 8 and 20. Compute the height of the ripples. 5. What is the frequency response for the maxflat Daubechies filter (2.31) with p = 2? Graph H() by hand or by computer. What are its symmetries? 6. Graph the maxflat Daubechies filter response (2.31) for p = 8. What are the differences from the truncated ideal filter and the equiripple filter? 7. Construct a 4-tap lowpass filter that you approve of. What properties have you achieved? 8. Derive the Bernstein form (2.32) of the Daubechies polynomial H (e/”) from her original form 231). 9. Formulate the eigenfilter design for a highpass filter with cutoff frequency @. 10. Compute h(n) for the halfband Daubechies filter with p = 5. Verify that H (e/”) has four zero derivatives at @ = 0 and w = 7.2.4 Fourier Analysis 61 2.4 Fourier Analysis This section might be called “Notes on Fourier Analysis.” The subject is enormous — too large for our book! We pick out key points that are needed for signal processing and for wavelets. The Fourier transform of a signal x() is a function X(w). The signal is in the time domain, X (w) is in the frequency domain. The time variable n is discrete, the frequency variable w is continuous. X (w) is 2-periodic because each exponential e'"” is 27-periodic: XO) = ye tiene: (2.33) n=—00 Fourier analysis studies the connections between x(n) and X(w)—how the properties of the signal are reflected in its transform. The inverse Fourier transform recovers x(n): x(n) = xf X(w)e"de. (2.34) ond, This formula synthesizes x by combining the complex exponentials. To find x(N), multiply equation (2.33) by e'® and integrate from —x to 1. The integral of e7!” times e'N” except when n = N. That integral is 277, leading to (2.34). Fourier analysis usually starts with f(t) and computes its coefficients. Signal processing starts with the coefficients x(n) and transforms to X (w). ine is zero Note about orthogonality. Real vectors are orthogonal (perpendicular) when x - y = 0. Real functions are orthogonal when f X(w)Y (w)dw = 0. If the vectors or the functions are complex, there is a small but important change. We take complex conjugates of one vector (say x, in the physics convention) and of one function X (w). Orthogonality in the complex case means ¥-y=ox@ym) =0 and (X,Y) = [ XO) (w) do = 0. (2.35) xv) = f X(@)Y(w) do. (2.36)62 Chapter 2. Filters For proof, substitute > x(W)e!%® for X (@) on the right. Also substitute > y(n)e~"® for ¥(w). Integrate from —z to x. By orthogonality, the only terms with nonzero integrals are those with N =n. The integral is 2z, multiplied by the number x(m)y(n). The left side of (2.36) is the sum of these nonzero terms. Special case when x = y. Now the two vectors are the same. Their transforms are the same. We are integrating X(@)X (w), which is |X (w)|?, because a + ib times its conjugate a — ib is a’ +b”. For the same reason x(n)x(n) = |x(n)|?. With x = y, the energy in the signal (times 2st) equals the energy in the transform: Qn » be)? = i IX(@)P do. (2.37) Example 2.8. Suppose x is (1, 6, 6?,...). Its energy is 1 + 6? + 64 +++» = 1/(1 — 6°). Its transform is a one-sided sum, in this case a geometric series: =. 2 1 X@)= Yipte = 14 fev + (Be) fo = Tope 9 Consider the inverse transform from X (w) back to x(n): Lx i Lf dey x(n) = — (wedo = > | ee How do we see that this integral gives the correct signal (1, B, B*, ...)? The direct way is to write the integrand as e'"® (1 + few!” + Be” + .--). Integration picks out the correct power B". (ifn < 0 then the integral gives zero.) The indirect way is to substitute z for e’”, and integrate z"/(z — B) around the unit circle: mode [ igj=t wl #18 ‘There is a pole at z = B. The residue at this pole is 6". This is the answer we want. Again the case n <0 is separate (with two poles) and gives zero. The actual calculation of such an inversion integral is generally difficult or impossible. We seldom need to do it. For a ratio of polynomials it can be done in an emergency by the residue method of complex integration. Example 2.9. An allpass filter has |H («)| = 1 so that |Y ()| = |X(«)|. The filter conserves energy. The integral of |Y («)|? equals the integral of |X (w)|?, because these functions are the same: 7 Allpass: i IY) Pao = W@x(odo= f [X(@)P?do. x x -" Therefore the output energy equals the input energy: = 7 YLv@e? = ke@P? or iyi? = IP. (2.38) = oe Please do not think that each |y(m)| = [x(n)|. It is the frequency response that has |H (@)| = 1. The energy in each frequency band is conserved by an allpass filter, not the energy in each time band.2.4 Fourier Analysis 63 Convergence of the Fourier Series In defining X (w), we have been assuming that this series converges. Otherwise what meaning do we give to X (w)? We are touching here on a central problem of mathematical analysis (with a literature that goes back for centuries). Touching this problem is as much as we can do— by identifying three types of convergence, and the signals that produce each type. As with all infinite series, a few terms have nothing to do with convergence. By changing a finite number of inputs x(n) we certainly change the transform— but we do not alter conver- gence or divergence. It is the behavior of x(n) for large n that is crucial. Here are three types of convergence: 1. Uniform convergence with > |x(n)| < 00 2. Strong convergence (in L?) with }° |x(n)|? < 00 3. Weak convergence allowing polynomial growth in x(n). 1. Uniform convergence Suppose the magnitudes |x(n)| have a finite sum. These are also the magnitudes of x(n)e~"”, because |e~!”®| = 1. Those terms have different phases, which may produce cancellation when we add them. When 5= [x(n)| converges, we don’t need that help. The series of magnitudes converges “absolutely,” and the series 2° x(n)e~"® converges “uni- formly.” Then the sum X (w) is a continuous function — with no jumps. In the example with x = (1, 8, 6°, ...) we imposed |8| < 1. That produced uniform con- vergence. The transform X (w) = 1/ (1 — Be~‘*) is a continuous function. But not all continu- ous functions have )° |x(n)| < 00. In the brick wall filter, the odd coefficients have magnitude |A(n)| = * The sum of magni- tudes does not converge. The terms = do not go to zero quickly enough. The sum H(w) = YA(nye-i"® does not converge uniformly. And, in fact, H(c) is a step function (or square wave) with a jump. 2. Convergence in energy (L? convergence) Suppose the squared magnitudes |x(n)|? have a finite sum. By squaring, small terms become much smaller. Convergence is easier to achieve. If the sum of |x(n)| is finite, the sum of squares is certainly finite. Then the Fourier series converges in L?. This “squared” test is passed by the Fourier series for a step function: 1 : 1 Ih@n)| = = has Yaw! aes 5 Comparing the sums of 1/n and 1/n? is like comparing the integrals of 1/x and 1/x?. One in- tegral is logx, which becomes large as x > 00. The area under 1/.? stays finite as x + 00. When the squared magnitudes |x(n)|? have a finite sum, the squared magnitude | X («)|? has a finite integral. They are equal apart from 27. Then X(«) is a function in the Hilbert space denoted by L?, just as x(n) is a vector in the Hilbert space denoted by £2. These spaces contain all functions and vectors with finite energy: the square of the L? norm or the €? norm is the energy. Functions in Hilbert space may have jumps. Those jumps have no energy (no contribution to the integral). The derivative of a jump is a Dirac delta function. Its coefficients are all x(n) and its energy is infinite, so this function is outside the space L. But the delta function is included in the third type of convergence. = convergent series.64 Chapter 2. Filters 3. Weak convergence (to a distribution) Distributions F(@) are defined by their inner products with smooth functions G(w). For f F(w)G(w) dw, use integration by parts. We integrate F (w), which needs it, and we differentiate G(w), which can take it. The indefinite integral of the delta function F(w) = 5(w) is the unit step H(w). The definite integral of 5(@)G(«) is the number G(0): 5(@)G@) dw = [H(@)G(o) |", -f H(w)G'(w) dw = G(x) — fl G'(@) dw = G0). Ee 0 This defines 5(~). It is a distribution, a derivative of a true function. Still it has Fourier coeffi- cients that are easy to find, with G(w) = e"®: ce ele a= 5 [ore do = 5-1. All frequencies are present in the same amount. The Fourier series is > 8(o) = — eine, (2.39) ote aoe On the left is a distribution. On the right is a divergent series. The terms don’t even approach zero. But in a weak sense those terms cancel each other to produce zero, away from the spike at @ = 0 where they reinforce. Weak convergence is based on the same idea of testing inner products with smooth G(@). The series converges weakly, say at @ = 0: G0) = | 5@)Gw)do = y( [ * ene (w) de)e”. (2.40) You see the Fourier coefficients of G(w) on the right side, adding to G(0). Numerically, this weak convergence is not so great. Figure 2.8 shows the sum of 41 terms. In a pointwise sense, and in area, those side lobes will not shrink to zero! In the Ly sense, the energy is growing with every term. But in a weak sense, this sum is approaching 6(w). As the number of terms increases, the oscillations become faster (not smaller). Multiplied by a smooth G(@), the main central lobe picks out G(0) and the integral over the oscillations approaches zero. Question: What is the weak limit of pure oscillations e'"° as n + 00? Answer: The limit is the zero function. The inner products f e"°G(w) dw approach zero. Os- cillations converge weakly to their average value. Question: Does the series 8'(w) = = Yn el" for the derivative of a Dirac function also con- verge weakly? The Fourier coefficients are growing with n. Answer: Yes. Integrate again by parts. The integral of 5’(«)G(w) is —G’(0). The Gibbs phenomenon is just the integral of Ye”. The integral of 6(w) is a step func- tion. The integral of e'"® introduces 4 so there is L? convergence to the step (but not uniform convergence). With integration, the area under the side lobes become crucial. The area shows as a height in the Gibbs figure (the integral of the delta function is a step). This undesirable os- cillation of Fourier series at a jump (an edge in image processing) has brought forward localized bases like wavelets.2.4 Fourier Analysis 65 height 41/27 Figure 2.8: The Fourier series + Se! converges weakly to 8(w). Its integral shows the Gibbs phenomenon (nonuniform convergence). It is significant that the same Gibbs difficulty in shock calculations has been handled very differently. The finite difference schemes are made non-oscillatory by nonlinear terms. The nonlinearity is active where it is needed. It is inactive where the solution is smooth. Morel’s book [Mo] gives a strong impetus to this nonlinear idea for image processing. For iteration of the filter h = 0,0, >, we will see weak convergence in Chapter 7. The cascade algorithm (= lowpass iteration) produces functions that oscillate between 1 and 0 on the interval [0, 3]. They have no limit in L?. The weak limit of the oscillations is a stretched box @(t) with constant value 4. Good filters give L? convergence (and usually uniform convergence) when iterated. The necessary and sufficient Condition E is in Section 7.2. But a “good filter” in the classical sense, with small errors in the passband and stopband, may fail in iteration! The requirement for success in iteration is flatness of the response H(w). The number of zeros at @ = 1 is absolutely critical. This is the new property that has become important. It makes the iteration process strong and not weak, regular and not oscillatory. It must be built in or iterations will oscillate — as happened with highly regarded filters. Poisson’s Summation Formula The Fourier coefficients of the Dirac delta function on [—z, x] are all =. By periodicity, the Dirac delta becomes a Dirac comb. There is an impulse at every multiple of 2x. The Fourier series for this periodic train of delta functions is Y 6-278) k=—00 eine, (2.4L) The left side is a sum of impulses. The right side is its Fourier series, converging weakly as above. The formula is often seen with t instead of w, because Fourier analysis usually starts with continuous time. Then the frequencies are discrete. In signal processing it is the other way around.66 Chapter 2. Filters Equation (2.41) is a short statement of the Poisson formula, but it involves weak conver- gence. The terms don’t approach zero. To see ordinary convergence we take inner products with any smooth function G(@). With integrals over the whole line, we ask G(@) to decay as |o| > oo. The inner product with delta functions gives point values G(2zk). The inner prod- uct on the right gives point values G(n) of the Fourier transform. The formula says that the two sums are equal: a : 1 oo Poisson's Summation Formula =~ Y) GQmk) = 2S Ga). 2.42) TT =o This is a remarkable formula. It relates the samples of G to the samples of its transform G. We can change the spacing of one set of samples to any T > 0, provided the other set of samples is spaced at 2%. Smoothness of X (w) and Decay of x(n) The regularity of a function is partly revealed in its Fourier coefficients. When X («) is contin- uous, its Fourier coefficients x(n) approach zero. But the coefficients can approach zero when X(«) is not continuous; the step function is an example. There is a gap between x(n) > 0 and ¥ |x(n)| < 00, which is at the heart of Fourier analysis: > Ix(n)| < 00 => continuous X (w) => x(n) > 0. One virtue of wavelets is that such gaps can be closed. Instead of necessary conditions for smoothness, and then sufficient conditions for smoothness, we can find necessary and sufficient conditions. These are conditions on the wavelet coefficients x(n), for the function X (@) to have specified regularity. X (w) lies in a specified function space when x(n) lies in a corresponding vector space [DeLu]. The function norm and vector norm are equivalent —as for X(w) in L? and x(n) in €? by Parseval’s formula. Fourier coefficients give partial information from the magnitudes |x(n)|. Each extra order of smoothness in X («) is reflected in one extra order of decay in |x(n)|. Theorem 2.2. Suppose X(«) has s continuous derivatives. Then n° |x(n)| — 0 as |n| — 00. When s = 0 this is the Riemann-Lebesgue Lemma. The Fourier coefficients approach zero (we really only need f |X ()|dq to be finite). For integers s = 1, 2, 3, ... we look at the Fourier co- efficients of the derivative X\)(). Those coefficients are (in)°x(n). They approach zero (again by Riemann-Lebesgue) when X‘)() is continuous. This is the theorem. In compression of a signal, this decay of coefficients is crucial. Small coefficients are re- moved (partly or completely). For a Fourier basis, the smoothness of an image determines the decay of coefficients. For a wavelet basi are well adapted to piecewise smooth functions (with edges). Wavelets are local where Fourier waves are global. The theorem was stated for integers s = 0, 1,2, ... but fractions can be allowed. This is important for a satisfactory theory, because functions like X (w) = |o|!/? are more than contin- uous (s = 0) and less than differentiable (s = 1). The in-between smoothness is measured by the Hélder exponent. The function X («) belongs to the space C*, 0 < s < 1, if it satisfies the Hélder condition , it is the piecewise smoothness that matters. Wavelets |X(@2) — X(w1)| < constant Jor — wil «2.4 Fourier Analysis 67 The square root function ||'/? belongs to C!/?. Similarly |@|° belongs to C°. For s > 1 we would take [s] derivatives first. The Hélder exponent of that derivative is the fractional part — [s]. Then the Fourier coefficients decay to order s at least: n*|x(n)| > 0. The Fourier basis automatically takes advantage of high regularity. The wavelet basis takes full advantage of a high s, only if it is designed to do so. The lowpass filter must have more than s zeros at 7. Haar wavelet expansion does not converge faster as the function gets smoother. Therefore Haar compression is poor. For biorthogonal wavelets, the analyzing filter Hy governs the decay of coefficients. The synthesizing filter Fy governs the smoothness of the output. We would like both to have many zeros at =! When forced to choose, whether to put zeros into Ho or Fo, the preference often goes to Fo —then the synthesis basis functions will be smooth. We emphasize one more point. In the L? norm, where “mean square” replaces “maximum,” the Fourier coefficients exactly reflect the smoothness. The coefficients are in €? precisely when the function is in L?. The energy is the same for both, by Parseval’s identity. This equality ex- tends to the sth derivative X'*)(w), whether s is an integer or not: f |X) do = 20 J |n'xiny). (2.43) When s is not an integer, we define this derivative X)(w) by its coefficients (in)’x(n). The equation above is just Parseval itself, for the derivative. Wavelet theory is greatly simplified by working in the Hilbert spaces of functions with s derivatives in L’, rather than the Hilder spaces C* of functions with s continuous derivatives. The number p of zeros at is crucial in both cases. For L? spaces we will find in Chapter 7 the exact smoothness of #(t) and w(t). Heisenberg’s Uncertainty Principle The underlying property of wavelets is that they are pretty well localized in both time and fre- quency. The functions e' are perfectly localized at @ but they extend over all time. Wavelets are not at a single frequency, or even a finite range, but they are limited to finite time. As we rescale, the frequency goes up by 2/ and the time interval goes down by 2/. This suggests that the product of frequency interval and time interval is a stable quantity. The Heisenberg Uncer- tainty Principle makes those definitions precise, and gives a lower bound for the product. ‘We emphasize that the wavelet can only be “pretty well localized.” It cannot have finite sup- port in both ¢ and w. (A famous theorem.) Instead of the support length we use the variances 2 4 2 2 i bt - nef Pif(Pdt and G = x/ o | flo)? do. (2.44) 7 Ms 0 ‘We are no longer working with periodic functions, so ¢ and w extend from —oo to co. Theorem 2.3 (Heisenberg) —_ /f || f || = 1 then the product o@ is at least 5. The lower bound } is attained by the Gaussian function f(r) = e~". Its transform f(«) is also a Gaussian. These have infinite support! But they are as local as possible, measured by o and@. We can rescale time by c and frequency by +, we can shift to tf, and we can modulate by e'". The variances 0 and 6? would be computed around fp and wp, and the bound + is still reached.68 Chapter 2. Filters This led Gabor to use Gaussians in constructing “time-frequency atoms.” But numerically, finite support in time is better. Proof of the Heisenberg Principle. The key is that o = ||rf(t)|| and@ = || f’()||. The principle applies to pairs of operators, like position P and momentum Q, that have the property QP—PQ = T. When this property holds, the proof comes directly from the Schwarz inequality: 1= [IFIP = (%. QP -PO)f) <2 OF IPfl| = 20. (2.45) In our case P is multiplication by ¢ and Q is differentiation: Say ; (QP — PQ) F(t) = (Ff) —1F- = LF as required. Problem 4 develops this proof directly from the definitions of o and @. Problem Set 2.4 1 1, Add the terms oe". This is a partial sum of the Fourier series for 5(). At what o does at Ww it come down to zero, at the end of the main lobe in Figure 2.8? Where is the end of the first side lobe? 2. Describe a continuous function X (w) whose coefficients have infinite sum > |x(n)| = 00. 3. What is Poisson’s summation formula for the Gaussian G(w) =e? 4. Heisenberg’s Uncertainty Principle is o@ > }. The Schwarz inequality gives |[roroa s furora firora. Identify the right side as o?G?. Integrate by parts on the left side to get ~ 7 far |~ = fo? fore) at wo - fea. If the integrated part is zero at -too, and if || {|| = 1, deduce that o@ > } Equality holds when /f(¢) is proportional to f’(1), which leads to Gaussians f(t) = e~ 5. What power of + gives the decay rate for the coefficients x(n) of . X(@) = w = discontinuous function at @ = 7. . X(w) = . X(w) = @ = continuous function with period 27. |. X(w) = spline with jump in the third derivative X’"(w). o| = continuous function with period 2:1 Beye 6. Compute the Fourier coefficients and the energy for 1 Te-ie —He (a) X@)= (b) X'(w) 1 O<@ 0as N > oo. For L? spaces this norm is the square root of the energy. Section 1.5 mentions four reasons for that choice. There are two separate properties to be established for any proposed basis: (1) linear inde- pendence and (2) completeness. Adding extra vectors will destroy independence. Removing vectors from the basis will kill completeness. Linear independence is automatic for orthonor- mal vectors. When all angles are 90° and all lengths are unity, there is no chance of degeneracy. In infinite dimensions we meet the possibility that angles can approach zero without reaching zero. In that case the basis is unstable. The coefficients b; are out of control. In the good case, the coefficients satisfy All? < le? < Bly? with A>0. (2.46) This is the defining property of a Riesz basis, also known as a stable basis or an unconditional basis. A key assumption in wavelet theory is that the translates ¢(t — n) of the scaling function are a Riesz basis (for the space Vp inside L?). We will find the test to be applied to #(«), and the equivalent Condition E on the filter coefficients, to produce a Riesz basis of scaling functions. Dual Bases and Dual Frames Ina few lines, we can give the main points about bases and frames. These ideas will be developed. below — bases will come first. Here are the key points in a hurry: Dual bases come from columns vy, of T~! and rows r, of T: T7'T =I Dual frames come from columns v7 of T+ and rows r, of T: T*T = I. The difference is that the frame vectors v;* need not be independent. They can be redundant (Figure 2.9). We still require the coefficients b; = (r;, v) to satisfy (2.46), but other combinations of the vt can reconstruct the same v. T* is only a left-inverse for a frame. It becomes a two-sided inverse for a basis. In N-dimensional space, frames contain M > N vectors. The matrix T is M x N, and its left-inverse T+ is N x M. The equation TT* = Tis not true. For finite dimensions the theory and examples are particularly clear — our eventual applications are to infinite dimensions. Note 1 Change of Basis Suppose T is a bounded linear operator with a bounded inverse. If the sequence {¥,,} is a Riesz basis, so is the sequence {T~'v,}. When we expand Ty = J cy¥» in the70 Chapter 2. Filters Figure 2.9: Good basis, bad basis, good frame, bad frame, all in R?. original basis and multiply by T~', this gives the expansion y = 7c, (T~'y,,) in the new basis: A = B = 1, The new Riesz constants A, B come from the original A, B and the norms of T and T~. The classical transforms of mathematics have a stronger property than invertibility. T is often aunitary operator. The inverse of T becomes the conjugate transpose T*. The norms are ||| = |7~'|| = 1. If the original basis is orthonormal so is the new basis: A = B = 1. The Fourier transform and wavelet transform are unitary operators — when we use orthonormal wavelets! Biorthogonal wavelets lead to non-unitary operators T and to Riesz bases — but not orthonormal. Fortunately ||T\| and ||7~'|| are in practice surprisingly close to 1. In finite dimensions, every basis is a Riesz basis. The basis vectors for RN are the columns of an invertible N x N matrix. That matrix is T~!. It is essential to distinguish the change of coordinates (which uses T) from the new basis (the columns of T~'). The fundamental fact is T"'T =I. We use it now, multiplying columns times rows: Nv v=T'Ty = Y (column n of T-')(row n of Tv mi The nth basis vector is v, = column n of T! The nth coordinate is by, = (row n of Tyv = (ry, v). Those three lines give ¥ = )> by ¥n- Every N x N matrix is a bounded operator. When the inverse exists, it is also bounded. But infinite matrices can represent unbounded operators. The averaging filter with coefficients 3 is bounded. The inverse filter with coefficients 2, —2, 2, —2, 2, —2, ... is unbounded. The z requirement, that T and T~! are both bounded, is needed for a stable change of basis. An orthonormal basis (A = B = 1) has condition number 1. Then the product ||T| |7~'|| is the condition number of the new basis. Bases from Filter Banks The important examples for us are the bases from filter banks (discrete time) and the bases from wavelets (continuous time). Chapter 1 gave an example of both. The discrete time basis vectors had only two nonzero components: ot oe ioe.2.5 Bases and Frames 71 This is an orthogonal basis because the vectors are mutually perpendicular. It is an orthonormal basis when divided by V2. The heart of the book is the construction of other (and better) filter banks. The filters will be longer and they will overlap. Orthogonality will not be automatic and it may not be true. The bases are the impulse responses of the filters — with a double shift as above. Here is an example with four nonzero components: 1 1 0 0 3 3 0 0 3 —3 1 i eed 3 3 0 0 3 -3 0 0 1 1 Those basis vectors are not orthogonal. We are alternating lowpass filters with highpass filters, but the first and third (lowpass and shifted lowpass) are not orthogonal. Remember that these vectors are the columns of T~!. In our other language they are the impulse responses from the synthesis filters. The synthesis filters combine to reconstruct the signal, which is exactly what the basis vectors do. In this nonorthogonal case, the inverse is not the transpose. The basis is not self-orthogonal. It is biorthogonal to a different basis — which we now discuss. Biorthogonal Bases (Dual Bases) The basis {r,,} is biorthogonal to the basis {v,,} if the inner products are (ri.v;) = 6G — j). (2.47) This is the same property that governs the rows of a matrix T and the columns of T~'. It comes directly from TT~! = I. So when the columns of T~! are a basis (as above), the rows of T are the biorthogonal basis — the dual basis. We transpose T to turn its rows into columns. We transpose T~'T = Tinto T*T~* = I. Then the same idea that gave the basis {v,,} from T~! now gives the biorthogonal basis {r,} from the columns of the transpose matrix 7*: N. v=T'T*y = ) (column n of T)(row n of Tv n=l The nth dual basis vector is r, = column n of T* The nth dual coordinate is d, = (row n of T~*)v = (Pn, ¥)- When T* = T-!, the basis is self-dual and v, = r,. We have an orthonormal basis. In general the rows of T and columns of T~! produce biorthogonal bases — provided always that T and T~' are bounded. In this case we have one expansion of v coming from T~'T = I, and another expansion from (T~'T)* = I: Theorem 2.4 = /fr,, and v,, are biorthogonal bases then any v has two expansions: evra = etn vn and v= Yo duty = Yn vn (2.48) In other words © v,r= = Land also or,vt = 1. voe Chapter 2. Filters Example 2.10. The four-tap filters in the example above are biorthogonal to these four-tap filters (after we divide by 16). Check the inner products: -1 -1 0 3 3 0 0 3 —3 -1 -1 ‘| -1 Lele 3 3 0 0 5) -3 0 0 -1 1 The construction seems magical. We want more like this. Chapters 4 and 5 show how to get more —and the filters need not have equal length. These are the synthesis filters (in T~!) and the analysis filters (in the rows of T) of a perfect reconstruction filter bank. The filter bank is biorthogonal. The norms of T and T~', and therefore the condition number ||T'|| || 7~! ||, are easily computed for these bases. The double shift in T means that the transform involves a 2 x 2 matrix function of . Maximizing its norm over @ yields |||, and maximizing the norm of the 2 x 2 inverse yields ||7~'. A small note of caution. The scaling functions and wavelets will come from iterating the lowpass filter, with rescaling. Not every biorthogonal filter bank leads to biorthogonal bases. Sometimes the iteration diverges. There is a serious step from L?(Z) in discrete time to L?(R) in continuous time. The example above fails! Those particular filters with -E1 and -£3 are only good when they are not iterated too often. Wavelet Packets and the Best Basis In Chapter | only the lowpass filter was iterated. It was assumed that lower frequencies contained more important information than higher frequencies. For many signals this is not true. A wavelet packet basis allows any dyadic tree structure (Figure 2.10). At each point in the tree we have Complete tree (Walsh) Wavelet tree (Haar) Wavelet Packet tree pt —L —_ > P Figure 2.10: Each wavelet packet tree yields a basis, including Walsh and Haar. the option to send the signal through the lowpass-highpass filter bank, or not. One possibility is the logarithmic tree, with lowpass iteration only. Another possibility is the complete tree, analogous to the Short Time Fourier Transform. Wavelet packets make up the entire family of bases. Each one is associated with a particular quadtree, because it comes from splitting into two (or not splitting) at each step. The decision to split or to merge should2.5 Bases and Frames 73 be aimed at achieving minimum distortion D — subject to cost and capacity constraints on the rate R. The main point is that a library of wavelet packet bases is a practical possibility [W]. For a given signal and a given point P in the tree, we have the coefficients of the basis functions w(t) for that point. If we choose to split, that set of functions is replaced by two half-size sets of functions —created by lowpass and highpass filtering: N N Show pr —k) and Shwe ee -k. k=0 k=0 Together with their time shifts, these are the new basis functions for the points Pp and P, in the tree. At each of those new points we again have the option to split. A point is at level j in the tree if it is reached after j splittings. The basis functions at the root (level j = 0) are the shifts @(¢ — k) of the scaling function. ++ Haar Functions ae cs w(t) w(2t) w(2t— 1) Walsh Functions + then + + then — — then + — then — Figure 2.11: Haar iterates only the lowpass filter. Walsh iterates both filters. Example 2.11. When (1) is the box function, the wavelet w(t) is Haar’s up-down square wave. The filter coefficients are ho(k) = 1, 1 for lowpass and hy(k) = 1, —1 for highpass, all divided by V2. We describe three special bases for the functions that are piecewise constant on intervals of length 2~/. Within [0, 1] this is a space of dimension 2/: 1. Box basis: $(2/t — k) for0 |c,|. But the requirement of linear independence is dropped. A frame is associated with “oversampling” or “redundancy.” There are too many vectors for a basis. We could even repeat the same basis vectors several times — this produces a frame but not an interesting one. More interesting is a set of functions like {e’“"'}, which are a basis for L7[0, x] when c = 1 anda tight frame for ¢ < 1 (higher than Nyquist rate). For c = } this frame is a union of two bases {e'"’} and {e"'e'/}, The place to start is in finite dimensions. The M rows of a rectangular matrix give a frame for RY — provided the columns are independent: T is any M x N matrix with N independent columns. In general M > N and T~' does not exist. The left-inverse T* = (T*T)"' T* does exist and T+T = I.2.5 Bases and Frames 75 Linear algebra [S] says that T*T always has the same rank as T. By assumption of indepen- dent columns, this rank is N. The N x N matrix T*T with this rank is invertible. The identity (T'T)' T°T =I shows that T*T = I. This matrix T* = (T*T)"' T* is a left-inverse and a “pseudo-inverse” of T. The columns of T* and the rows of T are two frames, dual to each other. The key is T*T =I. Example 2.12. The three rows r1,r2, 3 of T constitute a frame for R?: 2 0 T=] -1 V3 has independent columns and | . : |: -1 -v3 This example is actually a tight frame, because T*T is a multiple of J. The left-inverse is (°T) | T* = T*. This matrix T* is 2 x 3. It cannot possibly be a right-inverse of T. The frame vectors are not independent but they span the space. We can recover v from the columns vt, vz, vt by using inner products (r, v) and (rz, v) and (r3, v). The recovery is based on MT =r uw v= TT = ) (column n of T*)(row n of Tv nl The nth analysis frame vector is r» = row n of T The nth coordinate is (rm, ¥) = (row n of Tv The nth synthesis frame vector is v; = column n of T*. In this example the column vector v = [1 1’ has coordinates (ry, v) = 2 and —1-4+V3 and —1— V3. To synthesize v from the three columns of T*, multiply the columns by those coordinates: 2 1p 20-10 +1 1 r= | -14+V73 -[i]- 6L0 V3 -v3 ae 1 The point of the pseudo-inverse T* (which is one of many left-inverses) is that the sum of squares of the coordinates is as small as possible. This is the key to frames: control the sum of squares of coordinates. Now we define a frame in infinite dimensions. Definition. An analysis frame is a set of vectors r, such that Ale and B > Oare the “frame bounds.” A tight frame has A = B. The frame vectors r, are not required to be independent. But they span the space! The only vector v orthogonal to every r, is the zero vector, by (2.49). The frame operator T transforms v into the sequence of numbers (r,,,v). In N dimensions these are M numbers, with M > N. In infinite dimensions we have infinitely many numbers, and the key is to recover v from these numbers. The purpose of the frame bounds A and B is to make T*T and (T*T)~' bounded operators: Note carefully that (v, T*Tv) = (Tv, Tv) = Yo \drn,)?. (2.50)76 Chapter 2. Filters Inserting into (2.49) gives Allv||? < (v, T*Tv) < B\|v||?. The operator T*T is bounded by B. Its inverse is bounded by 1/A. When we choose the tightest A and B, which we might as well do, the norms of T*T and its inverse are exactly B and 1/A. The frame ratio B/A is the condition number of T'T. The frame bounds ensure that v can be stably recovered from the components (ry, ») of Tv. The recovery operator — the synthesis operator —is the left-inverse T* = (T*T)! T*: v =T*Ty = )> (column n of T*) (ry, ¥) = Yo envy (2.51) The coordinates are c, = (rn, ¥). The synthesis frame vectors v¥ are the columns of T+. When (r,} is a basis, T* is T~! and the dual frames are dual bases. Example 2.13. The three rows of T are an analysis frame (not tight) for R?: 1 1 T=|1 0 and cae a a 0 2 Any two rows of T are a basis. The three rows are a frame. The frame bounds are A = 2 and B = 3. Those are the extreme eigenvalues of T*T —easy to find in this example. There are infinitely many left-inverses of T, and here are three: 0 1.70 0.10 Gelae es as [2 fg} om [Sb SJ aw refh 4]. 2 2 The last one is the best one! Itis the pseudoinverse (T*T)-'T*. You see + and } from inverting T'T. The columns of T* are the dual frame vectors vi and vf and >. Why is T+ best? Because it is the “smallest” left-inverse. We are reconstructing any vector v in R? from the three components c1, c2, c3 of Tv. In exact arithmetic, the only solution to Ty = ¢ is v—and every left-inverse will find it. In actual arithmetic, and in actual measurements, there are errors in c. We choose the least-squares solution to Ty = e. That comes from the normal equations T*Ty = T*c, whose solution is exactly y = Ttc. The synthesis frame reconstructs the truest v. Frames are associated with oversampling and redundancy. We meet this in irregular sam- pling when it is difficult to sample at exactly the right rate. Better to sample too often than too seldom. Oversampling is usually better than undersampling. The interpolating functions sinc(t — f,) are a basis for the band-limited space when the sampling rate is exactly right, and they are a frame when we oversample. They cannot reproduce all functions if we undersample. For regular sampling at the times f,, = nT, the perfect rate is the Nyquist rate in Section 2.2. It requires two samples per period — the exact band of frequencies is |co| < /T. In reality we often take more measurements than necessary, and reconstruct the signal by least squares. The least-squares matrix is exactly T*T! This is the matrix in the normal equations, when T has too many rows (M > N). The coefficient matrix T is M x N and not invertible. But T*T is invertible exactly when the rows of T are a frame. Instead of T™ the least-squares solution uses T* = (T°T)"'T*. Note that a pseudoinverse T* is defined for all matrices, of arbitrary rank [S]. T* is a left- inverse when the rank is N and the columns of T are independent and the rows of T are a frame. Those are equivalent statements about T in finite dimensions.2.5 Bases and Frames 7 Mallat [Mt] identifies two approaches to the reconstruction of the signal ». If T* will be used often, we may compute it explicitly. Its columns are the synthesis frame vectors v;. For limited use we do not want the inverse of T*T, only the solutions to specific equations with this coefficient matrix. Just as in linear equations Ax = b, we seldom want A~! and we more often want.x = A~'b. The iterative solution of the reconstruction problem for irregular sampling has been studied by Grochenig. This is an important application. His algorithms are much better than the simplest iterations. We mention Kadec’s + Theorem for L?[0, 1]: the exponentials {e?'%"} are a basis if |¢n — 7] stays below a number L < 4. This is nearly a regular sampling. Tn our applications, the “rows” of T are usually functions r,,(t). Then T maps functions f(t) in L? to sequences cy, = (rn, f) in €?. The matrix TT* contains the inner products of those rows with themselves. It is the “Gram matrix” whose (i, j) entry is the inner product (r;, rj). The Riesz bounds A and B come from this matrix. Its norm is B and the norm of its inverse is i. Those numbers measure the linear independence of the functions r,(t). A case of special interest is when the rows r, (/) are translates (1 — 1) of a single scaling function. Theorem 6.6 will give a formula for A and B in this case, involving ¢(w). Summary: The bounds on TT* are the Riesz constants — measuring independence of the rows of 7. The bounds on T*T are the frame bounds. When T is invertible, these ideas are the same — we have a Riesz basis. When T is not invertible the Riesz lower bound is A = 0. Question: Could a frame include the zero vector r = 0? Answer: Yes. The inner product (0, f) = 0 would not harm (or help) the frame bounds. A zero row of T has no effect at all on T*T. And it destroys TT*. Question: Do a lowpass and highpass filter jointly produce a frame? Answer: Yes, if their frequency responses have |C(w)| + |D(w)| = A > 0. Question: When would the two filters jointly produce a tight frame? Answer: When |C(w)|? + |D(@)|° = constant. This happens in an orthonormal filter bank. The frame constant is 2! Subsampling removes the redundancy and reduces that constant to 1. The tight frame becomes an orthonormal basis. Construction of Frames In finite dimensions, any set of vectors that spans R* is a frame. In infinite dimensions, the frame condition is not so simple. The good constructions start with only one function. We never want to compute with an infinite set of totally unrelated functions. The natural idea is to create an infinite family from that one function, in a systematic way. Two systems are of special importance and they lead to windows and wavelets: 1. Windows come from one window g(t) by modulation and translation: B&nn(t) = e'™ g(t — nT). (2.52) 2. Wavelets come from one wavelet w(t) by dilation and translation: w(t) = a!?w (alt — kT). (2.53)78 Chapter 2. Filters The indices m, n, j, and k extend over the set Z of all integers. We have a family gmn(t) of win- dows and a family w jx (t) of wavelets. The numbers Q and T and a are fixed. Those numbers de- cide whether the family is a frame. Dividing T by 2 gives twice as many functions — increasing the likelihood of a frame. Dividing Q by 2 works the same way; so does replacing a by \/a. The crucial parameter is QT for the window family and T log a for the wavelet family. When these parameters are small, we have more functions and more likely a frame. As T and Q and log a approach zero, we begin to lose inefficiency. The ratio B/ A eventually increases — the frame becomes excessively redundant. (Possibly B/A is a convex function of QT.) Most of this book is about a = 2 and T = 1 and special wavelets w(t), leading not only to a frame but to a basis. Section 8.4 is about 2 = 1 and T = 1 and special windows, again leading to a basis. Here we are only aiming for frames — much easier. A point about normalization. The factor a//? is included in the wavelets to keep the same norm. Without this factor the norms would go to zero as j — 00. The information to reconstruct f(t) is still available in the inner products with the frame vectors, but the scaling is poor. This emphasizes that in infinite dimensions, the frame must span the space stably. Window Frames: Examples and Theorems Example 2.1. Suppose g(1) is the unit box function on [0, 1]. For T > 1 the giny(t) are not a frame. None of the boxes g(r — nT) overlap the interval [1, 7]. Any function f(t) supported on this interval has zero inner product with all the windows. These spaced-out windows do not span L?. For T = | the translated boxes g(t — n) fit tightly. Within each box we have exponentials e'"2!_ For Q = 2x this is an orthonormal basis on the interval [0, 1]. For Q < 27 itis a frame. We see how a basis appears on the “edge” of a family of frames. As soon as & passes 2, there are not enough exponentials e’”" and the frame is lost. (For Q = 4zr we will completely miss ¢!2*".) In the next example, and often, no basis appears — we go directly from a frame for small QT to failure for larger QT. 2/2 Example 2.2. Suppose g() is the Gaussian function e as in [Gabor]. It produces a frame if QT < 2m. It does not produce a frame if QT = 27, although this was Gabor’s favorite! At the edge of the frames, these functions gnn(t) = e2™e-¢—""/2 do span the space — but not stably. The inner products (f, gin) uniquely determine f(t), but the operator T*T is not bounded below. The reconstruction formula from those inner products is not numerically stable. For QT < 2x the tables in [D, p. 87] show how B/A depends on Q and T. We now state four general results about window frames, without proof. Three are positive, one is negative. The negative one is famous and comes first: A. Window frames are impossible with QT > 22. Smooth and decaying window frames are impossible with QT = 2x. “Smooth and decaying” in this Balian-Low Theorem [BeHaWa] means that g’(r) and tg(t) are in L?(R). The window bases in Section 8.4 escape this restriction by a simple change in their construction (cosines replace e2*'"'), The impossibility for QT > 27 first came from Rieffel.2.5 Bases and Frames 79 B. The dual to a window frame comes from a dual window 3(t). This makes it worthwhile to compute the dual window. One function g(r) gives the whole dual frame. In many other cases it is too much work to construct (T*T)~'T*, whose columns contain the dual frame. Instead we solve a linear system for the synthesis coefficients in 7 Carn. C. Suppose Y> \g(t — nT)[° stays between positive constants for all t. Then there is a positive threshold Qo such that 2 < Qo yields a frame [D, p. 82]. This assumes that g(t) has compact support or |t|“|g(¢)| > oo for some @ > 1. * D. The frame bounds satisfy A <2z|g\|/QT < B. Wavelet Frames: Examples and Theorems Example 2.1. The Mexican hat w(t) = (1—1?)e~"’/? is the second derivative of the Gaussian. This wavelet is often used in analyzing vision. It has very rapid decay of both w and @. For a = 2 and small T, the tables in [D, p. 77] show that B/A is very near 1. As T increases we suddenly lose the frame. Example 2.2. The Morlet wavelet is a modulated Gaussian (complex for real signals): 2 2 twat g-F/2 with ae e with @ aD This shift in frequency almost gives { w() dt = 0 as required for wavelets. The actual value is below 10~°, and negligible. The phase plot of the wavelet coefficients (f, wj,) becomes very useful [Mt] in locating singularities of f(r). w(t) = = 5.336. Some applications of these two examples use N different wavelets, called voices. This usu- ally produces a tighter frame; B/A comes near 1. A good way to spread the N voices over an octave is by fractional dilation of a single wavelet: w(t) =w(2%t), €=0,...,.N—1. (2.54) The negative statement A and positive B in the window rules are reversed for wavelets. A’. There are wavelet frames (even orthonormal bases) for large values of T log a. The restric- tion QT < 2x on smooth window frames does not apply to wavelets. B’.. The dual to a wavelet frame does not always come from one dual wavelet. C. If |B (aie)/? stays between positive constants for all w, and w(t) is smooth, there is a positive threshold such that T < To yields a frame (D, p. 69]. D’.. The frame bounds A and B are related to the wavelet constant C by pw As lol oo oo. Note. For real wavelets w(t), when |@|* is an even function, integrate only over a > 0 and compensate by taking only half of the constant C: 0 dl c= f ly? 2 = te, 7 @ The Wigner-Ville Transform The time-frequency analysis of a signal goes much further than windows and wavelets. By re- stricting to the choice of one function g(t) or w(t), we create good algorithms — which is our principal purpose. By allowing more general transforms, we can hope to get precise information about the “instantaneous frequency” and “instantaneous spectrum” of a signal. The windows and wavelets do a little smearing — not too much, and the time-frequency trace of the signal can be followed. But Wigner and Ville and many others wanted a sharp trace. We shall devote just a very short space to this basic topic — the correlation of a signal with itself. Everywhere else we are correlating the signal with chosen windows or wavelets. As those84 Chapter 2. Filters move and rescale, we identify our signal from the correlations. The self-correlation gives an energy density that should automatically pick out the position and frequency and scale of the signal. We define the Wigner- Ville transform of a finite energy function f(t): oa t t it W(t.e) = [. F(t fi S)e(r S 5)e dt. (2.67) Notice especially that W is quadratic in f. We expect W(t, @) to be an “energy density” —like the square of the window and wavelet transforms, but better. This density has several desirable properties: x | Merdo= If@P and fweerar= |Flo. (2.68) W(t — T, @ — Q) is the transform of e!® f(¢ — T). W (£, aw) is the transform of + f (£). W(t, w) determines f(t) up to a constant multiplier |c| = 1. The transform goes from one variable to two variables. As with wavelets and windows, most functions of two variables are nor transforms of any f(r). The transform of f flips the variables to reach W(w, —1). And the Gaussian plays a special role. We get W = e from the unit Gaussian and we get modulations of W = e PEP o=PIO From all quadratic “chirps.” What good properties does this transform not have? First, it is not always positive. Second, the sum of two Gaussians (modulated and shifted) has a transform with four terms. Two terms are in the expected positions (T;, 21) and (7, Q2), as desired from the Gaussians. Two other terms are in the wrong positions (T;, 2) and (T>, 21). These ghosts are highly oscillatory. “Analytic signals” have no negative frequencies: f(w) = 0 for @ < 0. Restricted to these signals, the transform is a success. Then W(t, w) earns the name “instantaneous spectrum” of f(t). And the instantaneous frequency — the derivative of the phase @(t) — equals the average + [ W(t, @) do for a large class of signals. Summary: The integral transforms, by windows and wavelets, make minimal demands on the functions g(r) and w(t). We can analyze and synthesize (transform and inverse transform) with extremely general functions. Reality sets in when the transform becomes discrete — the shifts are multiples of 7, the modulations are multiples of @, the scales are powers of a. The elegant books by Meyer give an overview of this whole picture. The rest of this text deals with the discrete transform. We will have very strict conditions on the wavelets! The scales are powers of a fixed integer a = M, most often a = M = 2. Extra conditions are imposed on the wavelet. There are absolute requirements and optional properties. The requirements make it work, the extra conditions make it work well. The goal is to achieve these properties with a fast algorithm: 1. Two-scale or M-scale equation (with special coefficients) 2. Smoothness of the wavelet (optional but desirable) 3. Symmetry or antisymmetry (optional but very desirable) 4. Vanishing moments (one required, more desirable).2.6 Time, Frequency, and Scale 85 Atomic Decompositions Whether they are wavelets or sinusoids or cosine packets or Gabor functions, we are approximat- ing f(t) by “atoms”. A collection of atoms is a “dictionary”. We need a reasonable algorithm to choose the nearly best M atoms for a given f(t), out of a dictionary of P atoms. A single basis may be too inflexible, and the algorithm must adapt to the signal. Major effort is going into algorithms nor based on one orthonormal basis. We will only mention three ideas and their developers. The first is matching pursuit (S. Mallat and Z. Zhang). The M atoms are chosen one at a time. The choice at step k+1 is the atom that comes closest to the current difference f,(t) = f(t) — c1@i(t) — +++ — ce@(t). In practice matching pursuit takes (normalized) inner products (f,(t), 6(t)) and chooses a large one —an atom @.1(t) that is highly correlated with f(t). This is a greedy and sub-optimal selection process — greedy because each choice is ignorant of the later choices, sub-optimal because we don’t insist on maximizing (f,(t), @(t)). The complexity is O(N In N) at each iteration, for a signal of length N. The pursuit ends at k = M. We can then compute the best combination of the M choices (this is back-projection). Experimentally the error approaches white noise as Moo. A second adaptive method is the best basis algorithm (R. Coifman and V. Wickerhauser). This begins with a dictionary of bases, often orthonormal. The algorithm chooses the best basis to represent f(t). For wavelet packets from a family of binary trees, the method is particularly well adapted. Available software includes the Wavelet Packet Laboratory for Windows (AK Pe- ters, Wellesley MA 02181-5910). A modification that is optimal in the rate-distortion sense, for compression, is described in [VK, p. 426]. With orthonormal bases at every step, rates and mean-square distortions of the two branches are additive. This allows a fast algorithm to choose the packet that is best adapted to the partic- ular signal. The reader understands that an actual implementation leads to many options. The mean-square €2 norm may be replaced by other norms (losing additivity at each split but gaining in perceptual quality). A third method is basis pursuit (D. Donoho). The dictionary is still overcomplete. The synthesis f(t) = Yci¢;(t) (modelled by f = ec) is underdetermined. Frame theory chooses ¢ 10 have a small [? norm (sum of c? is minimized, leading to linear equations and generalized inverses). Basis pursuit chooses ¢ to have a small I' norm (sum of |c;| is minimized, leading to nonlinear equations and linear programming). This minimizer is generally much sparser — fewer c; are nonzero. The /! minimizer is also more expensive to compute. In place of the simplex method, inte- rior point and log barrier algorithms associated with Karmarkar solve a sequence of weighted F problems. This can give reasonable success for large dictionaries (P = 10' and N = 10°). The method can distinguish two nearby bumps in f(t), where matching pursuit will select one centered bump that has high correlation. Linear programming pursues the best basis — which de- pends on the signal f(t). Donoho also studies “empirical atomic decomposition” for P >> N. This algorithm strongly controls the number M of atoms ¢ (ft) in the approximation, by mi mizing || f (0) — Xcidi(0)||2 + AM. The key is in the selection of 4 = o /Zlog P. Software is on the web at http: //playfair.stanford. edu. The multiplier . becomes a ,/21og N for de-noising with an orthonormal basis. The noise is assumed Gaussian with standard deviation 0. The recommended solution is soft thresholding86 Chapter 2. Filters of the inner products y; = (;, ) with threshold 2. Each scalar y is shifted toward zero but not beyond: Thresholding sop: =(y —2)4 for y>O and (y+A)— for y <0. Donoho notes that cso; minimizes }(y — ¢)? + Ale]. That |c| is the /! norm again. Finally we emphasize that the construction of wavelets has not ended. While writing the book we have learned of directional and translation-invariant and steerable wavelets. And the world of atoms is by no means restricted to wavelets! Problem Set 2.6 1. Find the Wigner-Ville transform W(t, @) when f(t) is the Dirac function 4(t). 2. Show how W(t, @) can also be expressed by an integral of WO, 8) = x f iG - S)ale + Z)er de.Chapter 3 Downsampling and Upsampling 3.1 The Matrices for Downsampling and Upsampling The previous chapter was about individual filters. The next chapter combines two or more filters into a filter bank. The idea is to separate the incoming signal into frequency bands. Often we will use a bank of two filters to explain a point clearly. A lowpass filter C and a highpass filter D will split the signal into two parts. Those parts can be compressed and coded separately (and efficiently). Then they can be transmitted and the signal can be recovered. When the synthesis bank recovers the input signal exactly (apart from a time delay), it is a perfect reconstruction filter bank. These PR banks interest us most. One problem to overcome. We don’t want two full length signals in place of one, if we can help it. It is not desirable to double the volume of data. The information has not doubled; the outputs Cx and Dx from the two filters must be redundant. The solution to this problem is beautiful. We keep only half of the outputs Cx and Dx. More precisely, we downsample those vectors by removing every other component. This operation is called decimation. Downsampling is represented by the symbol (| 2) (pronounced “down two”): y-) y(—2) y(0) —| yO) G2) ya |=} ye y(2) yay Make no mistake. This operation is not invertible. Most vectors y cannot be recovered from ({ 2)y. The odd-numbered components are lost. Normally the even-numbered components of both y = Cx and y = Dx will be needed to recover all components of x. We note that recovery of x from (|, 2)x is possible if the transform X () is zero over a half- band of frequencies. Such a signal is “band-limited”. It may be limited to the upper half-band or the lower half-band: X(w) = 0 for 0 < |o| < ; or X(w) = 0 for 5 <|ol <2. It is amazing that for band-limited signals we can recover the odd-numbered components from the even-numbered components. This is achieved by the Shannon Sampling Theorem in Sec- tion 2.3.88 Chapter 3 Downsampling and Upsampling Example 3.1. Downsampling a vector of alternating signs produces a vector of constant sign: W2¢6..,1,-11,-1 The —1’s in the odd-numbered positions have disappeared. You will quickly think of other inputs that give the same output. One in particular is the input vector (..., 1, 1, 1, 1, ...) of all ones. Downsampling produces the same vector of ones. It is an eigenvector of the linear operator (| 2), with eigenvalue A = 1. Another eigenvector is 6 = .--,0,0, 1,0, 0,...). This also has 4 = 1, and (| 2)6 = 6. It is valuable to compare the alternating and constant inputs. Vector (..., 1, —1, 1, —1,...) is at the frequency @ = zr. That is the highest possible frequency in discrete time. No vector can oscillate faster. Its components are x(n) = e!*", which is (—1)". By contrast, the input of all ones is at the lowest frequency w = 0. Its components are x() = ei" = 1. This is an example of aliasing. Two different frequencies, after downsampling, cannot be distinguished. The “alias” of the high frequency @ = is the low frequency @ = 0. Certainly we cannot recover the input (which input would it be?) from the common output ({2x =(..., 1, 1, 1,...). If we know that all frequencies in the input satisfy || < 3, then we can “invert” the downsampling operation. The only possible input would be the constant vector, the DC input with w = 0. Example 3.2. When the impulse response C6 = the odd-numbered responses are lost: ¢(0), e(1), €(2), .. .) is downsampled, (12)€6 = (...,€(0), e(2), e(4),...). This is the impulse response of a “decimation filter” — filtering followed by downsampling. The impulse 6 came at time zero, and downsampling leaves the even coefficients. The response is entirely different if the impulse is delayed — we compute that now. Downsampling is not time- invariant. The delayed impulse is $6 = (...,0, 1, 0, ...). The response from the time-invariant filter C is similarly delayed: CS6 = (...,0,¢(0), e(1),.. GB.1) Because of the delay, downsampling now destroys the even-numbered components (0), ¢(2). ‘We pick up the odd components: (L2)CS6 = 0, e(1), €(3), . First conclusion: Complete information may still be there if we downsample two vectors. Here the vectors were C6 and its shift SC6 = CS6. Those outputs are not as “separated” as we want —the two pieces are probably far from orthogonal. Downsampling Cé and CS6 may be a waste of time, but we do see that the whole vector (c(0), e(1), ¢(2), e(3), -..) is recoverable from the two samples. Upsampling will be the way to recover it. A better filter bank will use two different filters C and D. Part of the design problem is to ensure that ({ 2) Cx and (| 2) Dx have all the information to recover x. Even better if those vectors are orthogonal, which is not true for this easy choice D = shift of C. Definition 3.1. The nth component of v = (.2)x is the (2n)th component of x: v(n) = x(n). (3.2)3.1 The Matrices for Downsampling and Upsampling 89 We will now identify the “downsampling matrix”. /t is the identity matrix with odd-numbered rows removed. You could say that the identity matrix has been downsampled! The matrix form of downsampling is x(—2) 1 x(-1l) x(-2) (Oj Osa x(0) = x(0) G.3) One x(1) x(2) x(2) We are defining the linear operator ({ 2). Notice that the rows of the downsampling matrix are orthonormal. They are unit vectors and their dot products are all zero. The matrix multiplication (42)(L 2)" contains all these dot products, so it must be the identity matrix. The matrix product (12)" (12) is different, as we see below. The operator (|.2)" is important. The transpose of downsampling is upsampling: (2)7 = (42). GA) Upsampling places zeros into the odd-numbered components: v(-1) : 0 7 i 1) |e 0) Definition oF) 2) (12) oe al 0 v(1) vd) : 0 We need a two-part formula to describe those components of u = (2): v(k) 0 ifm=2k+1. u(n) = 3.5) To do this with a matrix, insert zero rows between the rows of J. You could say that the identity matrix has been upsampled. The matrix form of upsampling is 0 : v(—1) 0 0 1 0 rl) (0) G.6) 0 v(0) 0 v(1) v(1) The matrix in (3.6) is the transpose of the matrix in (3.3). If we multiply them we do get the90 Chapter 3 Downsampling and Upsampling identity matrix, verifying (| 2)(t2) =I: 1 1 ; 1 O20 1 Aa 1 ; = ; (7) 0 1 0 If we upsample and then downsample, we put in zeros and take them out. The original x is re- covered. This is not the usual order of the steps! Normally we downsample and then upsample. This product is not the identity matrix. The output ( 2)(J 2)x is different from x. Watch the steps: x(—2) x(—2) a Down: (\2e=]| *7O Then up: (t22x=| 0 |. (38) x(2) oa 0 In this usual order, (| 2) removes the odd-numbered components and (t 2) puts in zeros. The product (2)({ 2) replaces the lost components by zeros: 1 0 4242) = 0 (3.9) Thus (| 2) and (2) are one-sided inverses but not two-sided inverses. This cannot happen for square matrices of finite size. For n by n matrices, AB = I means that BA = I. For rectangular matrices and infinite matrices, a one-sided inverse is possible. Downsampling is a very important example. The matrices for (J 2) and (7 2) came from downsampling and upsampling the columns of I. Here is another way to describe those same matrices: e The ({ 2) matrix has the rows of I with a double shift. ¢ The (42) matrix has the columns of J with a double shift. Problem Set 3.1 1. Downsampling the alternating vector x = (...,1,—1,1,—1, 1,...) produces the vector (2x = 6.2.1. 1, 1...). Ifyou delay x to Sx(n) = x(n — 1), what is (1 2)Sx? 2. Describe the operator (4 2)(1 2)(t2)(4 2). s Describe the operators ({ 3) and (f 3) and (¢3)(1 3). What are the components of (f 3)({ 2)x and (1 2)( 3)x? >3.2 Subsampling in the Frequency Domain 91 3.2 Subsampling in the Frequency Domain For a pure exponential signal x(k) = e*®, the effect of downsampling is particularly clear. The kth component of vy = (J 2)x is v(k) = e*°. This is a pure exponential with frequency 2. Frequencies are doubled by downsampling. It looks as if V(2w) = X(w), in other words V(w) = X ($). That is wrong. Suppose x’ is another pure exponential, but with frequency @ +7. Then x"(k) = e+” is downsampled to v'(k) = e+, The factor e!?* is always 1. This output is the same e?*° as before! Adding zr to the frequency only reversed the signs of the odd-numbered components of x’, Those components disappear anyway in downsampling, and v'(k) = v(k). The frequency 2w appears by doubling «, and it also appears by doubling @ +. In other words, enters the downsampled vector v not only by doubling $ but also by dou- bling $ +2. There are two sources, not just one, contributing to V(@): Theorem 3.1 The transform of v = (| 2)x is Vw) = $[X (¥) +X ($+2)]. Proof. Downsampling in the time domain is very direct: v = (| 2)x means v(k) = x(2k). (3.10) The Fourier transform of v is Vo) = Dive = Vox 2ke™. GAL It is natural to set = 2k, but do it carefully. The reverse direction k = 5 is not safe. You might think that our sum is) x(n)e~i"”/?, and the frequency variable w is just halved. But the sum is only over even n = 2k. So we start again. Let wu be the vector x with its odd-numbered components set to zero: wn = | Bel ay | sothatu =...) 0,42), 0,...). (3.12) Certainly (| 2)u equals (| 2)x. The transform of w has two terms. The second term includes (—1)" or e~"" so that addition knocks out odd n: u(n) = > x(njene = I So xien +4 xe nor, (3.13) even am am In frequency, this says that Uo) = $1X@)+ X(t). (3.14) Now V (w) comes safely from U (w) by halving the frequency, because u(n) only involves even n: Vw) =u (). (3.15) Equations (3.15)-(3.16) mean that the downsampled signal v = (J 2)x has transform V(o) = 3[X ($) +X ($+z)]. (3.16)92 Chapter 3 Downsampling and Upsampling This is downsampling in the frequency domain. It is not time-invariant, because X («) is not just multiplied by C(w). The rate changes and a new term appears. Figures 3.1 and 3.2 and 3.3 show extreme aliasing and no aliasing and typical aliasing. You can see the two terms in Figure 3.1, where X(@) is a sawtooth function. The average in U(@) is a constant. That second term is responsible for aliasing: X ($) overlaps X ($ + 7) in V(q). Their sum happens to be constant, so this special input x gives v = (| 2)x = 16. Allits even-numbered samples are zero, except x(0) = 4. X(@) has period 27 x(@) and X( San ) have period 4 Figure 3.1: Downsampling stretches X (w) to X (#) and it also creates the alias X ($ +77). Be- cause the alias overlaps, we cannot recover X («) from U (w) or from the downsampled Vw). In Figure 3.2, downsampling does not lead to aliasing. The input X(«) is properly band- limited. The stretched transform X ($) does not overlap the aliasing term X ($ +1). The out- puts V() and U() from downsampling and upsampling still contain this aliasing part! But a non-overlapping alias can be filtered away. It is striking that the adjective “halfband” is used in both of these extreme cases, but in dif- ferent ways: x in Figure 3.1 is a halfband impulse response (x(2n) = 0 for n # 0). x in Figure 3.2 is a halfband signal (X (w) = 0 for $ < |o| < 2). Figure 3.3 is somewhere between Figure 3.1 and Figure 3.2. Aliasing is expected. The orig- inal x cannot be recovered from (J 2)x. But we don’t expect the unusual result in Figure 3.1, where downsampling produced a pure impulse.3.2 Subsampling in the Frequency Domain 93 X(q) is bandlimited to || <2 X(@) and X( P+7) have period 4x ex may . se NS . . 1 Bey Lu é 1 “non o -m om gl wy =U) ® ® U(@)=0.5 [X@) +X( on] V(@}=UP)H0.5 [xg d+ x( Pen J 0.5X(0) 0.5X(w+n) Aone UA . . “ . Figure 3.2: Downsampling this bandlimited signal stretches its transform to [—zr, 7], no aliasing. Vow) L X(w) Nee Aliasing Figure 3.3: Typical aliasing from ({ 2). The stretched X ($ ) overlaps X(% +2). Upsampling in the Frequency Domain In the time domain, upsampling needs separate formulas for even n = 2k and odd n = 2k + 1: u(2k) = v(k) uk +1) =0. CY u = (t2)v means | In the frequency domain this becomes simple. Where downsampling v(k) = x(2k) goes from one term in the time domain to two terms in frequency, upsampling goes from two terms to one. The zeros in u(2k + 1) = O contribute nothing to U(w). We now establish the upsampling formula in the @ domain: Theorem 3.2. The transform of u = (4 2)v is U(o) = V2e). (3.18)94 Chapter 3 Downsampling and Upsampling Proof. Only the even indices n = 2k enter because u(2k + 1) = 0: Uw) = Yulaye i? = Puke? = Yv®e 7 = V2w). (3.19) Where V (cw) has period 277, this new function V (2«) has period z.. The original graph is com- pressed into |o| < 3. An image of that compressed graph appears next to it. Upsampling creates an imaging effect! Figure 3.4 shows how two compressed copies of the graph of V(w) appear in the spectrum of U(@). X(w) U@)=X(20) VC. oa ey aa 2m Figure 3.4: The upsampled spectrum U(w) = V(2«) has compressed images of V (c). This imaging is the opposite of aliasing. In aliasing, two input frequencies @ and w + 3 give the same output. In imaging, one input frequency « is responsible for two outputs —at frequency $ and also at ¥ + zr. Upsampling produces imaging where downsampling produces aliasing. ot Upsampling after Downsampling The analysis bank includes downsampling. Then the synthesis bank includes upsampling. It is very common to see those two operations together: v=({2)x andthen w=(t2)y give w=(T2)(12x. Theorem 3.3 The transform of u = (+2) 2)x is U(@) = 1[X(@)+X(@+n)]. (3.20) This was equation (3.14). The aliasing X (w + 27) comes from (—1)"'x(n). When n is odd, this cancels x(n). The average of x(n) and (—1)"x(n) is _ { x(n), neven wn ={ 0, nodd. so In the frequency domain the proof has two steps: : ar A 2 down: V(w) = s[x(S)+x(G +n) thenup: U(w)= Vw) =1{X@)+X(+z)]. Conclusion: (+ 2)({ 2) produces aliasing and also imaging. Figure 3.5 shows them both; the image of the alias overlaps the original. We can’t determine the original X (w) from U().3.2 Subsampling in the Frequency Domain 95 U X(w) Or = 0 7 Ino 2m Aliasing Figure 3.5: The spectrum is stretched by downsampling and compressed by upsampling. The alias from downsampling generally overlaps the image from upsampling. Filter Bank without Filters The idea of a filter bank without filters sounds absurd. But this is a very useful example. The filters come from the identity matrix and don’t do anything (except possibly a delay). All the activity comes from downsampling and upsampling. It is good to see how the original signal x is reconstructed. Here is the block form of the simple filter bank, with no filters: (ny bee }2 v 12 u ; Ly a {2 $2 x(n-1) In words, the upper channel downsamples and upsamples. It produces the vector u described above, with all odd components of x replaced by zeros. Then this w is delayed before output. The lower channel delays x at the start. Then downsampling followed by upsampling re- moves the even-numbered components x(2k). The combined output is just a delay of the input: delay of (t2)(J 2)x 0 x(-1) + = | x | + ) = delay ofx. (12)( 2)(delay of x) 0 x(1) (2) 0 A delay multiplies the transform by e~". The delay in a filter bank is indicated by z~'. Instead of the Fourier transform we more often use the z-transform. A delay multiplies by z~!, and a two-step delay multiplies by z~?. An advance (non-causal!) multiplies by z. We will next express (| 2) and (2) in the z-domain, And we take this opportunity to do the same for (| M) and (tM). Downsampling and upsampling involve every Mth component of the signals in an M-channel filter bank. Problem Set 3.2 1. Draw V(w) and U(w) when X (@) is a constant function. What vector x has this transform? What is the result of downsampling that x? 2. What are the usual periods of the functions X (w), X ($), and V(w)?96 Chapter 3 Downsampling and Upsampling 3. For downsampling by 3 instead of 2, write down the equations corresponding to (3.11) through (3.16). In equation (3.10), » = (| 3)x means v(k) = x(3k). 4, What is the original input x in Figure 3.1, whose transform is the sawtooth function? To invert transforms use x(n) = [ X(w)e™® do. 5. By inverting X («) in Figure 3.2 find the original band-limited signal x(n). What is »(n) = ({ 2)x(n)? Compare with the input signal in Figure 3.1. 6. (Review) Verify that the transform of (J 2)(t2)v is still V(w). Then (| 2)(¢2) =I. 7. Draw the block form of an M-channel filter bank without filters. Imitating the M = 2 case above, the output equals the input after M — I delays. 3.3 Sampling Operations in the z-Domain This section includes a conversion from the frequency variable « to the complex variable z. The connection between them is always e’” = e/® = z. Real frequencies @ correspond to points on the unit circle |z| = 1. The transform is extended beyond this special circle of complex numbers, to include all z in the complex plane. In the process we overcome one inconvenience. The frequencies @ and +27 and w-++4z are impossible to distinguish. In the z-domain we don’t attempt to distinguish them. The complex numbers z = e™ and z = e!(#?) and z = e+4” are absolutely identical. This remains just as true when the “frequency” « is not real. It depends only on the fact that e?7! = 1. We repeat the definition of the z-transform of x: The signal x(n) is transformed to X(z) = }ox(n)z". What we earlier called X(w) is now X (e/). You will know immediately from the letter z that X(z) refers to the z-transform rather than the Fourier transform. At present, these infinite series are “formal power series”. We are not worrying about their convergence. In reality they probably converge for some z and diverge for other z. When posi- tive and negative powers are both included, the region of convergence is essentially an annulus. This is the region rinner < |Z| < Touter between two circles. Convergence may or may not occur on the circles |z| = rinner and |z| = Touter. There is also the extreme but quite likely possibility that rinner = 0 OF Fouer = OO OF Finner = Touter- What is the effect on our down and up formulas? The vectors are » = (| 2)x and u = (f2)v. In frequency, their transforms are Vio) =4 [x ($) + x¢ +n) and U(w) = VQ). We now derive and explain these rules for converting to the z-transform: 1. Halved frequency $ leads in the z-domain to z'/?, 2. Doubled frequency 2 leads in the z-domain to z?. 3. Shifted frequency by 7 leads in the z-domain to —z.3.3 Sampling Operations in the z-Domain 97 These rules follow directly from z = e’”. Immediately e'”/? is ¢'/ and e'® is 2? and e!*”) is —z. The rules produce the correct down and up formulas in the z-domain: Theorem 3.4 — The z-transforms of v = ({2)x and u = (+ 2)v are VQ) $[X (2?) + X(-2'?)] and Ue) = V(:’). 3.22) The transform of u = (t2)(L2)x is U(z) = § (X(@) + X(-2)). (3.23) The extension from two channels to M channels is straightforward and important. In the time domain, v = (| M)x and u = (+ M)v have components v(t), M divides k v(n) =x(Mn) and un =|{ a a tan Then (t M)({ M)x leaves every Mth component unchanged (the components for which M di- vides k), The other components become zeros. In the frequency domain there are M — 1 aliases in downsampling and M — 1 images from upsampling (it is very instructive to graph V(w) and U(@)): down: Vw) = i [*() +x(*) + x(t my) up: U(@) = V(Ma). (3.24) Eliminating V leaves U(w) = 7 [X(@) + X(w + 2m) +--+] for the upsampled w = (t.M)(J M)x. Those aliases overlap the original (not if it is properly bandlimited). To transfer into the z-domain, extend the three rules from M = 2 to any M: 1. Dividing w by M leads to z!/". 2. Multiplying w by M leads to 2”. 3. Shifting w by 27/M leads to ze?*'/™, From this correspondence we can express (| M) and ( M) in the z-domain: Theorem 3.5. The z-transforms of v = (| M)x and u = (t M)v are 7) = 1 S 1/M 20ik/M _ _M YO= Gy pee lequniae) rand U/(2) -aVa(zen) is (3.25) The transform of u = (+ M)({ M)x, down and up, is by eliminating V: Uj@)= a [X@ +X (ze) 4 +X (ze DIMY) | (3.26) The decimator (downsampling operator) by itself creates aliases. The expander (upsampling operator) by itself creates images. To prevent the aliases and to remove the images, we filter the signal. A filter comes before the decimator, to band-limit the input. The aliasing terms become98 Chapter 3 Downsampling and Upsampling zero. A filter comes after the expander, to wipe out the images. This filter removes (or nearly removes) the frequencies beyond || = %. The block forms are clear: Decimation |_y v U_] Interpolation |__, Filter iM mM Filter ‘The decimation filter suppresses aliasing. The interpolation filter suppresses imaging. It is easiest to see the decimation circuit in the frequency domain. The band-limiting filter cuts off X(@) outside ], to give ¥(w). Downsampling stretches that band to the full interval [—z, x]. But no aliases appear, as shown in Figure 3.6. X(w) Yw) 4 ° a eS 4 wiht ° wot b a € Figure 3.6: Spectra of X (w). ¥(w), and V(w) with M = 3. From V(w) we can recover ¥(w). Unless the original signal was band-limited, we cannot recover X (w). If it was, we can. The interpolation circuit would do the recovery! It is important to see the interpolation circuit in the time domain. The upsampling step inserts zeros into the signal. The interpolation filter F changes those zero values to smoother values w(n). Remember that a lowpass filter reduces oscillations to give a smoother output. Here M = 2 and the interpolation circuit produces w from v: vin) u(n) = (42) vin) w(n) =F u(n) tte, tite. Milt. a n oo. n 01234 n Fractional Sampling Rates To understand fractional decimation, study the product (2)({ 3) and also (J 3)(4 2). What are the outputs? In some way these should produce an output at a fractional rate, which is 3 of the original rate. The transform X (w) should be stretched by a factor 5. Let us see. The direct approach is in the time domain, When up follows down, we reach x(0), 0, x(3), 0: x(0) +3 x(0) 42 x(0) x(1) x(3) 0 x(2) : x(3) x(3) 03.3 Sampling Operations in the z-Domain 99 When down follows up, the remarkable thing is that the result is the same: x(0) 42 x(0) 43 x(0) x(1) 0 0 x(2) x(1) (3) x(3) 0 0 i. x(2) : 0 x(3) Thus (2) commutes with (3). In either order, every third component appears in every second position. Remember that (2) does not commute with ({ 2). A natural question arises. When does (+ L) commute with (| M)? The output will come at + times the original rate. It is best to reduce this fraction to lowest terms. The numbers L and M should be relatively prime, meaning that they have no common factors. The fraction 24 equal to + but it is a mistake to use (t 2) and ({ 2M). Those operators do not commute and they are inefficient. The natural choice is the good choice. is Theorem 3.6 — The operators (| L) and (‘} M) commute if and only if L and M are relatively prime. Their greatest common divisor is (L, M) = 1. In that case (+ L)({ M)x = (J M)(t L)x has components x(Mk) if 0 k is an integer u(n) = (3.27) if | is not an integer. This formula is always correct for the order (¢ L)(| M)x. With L = 2 and M = 3 it yields u(1) = Oand u(2) = x(3) as in the example above. With L = 2 and M = 2 it yields u(1) = 0 and u(2) = x(2). This is the familiar output from (7 2)({ 2)x. The formula (3.27) can be wrong if down follows up. In that order, the test is whether * an integer. For M = L = 2 this is true for every n. The output from (| 2)(f 2) is the same as the input, and (| 2)(}2) =. But for M = 3 and L = 2, the fraction % is only an integer when nis even. In this 3 case, up-down agrees with down-up. When M and L are relatively prime, the output in either order has u(n) = 0 unless L divides n. If % is not an integer, then “ is not an integer. Why do we alter the rate by a fraction? The rate may need to change from 60 Hertz to 50 Hertz. The choices L = 5 and M = 6 will do it. (Not 50 and 60; they are not relatively prime.) We will get five samples instead of six samples, and 50 samples per second instead of 60 per second. There is no aliasing when the input signal is band-limited to || < 2. Then, stretching by $ will create no problems: Mn is {5 lowpass 4] | 6100 Chapter 3 Downsampling and Upsampling Problem Set 3.3 1. Find U(@) from X(@) if u = (t L)(J M)x. Use (3.25) for (| M). Similarly find V(@) if v = ({M)(f L)x. Show that U(w) = V(@) if (and only if) L and M are relatively prime. 2. Verify formula (3.27) directly. What happens to the odd-numbered components of x when we compute (t2)({2)x? What happens to the odd part of X (z) in equation (3.24)? 3. Ifa band-limited signal v goes through an expander, how is (2)v related to v? 4, Fora band-limited signal x, draw in the frequency domain the steps of decimation and recovery by interpolation. Graph X(w), ¥(w), V(w), U(@), and W(w) 5. Draw H(w) for an ideal decimator that prevents aliasing by ({ 6). Draw an ideal F(w) that removes images created by (15). 6. What lowpass filter placed between (15) and (| 6) avoids images and also aliasing? 7. In smoothing w(n) to get the final output w = F u, which filters F will interpolate and not change the even samples: w(2k) = u(2k)? 3.4 Filters Interchanged with Samplers Our notation for the filter coefficients is #(n). When the input x is the impulse 6, the responses are h(0), ...,4(N). The vector h is the impulse response (in the time domain). The response in the z-domain (we are now using this form systematically and often) is H(z) = h(n) As an operator, the filter is a combination of delays (and possibly advances). Note especially that a delay corresponds to z~'. The filter is linear and time-invariant. Expressed as an infinite matrix, H has the number h(n) along its nth diagonal (counting down from the main diagonal). The i, j entry is h(i — j). First Noble Identity In the standard form, downsampling follows filtering. We apply H and then (| 2) or (| M). Taken literally, this would be highly inefficient. We are apparently computing all components of Hx, and then throwing away many of them. No sensible engineer would do that. The polyphase form of H(z) will tell us how to reorganize the calculation and do the sampling first. We cannot just reverse the order of (| M) and H (of course!). Here we note a special and important case, when the sampling can be done first. The filter has M — | zero coefficients be- tween each pair of nonzeros. The nonzero coefficients h(0), h(M), h(2M), ... will be renamed g(0), g(1), g(2),.... Then H(z) is the same as G(z™). For filters of this special type we can move the decimator outside the filter: x—|{M G(z) a Ge") 1M }—v The new order starting with (| M) cuts the sampling rate immediately. The old order applies H to the full vector x. Here is an example with M = 2 and h = (1,0, 2): (2x GUL 2)x Ax (1 2)Hx = G(2)x x(0) + 2x(—2) x(0) x(0) + 2x(—2) x(1) + 2x(-1) x(0) + 2x(—2) x(2) x(2) + 2x(0) or x(2) + 2x(0) (2) + 2x(0) : : x(3) + 2x(1) :3.4 Filters Interchanged with Samplers 101 In the z-domain, this example yields the even powers in (1 + 2z~*)X(z). The odd powers in X(z) don’t matter! So the quick way is to pick out the even powers by downsampling first. This “Noble Identity” represents a simple but very useful fact of algebra: First Noble Identity: G(z)(¥ M) = (J MG"). (3.28) Proof Downsampling x(n) leaves x(Mn). Then the filter G produces v(n) = Yawx(Man —k). (3.29) The other way (the slow way), H(z) = G(z") uses every Mth sample of the input. The filter gives )° g(k)x(n — Mk). Then the sampling operator replaces n by Mn. We reach the same v(n). You may find it useful to follow an impulse x = 6 through both systems, to see that the impulse response is the same. On the left we get the vector v = g of filter coefficients. On the right we keep all the zeros. Then downsampling ends with the same g: 6 Gee") +8(0),0,...,0,8(1),-..) | 4M g(0).g().. We used G instead of the basic symbol H to emphasize that the Noble Identities cannot be directly applied to a typical filter. The identity applies only when H(z) has the form G(z™). Only filters that have special impulse responses, with M — 1 zeros between the nonzeros, fit this form. The idea of the polyphase matrix is to decompose a typical H into a set of M filters with these special responses. Then the Noble Identities apply to each polyphase component of H, even if they don’t apply to the whole filter. Second Noble Identity The second identity concerns upsampling. It is a transpose of the first identity. The standard form of a filter bank has (tM) before a synthesis filter F(z). For filters of the special form F(z) = G(z™), that order can be reversed. But again we must change G(z"’) to G(z): x— G(z) +M u = x— tM Ge) u This time we verify the equivalence in the z-domain. The output (¢ M)Gx comes from upsam- pling G(z)X (z): U(z) = GE")X("). In the other order, the upsampled signal X(z™) is filtered by the stretched-out G(z"). Again U(z) = G(z")X (z"), which proves the identity: Second Noble Identity: (} M)G(z) = G(z")(t M). (3.30) As before, this identity does not apply to most filters. We still need M — 1 zeros between the nonzeros. For a simple delay, G(z") = <7! is not possible. The Noble Identities are derived for systems with one input and one output. They also hold for systems with multiple inputs and outputs, as long as the decimation (or interpolation) factors are the same for all channels.102 Chapter 3 Downsampling and Upsampling We soon move to the polyphase decomposition of a filter (and a filter bank). The phases have ‘M — 1 zeros and the special form involving 2". In the polyphase form, the downsampling can come first. And upsampling comes last. Problem Set 3.4 1. Verify the first Noble Identity G(z)(J M) = (| M)G(c") in the z-domain. 2. Verify the second Noble Identity (} M)G(z) = G(z"")(¢ M) in the time domain. 3. Simplify the following system: xn) 42 43 Lf 42 yin) What is ¥ (z) in terms of X (z)? Find y(n) for the following inputs: x(n) =-(n), x(n) ae es te Reade eee een ena 4, What is ¥(z) in terms of X (z)? What is y(n) in terms of x(n)? xo) 93 + 43 ” yin) 5. If H(z) = G(<™) has M — I zeros between h(0),#(M), ..., find y(n) in terms of x(n) and h(n) for this system: x(n) + pM H 1M yin) 6. What identity would you use to compute v = (|, 2)Hx efficiently, if H has only odd-numbered coefficients h = (0, h(1), 0, (3))?Chapter 4 Filter Banks 4.1 Perfect Reconstruction A filter bank is a set of filters, linked by sampling operators and sometimes by delays. The down- sampling operators are decimators, the upsampling operators are expanders. In a two-channel filter bank, the analysis filters are normally lowpass and highpass. Those are the filters Hy and Hi, at the start of the following filter bank: x) Hy POOP _ ae Hh y(n) y2 io fz} 7 Fin) input analysis decimators expanders synthesis output This structure was introduced in the 1980's. It gradually become clear how to choose Ho, M1, Fo, F; to get perfect reconstruction: ¥(n) = x(n — 1). The gap in the figure indicates where the downsampled signals might be coded for storage or transmission. At that point we may compress the signal and destroy information. Perfect reconstruction assumes no compression, so the gap is closed. To indicate that Ho is lowpass and Hj is highpass, we often sketch the frequency responses. Figure 4.1 shows that they are not ideal brick wall filters. The responses overlap. There is alias- ing in each channel. There is also amplitude distortion and phase distortion (our drawing does not show the phase). The synthesis filters Fy and F, must be specially adapted to the analysis filters Hy and H, in order to cancel the errors in this analysis bank, The goal of this section is to discover the conditions for perfect reconstruction. This means that the filter bank is biorthogonal. The synthesis bank, from Fp and F and 2, is the inverse of the analysis bank. Inverse matrices automatically involve biorthogonality. (The rows of T and the columns of T-! are by definition biorthogonal.) This will extend in Chapter 6 to biorthogonal scaling functions and wavelets. Perfect reconstruction is a crucial property. If the sampling operators (| 2) and (f 2) were fot present, a reconstruction without delay would mean that FoH + F)H, = I. A perfect re- construction with an [-step delay would mean (in the z-domain) that without (| 2)and (¢2): Fo(z)Ho(z) + Fi(2)Hi(z) = 27. 4.1)104 Chapter 4 Filter Banks Ho] [A Hg = lowpass filter H, = highpass filter Figure 4.1: Rough sketch of frequency responses. Do not expect |Ho(w)t + |Hi(w)| = 1. ‘We do expect an overali delay z~', because each individual filter is causal. Now take account of the sampling operators, which introduce aliasing. We recognize alias- ing by the appearance of —z as well as z (and w + as well as w). The combination of (|. 2) followed by (ft 2) zeros out the odd-numbered components. In the z-domain it keeps only the even powers of Ho(z)X (z): The transform of (t 2)({ 2)Hox is $(Ho(z)X(z) + Ho(—z)X(—2)). This is an even function, because the odd components are gone. The aliasing term Ho(—z)X(—z) is multiplied by F(z) at the synthesis step. This alias has to cancel the alias Fi (z) Ht (~z)X (—z) from the other channel. So there is an alias cancellation condition in addition to a reconstruction condition: soe ¥ Alias cancellation Fo(z) Ho(—2) + Fi(z) Hy (-z (4.2) Correction The sampling operators also produce a change in equation (4.1). The right side has an extra factor 2. You can see this by considering a simple set of filters: Ho(z) = 1 and H(z) = 271, Fo(z) = 27! and F,(z) = 1. This satisfies (4.2) and cancels aliasing. The left side of equation (4.1) equals 22~" rather than z~', ‘The overall delay is ! = 1 for this filter bank, and its perfect reconstruction comes from No distortion F(z) Ho(2) + Fi (2) F(z) (4.3) ‘The next page establishes these two conditions (4.2-4. One further point. In a genuine filter bank, the highpass filter has Hy = Oatz = 1 (orw = 0). Equation (4.3) becomes Fo(1) Ho(1) = 2. That equation is more natural if we include an extra factor /2 in the filter coefficients. For a similar reason the highpass filters can be normalized by an extra /2. We take this opportunity to assign single letters C = /2Hp and D = /2H; to the analysis filters, with no need for subscripts. The sum of lowpass coefficients c(n) = /2h(n) is V2. No Aliasing and No Distortion Conditions (4.2) and (4.3) for perfect reconstruction come directly from following a signal through the filter bank. The original signal is x(). The lowpass analysis filter is Ho. In the z-domain this produces Ho(z) X (z). Now downsample and upsample: First (| 2) produces }[Ho(z?)X (4) + Ho(~z4)X(—2})] Then (2) produces 3 [Hp(z)X (z) + Ho(—z)X(—2)].4.1 Perfect Reconstruction 105 (t 2) () 2) Hox has zeros in its odd-numbered components. Those zeros are produced by av- eraging Hox(n) with its alternating alias (—1)"Hox(n). In the z-domain this is the average of Ho(z)X(z) with Ho(—z)X (—z). The aliasing term has entered the filter bank. The final filter multiplies by Fo(z). This yields the output from the lowpass channel. Below it we write the corresponding output from the highpass channel (same formula with subscripts changed to 1): lowpass output 3 Fo(2) [Ho(2)X (2) + Ho(~2) X(-2)] highpass output = $F, (z) [M(z)X(z) + Ay(—z) X(-2)]. Now add. The filter bank combines the channels to get ¥(2). In the z-domain this is Re (z). Half the terms involve X (z) and half involve X (—z): 1 1 gl Fotz) Holz) + Fr@Hi@)1X@) + lFo(2) Ho(—z) + Fy(z) Ay (-2)]X (-2). For perfect reconstruction with / time delays, this R@ must be z~'X(z). So the “distortion term” must be z~! and the “alias term” must be zero: Theorem 4.1 A 2-channel filter bank gives perfect reconstruction when Fo(z)Ho(2) + Fi(z)Hi(z) = 2277 44) Fo(z)Ho(—z) + F\(2)Hy(-2) = 0. (4.5) In vector-matrix form these two conditions involve the modulation matrix H»(z): Ho(z) — Ho(—z) [Fo(z) ncaa H(z) Ay(—2) | = (227 O. (4.6) This matrix H,,(z) will play a very important role. It involves the responses Hy(z) and their alias terms H,(—z). For an M-channel bank the matrix will be M x M. But the real problem is clearly identified by the separate conditions (4.4) and (4.5) — how to design filters that meet those conditions? Alias Cancellation and the Product Filter Po = FoHo At this point we have four filters Ho, H, Fo, F; to design. They must satisfy (4.4) and (4.5). It is almost irresistible to determine some of the filters from the others: " For alias cancellation choose Fo(2) = Hy(—z) and F(z) = —Ho(—z) (47) Important: This choice automatically satisfies Fo(z)Ho(—z) + Fi(z)Fi(—z) = 0. Aliasing is removed; it cancels itself! This relation of Fo to H; and of F' to Ho gives the alternating signs pattern of a 2-channel filter bank: Ho a, 6 ¢} =, 5, t] Foe) = H(-2) A [parse Oe = F\(2) = —Ho(-2)106 Chapter 4 Filter Banks Now comes a definition that allows us to rewrite equation (4.4) for no distortion: Define the product filter by Po(z) = Fo(z) Ho(2). This is a lowpass filter. The highpass product filter is Pi(2) = F\(z)Hi(z). These products Po and P, are exactly the terms in (4.4). The crucial point is the relation between Po(z) and P\(z), when the synthesis filters are determined by Fo(z) = Hi(—z) and Fi(z) = —Ho(—z). We substitute directly to find: that P,(z) = —P(—z): Pi (2) = —Ho(—z) A(z) = ~Ho(—-z) Fo(—z) = —Po(—z). (48) The reconstruction equation Fo(z)Ho(z) + Fi (z)Ay(z) = 2z~! simplifies to Fo(z)Ho(z) ~ Fo(—z)Ho(—z) = Po(z) — Po(—z) = 2z7!. (4.9) ‘The design of a 2-channel PR filter bank is reduced to two ste Step 1. Design a lowpass filter Pp satisfying (4.9). Step 2. Factor Pp into FoHo. Then use (4.7) to find F, and Hy. ‘The length of Po determines the sum of the lengths of Fo and Ho. There are many ways to design Po in Step 1. And there are many ways to factor it in Step 2. Experiments are going on as this book is written, and undoubtedly they are going on as the book is read, to find the best factors Fo and Hp of the best product filter Po. Note that (4.9) is a condition on the odd powers in Po(z) = Fo(z) Ho(z). Those odd powers must have coefficient zero, except z~' has coefficient one. The shift to P(z) in equation (4.11) below will remove the even powers except z°. Alook forward To help the reader find the specific filters that are coming, we point to an out- standing choice for the product filter: Po(z) = (1 +27!" OG). (4.10) The polynomial Q(z) of degree 2 p — 2 is chosen so that (4.9) is satisfied. There are 2p — 1 odd powers in Po(z), and 2p — 1 coefficients to choose in Q(z). Then Q(z) is unique. This is the Daubechies construction, with a history that we will outline in Section 5.5. Since the construc- tion starts with the special factor (1 + z~')??, these filters are called binomial or maxflat. The binomial factor gives a maximum number of zeros at z = —1, which means that the frequency response is maximally flat at = 1. The binomial by itself, without Q(z), represents a “spline filter”. Q(z) is needed to give perfect reconstruction, Splitting Po into Fo Ho can give linear phase filters (symmetry in Fo and Ho separately). Itcan give orthogonal filters (symmetry between Fo and Ho). It cannot give both, except in the Haar case p = 1. Section 5.4 discusses the factorization, and Chapter 11 reports some comparisons for image processing. Simplification The equation Po(z) — Po(—z) = 2z7! can be made a little more convenient. The left side is an odd function, so / is odd. Normalize Po(z) by z! to center it: The normalized product filter is P(z) = 2! Po(z).4.1 Perfect Reconstruction 107 Then P(—z) = (—z)' Po(—z). Since ! is odd, this is —z' Po(—z). The reconstruction equation Po(z) — Po(—z) = 227! takes an extremely simple form when we multiply by z!. The factor z~! disappears and the minus sign becomes plus: Perfect Reconstruction Condition. P(z) must be a “halfband filter” P(2) + P(-z) =2. (4.11) This means that all even powers in P(z) are zero, except the constant term (which is 1). The odd powers cancel when P(z) combines with P(—z) — so the coefficients of odd powers in P(z) are design variables in 2-channel PR filter banks. Example 4.1. The maxflat product filter Pg(z) = (1 + 27')*Q(z) happens to be Po(z) = (—1 + 92>? + 16279 +. 92-4 — 2-5), The center term is z~? = z~!. The requirement Po(z) — Po(—z) = 227! is quickly verified, because the even powers in Pp cancel in the difference. The function Q(z) 14427! — 27? was chosen so that the odd powers z~! and 2~5 are absent from Po(z). A centering operation gives the normalized product filter P(z). Multiply by z! = 23 to sym- metrize the polynomial around the constant term 2°: PQ) = (2 +92 + 164927! — 273), This is halfband, because the only even power is z° and its coefficient is 1. The perfect recon- struction requirement P(z) + P(—z) = 2 is verified. The odd powers in P cancel in the sum. Notice the variety of factorizations into Po(z) = Fo(z)Ho(z). The polynomial Po(z) has six roots. The two roots from Q(z) are ate = 2— V3 and ! = 2+ V3. The other four roots from (1 + 2~')* are at z = —1. Each factor normally has at least one root at z = —1. (But the factorization into Hp = 1 and Fo = Po is quite interesting.) Thus Fo or Ho (either order is possible!) could be @ 1 degree N=0 ) e745 degree N=1 () (+27 Por tz -27) degree N=2 @ CteorGd +z -27) degree N=3 The number N + 1 of filter coefficients is one greater than the degree. Thus (c) can produce a 5/3 filter, with Ho(z) = $(—1 + 227! + 627? + 22-3 — 2-4) and Fo(z) = EL + 2274 427), The analysis length is given first. This is a possible choice for compression, with symmetric filters of very low complexity. The binomial (1 + 2-1)? in the synthesis filter means that its continuous-time scaling function will be a hat function. Reversing Fy and Ho gives a 3/5 pair, not as successful in practice. ‘The choice (b) is also of interest. One factor is 1 + 2~' so one scaling function is the box function. Experiments indicate better performance when this short lowpass filter is in the anal- ysis bank. Where 5/3 was preferred to 3/5 for odd-length filters, it seems that 2/6 is preferred to 6/2. Five roots go into Fo{z). ‘The other outstanding choices are length 4/4 from the factorizations in (d). We get linear phase from (1 +27')? and (—1+3z~! +327?—z73). The orthonormal Daubechies filter comes108 Chapter 4 Filter Banks 3/5 4/4 © =roots of Ho(z) and x =roots of Fo(z) from (1+2~')?(¢—27!) and (1 +27")?( -z"!). These are not linear phase. (Problem 7 shows that linear phase requires t to be a root when c is a root.) For the Daubechies orthogonal choice, the roots —1, —1, ¢ of one factor are the reciprocals of the roots —1,—1, ! of the other factor. Section 5.4 demonstrates that this balanced splitting (spectral, factorization) of a halfband filter produces an orthogonal filter bank and orthogonal wavelets. Example 4.2. (Haar filter bank) The average-difference analysis filter has Hato) =[ H(z) Ho(~2) l-3| l42% oz! | Ay(z)-Ay(-2) t-e' pee The synthesis filters are Foe) = M(-2) = te fv F@ = -Hol-2) = -U-27//72 * The product filters are Po(z) = Fo(z)Ho(z) = 4(1 + 271)? Pi@) = Fi@ A(z) = -31 — 2! = —Py(~2). Both Po and P, contain +2~', Perfect reconstruction Po(z) — Po(—z) = 227! means that! = 1. ‘The normalized product filter (symmetric halfband) is P@) =z Poa) = fel te" sdetd thet. Modulation Matrices The conditions for perfect reconstruction are expressed in (4.6) by Ho(z) Ho(-z) | _ 5-1 (Fo(z) nel MG) Wo) ]=w ol. (4.12) This displays the analysis modulation matrix H,,(z)— which is central to filter bank theory, With no extra effort we can also produce the synthesis modulation matrix Fn (z). The two ma- trices should play matching (and even reversible) roles. This balance between Fy, and H,, is achieved by expanding (4.12) into a matrix equation: [ Foe) Fi) | [ Ho(z) Ho(—2) ] 27 0 Fa(—2) Fitz) || i) Hi(-2) =[¢ A-2! [ 4.13)4,1 Perfect Reconstruction 109 The second row of equations follows from the first, when —z replaces z. Note that Fj (z) is in the (1, 2) position, but H(z) is in the (2, 1) position. This transpose convention between analysis and synthesis will appear again for polyphase matrices. The reader sees why it is necessary. The reconstruction condition (4.13) applies to F»(z)Hm(z). If we “center” the filter coeffi- cients around the zero position, the right side becomes the identity matrix! This is so desirable and memorable that we do it. It is the same normalization that centered Po(z) into P(z), and it is especially clear when the filters are linear phase (and / is odd): Theorem 4.2 /f ail filters are symmetric (or antisymmetric) around zero, as in H(z) = H(z~!) and h(k) = h(—10, then the condition for perfect reconstruction becomes a statement about inverse matrices: eseceeannueneecrenstenesnic F(z) Am (z) = 20. * (4.14) The H’s determine the F’s. The analysis bank is inverted by the synthesis bank. When we ex- press it that way, equation (4.14) becomes almost obvious. A Brief History of H, The reader understands that the filters Ho and H are still to be chosen. These choices are con- nected, Historically, designers chose the lowpass filter coefficients A(Q), ...4(N) and then con- structed H from Hy. Here are two possibilities that produce equal length filters. H, will be highpass whenever Ho is lowpass: Alternating signs: H,(z) = Ho(—z) comes from (A(0), —h(1), h(2), —h(3), ...) Alternating flip: Hy (2) = —z-" Ho(—z7') comes from (A(N), -A(N ~ 1), ...)- For convenience we are assuming real coefficients. The number N is odd in the alternating flip. The perfect reconstruction condition is still to be imposed. When that is satisfied, the overall system delay is / = N. Early choice. Croisier—Estaban-Galand (1976) chose alternating signs H\(z) = Ho(—z). The resulting filter bank was called QMF (Quadrature Mirror Filter). The highpass response. 1H (e/#)| is a mirror image of the lowpass magnitude | Ho(e/)| with respect to the middle fre- quency %—the quadrature frequency. Note that IIR filters Hy and Hj are allowed (and needed for PR, except for Haar!). This name QMF has since been extended to a larger class of filter banks, allowing M channels. Better choice. Smith and Barnwell (1984-6) and Mintzer (1985) chose the alternating flip A(z) = —27" Ho(--z~'). This leads to orthogonal filter banks, when Hp is correctly chosen. The Daubechies filters will fit this pattern. General choice. The product Fo(z)Ho(z) is a halfband filter. This gives biorthogonality, when aliasing is cancelled by the relation of Fy to H; and F; to Ho. Actually the synthesis bank has little freedom. Alias cancellation requires Fo(z)Ho(—z) + F\(2) i (—z) = 0. Croisier-Estaban-Galand wrote each F; directly in terms of Hy by Fo(2) = Ho(z) and Fi (2) = — Hi(2)-110 Chapter 4 Filter Banks With alternating signs H(z) = Ho(—z) inside the analysis bank, their synthesis construction agrees (as it must) with the anti-aliasing equations Fo(z) = Hy(—z) and Fy(z) = —Ho{—z). (4.15) Smith-Barnwell also made the anti-aliasing choice (4.15). With the alternating flip in H(z), their synthesis filters (remembering that N is odd) are Fo(z) = — Ai(~z) = 2% Ho(z7!) comes from (A(N), ACN — 1),..., 2(0)). Fi) = —Ho(-2) 27" Hy(2!) comes from (—A(0), A(1), A(2),..., A(N)). Notice! Each Fy in Figure 4.1 has become the ordinary flip of the corresponding Hy. In ma- trix language the synthesis matrices are the transposes of the analysis matrices. A shift by NV delays makes them causal. When we flip to get Fo and then alternate signs to get Hy, we have the alternating flip from Ho to Hy. The alternating lip automatically gives double-shift orthogonality between highpass and lowpass (to be explained). Conclusion: When the design of Ho leads to perfect reconstruction in the alternating flip filter bank, it also leads to orthogonality. Ho a, b,c, d| ea Fo(z) = Hy(-2) = alternating] fip ‘alternating signs .b-a [ab -0 @ | A@=-Hl-2 ® 3 Ho(2~!) Figure 4.2; Relations between the filters allowing orthogonality when N = 3. With aliasing cancelled, we now look at the PR condition Fo(z)Ho(z) + Fi(z) H(z) = 227, The early choice was alternating signs H,(z) = Ho(—z). With Fo(z) = Hy(—z) and Fy(z) = —Ho(—2), PR requires Hg (2) — Hi (2) = Hg(@) — H3(-2) = 2271, (4.16) Therefore H(z) has exactly one odd power z~. This is not easy for the square of a polynomial. An FIR filter is restricted to two coefficients (not good). Problem 4 asks for the reasoning, which forces the filters to be IR. The better choice is alternating flip. Perfect reconstruction is definitely possible. Product fil- ters FoH and Fi Hy become Po(z) = 2~" Ho(2~')Ho(z) and P)(z) = —z~"Ho(~2~") Ho(—2). Multiply by z' = z® to center these filters. The normalized product filter is P(z) and the recon- struction condition is (4.11): P(z) + P(-z) =2 with P(z) = Ho(z7') Holz). (4.17)4.1 Pertect Reconstruction WW This is spectral factorization of a halfeand filter! On the unit circle z = e!®, the product Ho(e7!”) Ho(e!®) is a magnitude squared: 2 N N Preity = Spine" = |S aime], -N 0 (4.18) The halfband coefficients are p(n) = p(—n) for odd n and p(n) = 0 for even n (except p(0) = 1). We design P(z) and factor to find Ho(z). This symmetric factorization coincides with the Smith- Barnwell alternating flip. /t yields orthogonal banks with perfect reconstruction. The flattest P(2) will lead us to the Daubechies wavelets. Anote on biorthogonality (PR) with linear phase Theorem 4.3. Ina biorthogonal linear-phase filter bank with rwo channels, the filter lengths are all odd or all even. The analysis filters can be (a) both symmetric, of odd length (b) one symmetric and the other antisymmetric, of even length. Proof: Odd and even lengths behave differently when we alternate signs: oddlength: a bc ba + a —b € —b a (remains symmetric) evenlength: ab b a > a—-bb -a (becomes antisymmetric). To cancel aliasing, there is sign alternation in Fo(z) = H,(—z). There is also alternation in F\(z) = —Ho(—z). The extra minus sign does not change the symmetry type. The two success- ful combinations are Ao = symm Fo =symm Hg = symm Fy =symm A, =symm F, =symm Ay = anti F, = anti odd lengths even lengths The other possibilities are excluded by the PR condition: Fo(z) Ho(z) has to be a halfband fitter, It must have an odd number of coefficients, and the center coefficient must be 1. For Fo(z) Ho(z) to have odd length (which means even degree), the factors Fo(z) and H(z) must be both odd length or both even length. If one is symmetric and the other antisymmetric, the product Fo(z) Ho(z) will be antisymmetric with zero at the center — not allowed. We conclude that Fo(z) and Ho(z) must match: both odd length or both even length, both symmetric or both antisymmetric. This leaves the two successful possibilities shown above, and two more: Ho and Fo both antisymmetric. But the sum of lowpass coefficients cannot be zero. So antisymmetry of Hp is tuled out. Perfect Reconstruction with M Channels In reality a filter bank can have M channels. Al- though M = 2 is standard in many applications, we often see M > 2. There are M analysis filters Hp, Hy, ..., H—1. The sampling is done at the critical rate by (| M) and (t M). There112 Chapter 4 Filter Banks td v u «Lo rap tM tT rey | r~_h@ 1M tM F(z) *) ‘9 Har [4 t Pf at LY Fyn @ Je Figure 4.3: Standard form of an M-channel filter bank (maximally decimated). are M filters Fo, Fi, ..., Fy) in the synthesis bank. The outputs from all channels are com- bined into a single output ¥. Our standard picture of this implementation is Figure 4.3. For this M-band case, the theory of perfect reconstruction was developed over several years. When we follow each channel from Hy through (| M) and (1 M) and F,, and add, we find M conditions for perfect reconstruction. These equations involve the M x M modulation ma- trix H(z) — corresponding exactly to the 2-channel case. The modulations are by multiples of 2n/M in frequency, and by powers of W = e-?"//™ in the z-domain. ‘We put that matrix on record here. H(z) Ho) ss Holz a) Modulation iG) Ai, (z) A(2W) see Ai (zw) (4.19) matrix eee fee eee eae Au-1@) Hu-@@W) +) Hya(gW-1) The last M — 1 columns represent the M — 1 aliases created by (J M), just as H(~z) represented the one alias (W = —1) created when M = 2. The transpose of H(z) is the alias component matrix, Another matrix will play an equally central role — in fact an interchangeable role, because it is very closely linked to H(z). This new matrix is the polyphase matrix H,(z). It is developed and explained in the following section, as the natural way to follow the “phases” when a signal is subsampled. We put on record the 2-phase matrix — this is the polyphase matrix when M = 2: Hoeven(z) — Ho,oaa(z) ole [ Hieren(2) —hoaa(2) ] : Still looking ahead, we mention especially the orthogonal case. The Polyphase matrix is then unitary for |z| = 1 (this makes it Pparaunitary). Hy,(z) is also Paraunitary, after dividing by /2 (Section 5.1). The analysis of Paraunitary matrices was led by Vaidyanathan. He and others built onto the early theory (and nearly indecipherable exposition) of Belevitch. For M = 2, the paraunitary matrix leads back to the Smith-Barnwell construction of an orthogonal PR filter bank. Several perfect reconstruction filter banks deserve special mention. Simplest is the average- difference pair from Chapter 1. This is a useful example but a poor filter. That is the first in a family of “maxflat filters”, corresponding to the Daubechies wavelets. The others in the family are orthogonal but not linear phase — since those two Properties conflict.4.1 Perfect Reconstruction 113 Different factorizations of the product Po(z) lead to linear phase (not orthogonality). Those filters have become favorites for compression. For M > 2, the design of separate filters Ho, Hi, ..., Hy—1 can become unwieldy. We look for constructions in which these all come from one prototype filter. A particular class is the cosine-modulated filter banks in Chapter 9. A phase change (= modulation) is the key to their construction, Those are efficient in every way. Problem Set 4.1 1. If G¥n(z))* is also.a polynomial, the synthesis bank as well as the analysis bank is FIR. Why must the determinant Ho(z) Hy (—2) ~ A(z) Hu(—z) in the denominator of H;,' be a monomiat cz? This determinant is an odd function, det H,,(2) = — det Ha (—2). Then the exponent ! must be odd. 2. The solution of both equations (4.4-S) for Fo and F, involves (Hm (z))~?: oy LO) Ai@l= Re! OI)" = z = aan, hd Ho(-2). If det H,,(z) = 227, as normal, this yields Fo(z) = Hy(—z) and F(z) = —Ho(-2z). Extra credit: Verify that the key equation P(z) + P(—z) = 2 still holds in the IIR case with the extended definition of the product filter _ 2 Fale) Holz) eae Oe 3. Find all filters if Hy(z) = (44)? and Pp(z) B14 92°? + 162° + 92-4 = 2°, 4. If an FIR filter Ho(z) has three ot more coefficients, explain why H2(z) has at least two odd powers. Then Hj (z) — Hi(—2) = 2z~' is impossible, The “alternating signs” construction is not PR. (This is extended in Theorem 5.3: Symmetry prevents orthogonality except with two coefficients.) 5. If Ho and Po are symmetric, why is Fy symmetric? Why is H; linear phase, and when can it be antisymmetric? 6, Prove Theorem 4.3 by observing that an order flip in linear-phase filters Hy(e) and Fo(e*) gives Hox and Fog: Hy(elt) = JM Hy p(e-I), Fale!) = e112 fy pei) Hole!) Foyle!) + Ho(eX@*™) Fy(ell#7 = 2e-Ito, No and N, are the degrees of Hy(z) and Fo(z), and € is odd. 7. A symmetric filter has H(z~!) = 2% times (__). If H(c) = 0 show that also H(1) = 0. 8. In the exampte with six roots, show a 4/4 linear phase pair—the roots c and 2 go together. Find the polynomials H(z) and F(z). 9% The 10th degree halfband polynomial Po(z) = (1 +2~')® Q(z) has four complex roots r, F, r-', 7" in the right halfplane (roots of @), Draw a figure to show the ten roots and how Daubechies 6/6 filters will divide them: r and F are separated from r~! and F~'. 10. For the same 10th degree Py(z), show how the ten roots can give 6/6 filters with linear phase, One filter has 5 zeros at z = ~1. IL. (Good problem) Find the actuat 4th degree Q(z) that makes Po(z) halfband. If possible com- pute its roots.114 Chapter 4 Filter Banks 12. Given a PR filter bank Hp, Hi, Fo, F), interchange H,(z) and F(z) (so that the synthesis bank has H,(z) and analysis bank has F,(z)). Verify that the new system is PR. Define another system #,(z) = Hy(—z) and F(z) = F,(—z). Us this new system PR? 13. Let Ho(z) be a symmetric lowpass filter with even length and Hy(z) = Hy(—2). Verify that H(z) is an antisymmetric highpass filter. Find the synthesis filters F(z) that cancel aliasing. Can this system be PR? (Is P(z) a halfband filter?) 4.2. The Polyphase Matrix This section establishes a key idea and a valuable notation. The word “polyphase” has gained a certain mystique in the theory of multirate filters. Perhaps we can begin by explaining the meaning of the word, and also the purpose of the idea. Then the notation and applications will come naturally. Meaning of polyphase: When a vector is downsampled by 2, its even-numbered components are kept. Its odd-numbered components are lost. Those are the two phases, even and odd. It is natural to follow the two phases of the input vector, Xeven and Xoga, as they go through the filter bank. They are acted on by the two phases Heven and Hoga of the filter. For downsampling by M there are M phases. The ideas still apply to this “several-phase” or “polyphase” decomposition. Instead of even and odd inputs we will have M vectors (phases of x). Instead of even and odd filters we will have M filters (phases of H). The vector of filter coefficients h(n) is separated into phases, exactly as x(m) is separated. Then we watch those phases during downsampling. The word “phase” is applied because the even filter with coefficients (0), #(2) has a differ- ent delay (phase shift) from the odd phase with coefficients #(1), #(3). Purpose of polyphase: The operation (| 2)Hrx, taken literally, is not efficient. We are com- puting all components of Hx and then destroying half of them. If we don’t compute them, the system is still working at a fast rate (high bandwidth). The output is at half rate, because of downsampling. Each output component needs N additions and N + 1 multiplications, to apply all the coefficients &(0), ...,A(N). The polyphase implementation works on the different phases separately. The input vector is separated into Xeven and Xoga- The operator (| 2) comes before the filter! It changes one input at a high rate to two (or M) inputs at a Jower rate. Then the separate phases of the filters act simultaneously (in parallel) on separate phases of the input. The notation has to keep track of each phase. Often we find that “even multiplies even” and “odd multiplies odd”. The Nobile Identities justify an interchange of filtering and sampling. For the whole filter this interchange is forbidden, but it is allowed for each phase. Polyphase in the time domain (block Toeplitz matrix): We can display the infinite matrix for a 2-channel analysis bank. Recall that e(n) = J2 ho(n) and d(n) = V2 h(n). The two filters /2Hy = C and ¥2H, = D are downsampled by (| 2). This removes the odd-numbered rows. Then we interleave the rows of L = (|2)C and B = (|.2)D to see the analysis bank as a4.2 The Polyphase Matrix 115 block Toeplitz matrix: 3) (2) e(1) ¢(0) 43) (2) d(1) (0) €3) ¢(2) (1) e(0) 43) d(2) (1) dO) Block Toeplitz H, = This takes the input in blocks (two samples at a time). It gives the output in blocks. It is time- invariant in blocks! By block z-transform, multiplication by the infinite matrix H, (which is block convolution) becomes multiplication by the polyphase matrix: , _f @ ct) J, af 62) 3) Polyphase matrix n= a0) dt) ]+ | dQ) | (4.20) The polyphase matrix is nothing but the z-transform of a block of filters. There are 2? or M? filters, from © phases of M original filters. Here those filters have four coefficients and their phases have two coefficients. Notice especially how the block matrix Hy relates to the two separate downsampled filters (12) and () 2)D: The efficient form downsamples the input first (to make blocks for Hp) The inefficient form downsamples /ast (after the filters C and D) The Noble Identities prove the equivalence. It is just a removal of useless odd-numbered rows and an interleaving of the remaining rows. Next we discuss the algebra and the implementation. Key identity in the z-domain: The even part of X(z) is 4X (z) + X(—z)). The odd part is }(X(z) — X(—2)). The first has even powers 1, z”, z*; the second has z, z’, z°. The original X is the sum of even plus odd (obviously). The same splitting holds for C(z), and furthermore for C(z)X(z). The key is to find the even part of C(z)X (2). It is the even coefficients of Cx that survive downsampling and appear in (| 2)Cx. In most of this section the lowpass filter is denoted by C, to avoid the subscripts on H. A simple and important identity shows how the even part of C(z)X(z) comes from even times even plus odd times odd: HC@X@+C(-DX(-D = HC) + C(—2) XQ) + X(-2)] ra 21) + HC@ — C-2UIX@) — X(-2))- In multiplying numbers, odd times odd is odd. But we are adding exponents, as in (z?)(z°) = 2°. So itis really odd plus odd, and even plus even, that yield the even part of the z-transform. This is the part that downsampling picks out, when Cx is decimated. The importance of the key identity is this. The left side involves all coefficients of C(z) and X(z). Each product on the right involves only half the coefficients. The multiplication in the z-domain, which is (| 2)Cx in the time domain, becomes computationally efficient. We don’t116 Chapter 4 Filter Banks want all of C(z)X(z), only the even half. The right side shows how to do half the work. Better still, it shows how even-even and odd-odd can be executed in parallel at half the rate. Downsampling an even function effectively replaces z by z'/?. It “closes the gaps” in 1, z?, 24 by changing to 1, z, 2”, For an odd function we will need a delay or an advance. We cannot change z and z° to z!/? and z¥/?, You will see how the coefficients of z~!, 2-3, 2-5 in C(z) become coefficients of 1, z~', z~? in the odd phase C,4a(z). There is a delay for the odd phase and a “delay chain” when there are multiple phases. This chapter works out the polyphase notation. We concentrate most on M = 2; the phases are even and odd. Then the polyphase forms of the analysis and synthesis banks lead quickly toa main goal of the theory. We find the perfect reconstruction condition on the polyphase matrices, when the filters are centered: = F(z) Hp) = This tells us, clearly and directly, what is required: 1, Ata minimum, H(z) must be invertible. (biorthogonality) 2. Better than that, its inverse F p(z) should be a polynomial. (FIR) 3. Better still, F(z) might be the transpose of H(z). (orthogonality) Incase 3, the polyphase matrices are “paraunitary”, The analysis and synthesis banks are orthog- onal. In the more general case 1, the banks are “biorthogonal”, In case 2, the synthesis bank is biorthogonal and also FIR. The rows of a matrix are always biorthogonal to the columns of its inverse. When the rows of one are identical to the columns of the other, the matrix is self-orthogonal. Then it is an or- thogonal matrix if real, a unitary matrix if complex, and a paraunitary matrix if it is a function of a complex parameter z. Polyphase for Vectors Any input vector x and any filter vector ¢ or # can be separated into even and odd: ¥ = (....4(0), 0, (2), 0...) +6..,0,2(1), 0, 2(3),0,...). The z-transform is separated into even powers and odd powers, as in XQ) = RO) HQ) 4-4) $4) $ (4.22) The even part has powers of z?. So has the odd part, when we factor out z~'. This is the poly- phase decomposition of x in the z-domain: X(]) = Xeven(2”) + 27! Xoaa(2) (4.23) Each phase has its own z-transform: x(0) ee (2) | > Xo(z) = Eix(2k)z* Xodd x(1) (3) | > X1@) = DV x(2k + De4.2 The Polyphase Matrix 117 Because of the z” in the definition, the in-between zeros are gone from Xo(z) and Xj(z). Please verify that the phases of X(z) = 27) + 2-7 +273 are Xeyen = 27! and Xoug = 1 +271. Now reverse the process, to recover x. Upsampling puts zeros back into Xeven and Xoag. Those zeros change z to z”. The odd phase is delayed by z~!, to move x(1) from position 0 to position 1, Then addition reconstructs equation (4.25), Here is the splitting and the reconstruction in block form. Notice that so far the filters are not included, and (| 2)x is exactly Xevea! 42 Aen) +2 x(n) ae . x(n) 42 Xeaa(t) tz = Important! The z at the start of the odd channel is because the odd phase has x(1) in its zeroth position. We have to advance the signal to achieve that. (See Problem 1.) But advances look bad in our flow diagram. So the advance can be replaced by a delay, if we make up for it at the end by delaying the even part too. Here is the “delay form” that we use in later sections. Please go through that form: with fa Delay 42 Xoaa(n — 1) $2 Only delays are involved! This is its advantage. Its disadvantage is that the output (n) isx(n — 1). The whole system equals a delay, where previously the system reproduced x. The delay form in the z-domain produces z2~'X (z) by delaying the even term: Polyphase x(n) 42 Xeven (tt) { t2 If x(a-1) ZX (2) = 2 Xo?) + 2-O-X1 (2). (4.24) This polyphase with delay is just the original definition multiplied by z~). Your eye will pick out this delay form. The output ¥ comes later than the input. We still call this perfect reconstruction, Here is the same delay form in the time domain: x(0) 0 1 (0) (0) | (42) #(2) =~ x(-1) x1) } ZA | x2) 7 0 x0) | _ 72 |N peep] \ pecn PE ceca x3) x(1) (t 2) 0 Z (2) x(1) 0118 Chapter 4 Filter Banks Polyphase Matrices for Filters The polyphase form of a filter C comes directly from the polyphase form of ¢ (the vector of filter coefficients). That vector separates into Ceyen and Coad. Its z-transform C(z) separates into phases. exactly as X(z) did: C]) = Colz*) + 271 Cy(2?). (4.25) The filtering step is C(z)X(z). This is ordinary filtering Cx, where even mixes with odd. But when downsampling picks out the even Part of the product C(z)X (z), it comes from even times even plus odd times odd. The transform of q 2)Cx is (C])X (z))even = Col2)Xo(z) + 27 1C1(2)X4(2). © (4.26) ‘The direct multiplication of C(z) times X {z) will have even parts from Co(z?)Xo(z?) and from z-?C; (z?)X1 (22). Those give the even part of C(z)X(z). Downsampling picks out those terms. It changes z? to z in their transform. The result is the z-transform of (12)Cx, Example 4.3, The moving average filter, downsampled. Chapter 1 introduced the lowpass filter 4x(n) + 54(n — 1). The coefficients are h(O) = } and A(1) = }. Thus } is the leading and only coefficient in Haven and Hogg. Both matrices have 4 on one diagonal — the main diagonal. Here is the two-phase form of (| 2)Hx: Pe TEE TERE IE] The polyphase components of } + 427 are constants: Heven(2) = Hoga(z) = $. Polyphase for one filter. The downsampling operator (| 2) follows the filter C. The outputs are Cx and then (| 2)Cx. This is the normal order for filter banks, but it can be made more efficient. A close look at the product (|. 2)C shows that the filter coefficients e(n) are completely separated into even n and odd n. Thus C has two parts (or phases), which have their own z-transforms Co(z2) = Ceven(z) and C1(z) = Coga(z). The 1 by 2 matrix C>(2) is the two-phase or polyphase form of C(z): Cp(z) =[Co(z) Ci (z)]. Warning. Co(z) is not 3(C(z)-+C(~z)). That involves 1,2, 24, . .. with zeros between. This is Co(z?). By definition, Co(z) closes the zero gaps and has coefficients ¢(0), ¢(2), e(4). Simi- larly Cy (z) has coefficients e(1), e(3), e(5). An advance or delay is involved. We don’t keep the zeros from the even powers in $(C(z) — C(—z)). ‘The same splitting occurs for the highpass filter D. Its even and odd Phases are represented by Do(z) and D,(z). Those g0 into the 1 by 2 polyphase matrix D,(z). Then the whole anal- ysis bank comes together when we combine the polyphase matrices for C and D into a single polyphase matrix Hp (z): ramecrinne mom [ $0 ]=[ G0 8] am ‘This shows the matrix that we are aiming for. Now we go back for the close look at CL 2)€. This operator is fundamental in the theory of multirate filters and wavelets.4.2 The Polyphase Matrix 19 Polyphase in the time domain. When downsampling follows the filter C, we get the crucial matrix L = (| 2)C. This has to display the separation of even and odd, and I would like to show how this happens. Most of polyphase theory is developed in the z-domain, and we will do that too. But first, look at the filter matrix as it produces y = Cx: (0) ~ €(0) x(0) yy | _} + eG) (0) x(1) yQ) |=] - €@) ef) e@) x@) | 28) ¥(3) + €3) (2) e¢(1) e@) x(3) Downsampling leaves the even-numbered components y(2n). To reach » = (| 2)y, we throw away the odd-numbered rows. This leaves the matrix Z = ({2)C: x(-1) y(0) > (1) (0) oo y2) |=] - ¢@) 2) e(1) ¢O) x(1) (4.29) w(4) - e(5) ¢(4) (3) (2) e(1) ¢(0) x(2) For polyphase here is the important point. Only the even-numbered coefficients c(2n) are multi- plying the even-numbered coefficients x(2n). The even and odd e’s are in separate columns. The even-numbered x(0) is multiplying the column that starts with ¢(0). The odd-numbered com- ponent x(1) is multiplying the column containing ¢(1), ¢(3), .... We can separate the matrix multiplication (| 2)Cx into even times even and odd times odd: (0) + c(0) ( - e) i yo) |=] - c@ eo sm | t]| - @ CD | aa0 ¥4) - e(4) ¢(2) (0) + e(5) ¢(3) e(1) a This is a matrix display of equation (4.21): (}2)Cx = Coven Xeven + (delay) Cogs Xoas- (4.31 ‘The two phases Xeven and Xogg are filtered by the two polyphase components Ceyeq and Cogs. We need a delay in the odd phase, because ¢(1)x(1) contributes to y(2) and not toy(0). Then (J 2)Cx is the sum from the two phases: Xeven Coven f~ v= (12)Cx = Lx. odd —- Coad delay Notice something nice. The two matrices in equation (4.30) have constant diagonals. The two operators Ceven and Coga are time-invariant filters. They have frequency responses Co(z) =120 Chapter 4 Filter Banks Coven (2) and C(z) = Coga(z). The delay in the odd channel can g0 before or after C), because it commutes with C,. (C; is time-invariant!) The two filters involve even coefficients e(2n) and odd coefficients ¢(2n + 1), without zeros in between. They can operate in parallel, more efficiently. Summary: Polyphase form of L = (| 2)c ‘The matrix Ceyen multiplies Xeyen in equation (4.31). The matrix Coag roultiplies xoaq (with a delay). We repeat this time-domain multiplication so you can compare it with the z-domain: x 2Cx = [Coven (delay) Coaal [ st | : (4.32) This polyphase form has two ordinary filter matrices side by side. The z-domain polyphase form has two ordinary transfer functions side by side: (C(EDX even = [Covea2) =eater[ esi ]. (4.33) If C = identity then Ceyen(z) = | and the output is Xo(z). If C = delay then Coag(z) = 1 and the output is X;(z). All straightforward, but the block form shows something remarkable: Downsampling comes before filtering ‘ The block form with an advance to compute Xoqq is Polyphase with Advance: 12 Xeven x(n) +] z f—, ($2)Cx(n) rl +2 *odd An advance has slipped into this form to get x(1) as the zeroth component Of Xogq. If we prefer to have only delays (and we do), that is possible. Delay x in the odd channel so that (| 2) produces *oaa(n — 1). Then after filtering, add a delay in the even channel. The result is to delay the whole signal by one time step in Polyphase with Delay: x(n) 42 even (Ht) Coveo zt cee 12 Xoua(n — 1) Coad (WDCx(n - 1) Key point: The polyphase form puts (| 2) before the filters. This order is more efficient. It was possible to use the Noble Identity on each phase separately, because Ceven(z2) and also Cosa(z?) appeared in the right place: (direct) (1 2)Ceyen(2) = Ceven(2?)(42) (polyphase).4.2 The Polyphase Matrix 121 Noteonour convention. It would be very satisfying to avoid the delays completely. We would prefer to have V@) = {Colz) cel ae il (434) To achieve this we can alter the definition of C,(z). The polyphase decomposition of X would stay the same, but the decomposition of C would change to C(z) = Cole”) + 2C1(z”). (4.35) Note z in the odd term where we had z~!. This is the convention chosen by [VK]. It is just as good as ours. The zeroth component of Cogg becomes e(—1). It multiplies x(1). But I am afraid that in later chapters you would forget (I would too) this convention for ¢, different from x. So we keep the same even-odd decomposition of ¢ and x, yielding parallel decompositions of C(z) and X(z) —and requiring the delay. Problem Set 4.2 1. Find Xeyea(z) and Xogg(z) when X (2) = 1+ 2z-5 + 2-19. Verify that Xeven(z?) = }(X(z) + X(—z)) and Xegs(z*) = $(X (z) — X (—z)). The odd definition involves an advance! 2, Express the z-transform of + 2(| 2)x in terms of Xeaa and/or Xeven- What operations on x would produce the vector whose transform is X(z?)? 3. The phases Co, C), Do, D, are all time-invariant, so they commute with delays. Does it follow [' aw We JL IL! ow b 4, Polyphase Representation of an HR Transfer Function Let H@) = pL where 0 < a < 1. Its impulse response is h(n) = a" for n > 0 (and zero for negative n). ‘The phases are Reyea(t) = (1,42, a, ...) and Royy(nt) = (a.a?,a°, ...). The z-transforms are Heven(2) = 1/(1 — 4727") and Heag(z) = a/(1 —a?2~'). This method is very cumbersome. One has to find the impulse response A(n), then its even and odd parts Keven (71) and Fogg (7), then the z-transforms. An alternate method is to write H(z) = —L- directly as H(z) = Heven(z?) + 27! Hoaa (2). ‘The denominator must be a function of 2. So multiply above and below by 1 + az: a ae ‘This displays Heven(z) and Hoya(z). An Nth order filter can be factored as a cascade of first- order sections, and this method applies to each section, (a) Let H(z) = Tesla Factor H (z) into two first-order poles. Find the polyphase com- rere ponents of H(z). alas? ‘ (6) Let H(z) = HEEPHES. What are its polyphase components? 5. For M = 4channels, we want the four polyphase components of H(z): H(@) = Haz!) +21 ya!) + 27 Hale!) + Ha"). (a} What polynomial muttiplies | — az~! to produce 1 — a*z~4? (b) Find the four components for Ly and #245 Heh122 Chapter 4 Filter Banks 6. If H is a symmetric filter, how many of its phases are symmetric filters? 7. Let H(z) = 142271 +3277 +4279 42-4 4-32-5422 4-277, Find the polyphase components Heven(2) and Hoga(z). What is the relation between Hyven(z) and Hyga(z) for antisymmetric filters of even length and symmetric filters of odd length? 8. What are the two polyphase components of a symmetric halfband filter? Generalize to an M-th band filter. 4.3 Efficient Filter Banks Let us repeat the key idea of the polyphase decomposition. The input is x, the filter is C, and we intend to downsample Cx. We want the even powers of z in the product C(z)X(z): even powers in C(z) times even powers in X (z) even powers come from) 44 powers in C(z) times odd powers in X (2). The polyphase decomposition is exactly this separation. Even times even is Co(z2)Xo(z?). Odd times odd starts with e(1)z~! times x(1)z7!. The product ¢(1)x(1) enters Cx(2) because indices add. Then Cx(2) after downsampling is v(1). Since ¢(1) and x(1) are the zeroth components of the odd phases, we need a delay to put their product into (1). The algebra with X(z) = Xo(2?) + 27'X1(@?) and C(z) = Coz?) + z-'C1 (2) is (EX)o(2") Colz?)Xo(2?) + 27C1 (22) X1 (22) (CX)o(z) Colz)Xo(z) + 2-1Cy(2)X1(z). = (4.36) We reached this answer by filtering and then subsampling (the step when z? becomes z). But now we see a better way. Sample first to get Xo = Xeven and X| = Xodd- Filter those separately (and in parallel) by Co and C,. Then combine the outputs with a suitable delay — the step down on the right: Xo) Polyphase x(n) 42 | a } with ee | ay zt Delay v2 Se re vq — 1) In this polyphase form, the filters Co and C; are half as long as C. The output can be com- puted twice as fast, if all operations are done at the same speed. We could also use two cheaper low bandwidth processors, both working full time. They do x multiplications and additions per unit time instead of N. By downsampling first, we reduce the input rate to the filters. The band- width is halved. Example 4.4, The averaging filter in its direct form computes 4x) + 4x(- 1). Then it sits idle for one clock step, before computing }x(2) + 4x(1). The polyphase form has Heyen and Hosa multiplying separately by 4. They do one multiplication each, in parallel. An easy-to-write code will execute the polyphase form.4.3 Efficient Filter Banks 123 Polyphase for Filter Banks ‘The polyphase idea extends from one filter to a bank of filters. The direct form of the analysis bank does the downsampling last. The polyphase form does the downsampling first. In the block diagram of the filter bank, the decimators move outside the filters. We can write C and D or Ho and H, for the lowpass and highpass filters (0 and 1 do not mean even and odd!): Direct x(7) 4 ¥o(n) i Polyphase -*("") 42 vo(n) with zo ei xi(n—1) Hy) Delay 42 sem _| I-— vi (n) ‘The polyphase matrix multiplies Xo(z) and 2~! X (z) to produce Vo(z) and V, (z): Vo(z) Cofz) Ci) Xo(z) Xol2) [ Vie) ] = [ Doz) Dice) ] [ ZX) ] = Holz) [ z'Xi@) | ca This defines and displays H(z). For FIR causal filters, the kind we expect to use, the polyphase components are polynomials in z~!. When the input x is also causal, the outputs are causal. Another point about notation. The indices in Xo and X, refer to even and odd. The indices in Vo and V, refer to the two channels. This is normal for matrix multiplication, when Hj; multiplies X, and contributes to V;. Rows of Hp(z) go with channels, and columns of H (2) go with phases. In an M-channel bank, i is the channel index and j is the phase index in Hjj(z). Then V; is the output from channel i, and X, is the jth phase of the input. We often reorganize a filter bank into its polyphase form. Example 4.5. Average-difference filter bank in polyphase form. The averaging filter Hox(n) = }x(n) + }x(n — 1) has polyphase components Hoeven = Fy odd = } (identity). The differencing filter Hjx(n) = 4x(n) — $x(n — 1) has components Hyeven = $1 and Hy ogg = —$1. Note that H(z) and H(z) are linear but the polyphase matrix is constant — typical of a block transform: Hoe) Hu@) ]_f 3 4 ayo =[ Mol) ule) |-[ fat |. Example 4.6. Four-tap filters yield two taps for each phase. The even phase Ceyen has two coefficients ¢(0) and ¢(2). The odd phase has Coag = ¢(1) + ¢(3)z7!. The same pattern holds for D. The polyphase matrix for the filter bank is —t 1 -1 Hy) = [ (0) +e(2)z7!_—e(1) +e(3)z | : 40) +d(2)z7 d(1) +4Q)z7! (4.38)124 Chapter 4 Filter Banks Even is separated from odd. This reflects what happens in the filter bank, when Cx and Dx are downsampled: vo |_| (2)Cx | _[ Coven Coas 1 ¥even b | | GQ 2)De || Deven Doss delay Xodd In case you like matrices, we are going to write the time-domain filter bank matrix in three ways. Downsampling is included in all three! First comes the matrix Hy = Hirec: that multiplies the input vector x in the direct form: e(3) (2) (1) (0) ¢(3) ¢(2) efi) e(@) Direct Hy = ae (4.39) (3) d(2)) d(1)_ (0) 43) d(2) dl) di) Downsampling has removed every other row, That leaves this “square” infinite matrix, Each column is completely odd or completely even. For the second form we rearrange the rows of Hy. The highpass outputs are interleaved with the lowpass outputs, both downsampled by 2. This produces the block-diagonal form (or block Toeplitz form) Hy = Hotock: (3) ¢(2) (1) e(0) d(3) dQ) d(1) d(0) 3) ¢(2) e(1) (0) - G40) dB) d(2) dl) dQ) Block Hy, = ‘Your eye will divide that matrix into 2 by 2 blocks. It is like an ordinary time-invariant constant- diagonal matrix, but the entries are blocks instead of scalars. The main diagonal block corre- sponds to the constants in the polyphase matrix. The subdiagonal block produces the z~! terms. ‘There are only two diagonals because the phases of C and D have two coefficients, The original C and D had four coefficients. The third form is the polyphase form H,. We are still in the time domain. For this third form we rearrange the columns of the direct form: ¢(2) (0) ¢(3) (1) (2) (0) e(3) (1) Polyphase Hy= | . |] [ a o | (441) 4(2) d(O) dQ) d(1) (2) d(0) 4(3) d(l)4.3 Efficient Filter Banks 125 When the columns are rearranged, the vector x must be rearranged. Here Xeven comes above Xoad(delayed). The transform of the time-domain matrix H, is the z-domain polyphase matrix H(z). This 2 by 2 matrix of filters becomes a2 by 2 matrix of functions. The block form H, is an infinite matrix of 2 by 2 blocks. The polyphase form H, is a2 by 2 matrix of infinite blocks. Each block is a time-invariant filter. Either form leads by z-transform to the 2 x 2 polyphase matrix 4 )(0) + 2~'Rp(1): Polyphase #,@) =[ (0) (I) | - | (2) (3) | matrix dQ) d(1) d(2) d(@3) (4.42) Relation Between Modulation and Polyphase To produce two vectors from x, one way is by polyphase. The parts Xeyen aNd Xogq are half-length. The other way is by modulation. Both x and Xmoa are full length (therefore redundant). The modulated vector Ximoa reverses the sign of odd-numbered components: Xmod(t) = (—1)"x(n) and Xgog(z) == X(-z). In the frequency domain, —1 = e'* and w is modulated by 7 — which explains the name: Xmos(w) = X(@+ 2). (4.43) The vector xmos appears naturally when upsampling follows downsampling. Remember that 4 = (¢2)(1 2)x is the average of x and Xmod: B= 4 (8 + Xmod) = (..-.4(0), 0, 22), 0, ...). (4.44) In the 2-domain, the same fact gives the formula we use constantly: U(2) = $IX@) + X21. For downsampling and upsampling by M, the frequency modulation is by multiples of 27/M. The z-domain equivalent is multiplication by W = e~?/#. There are M terms X(W*z). We continue with M = 2 terms, X(z) and X(—z). The extra term represents aliasing. This modulated signal can overlap the original signal (in the frequency domain). If it does, we cannot recover x from u. Now place this step (t 2)(1 2) into a complete filter bank: | Ho 42 $2 Fo x(n) 4 + ¥(n) {mi ¥2 }{ 2 F The transform of Hox is Ho(z)X(z). When this is downsampled and upsampied, its alias appears in 4 Ho(z)X (2) + Ho(—z)X (—z)]. Multiplying by Fo(z) gives the output from the top channel. The lower channel is the same with 0 replaced by 1. The final output is the sum of channels: 7 7 Ro = NR reo eo te Il es | 445)126 Chapter 4 Filter Banks This is the 2 x 2 modulation matrix Hy,(z). It contains Ho(z) and H; (z) along with their aliases. The transpose of H(z) is also called the alias component matrix. A perfect reconstruction filter bank has to avoid aliasing (and other distortions). There is no aliasing when the combination Fo(z) Ho(—z) + F(z) H(—z) is zero. Section 4.1 presented the synthesis filters Fo(z) = Hi(—z) and F,(2) = —Ho(—z) that cancel aliasing. Now turn to polyphase, the separation into even and odd phases: H() = HA@ + HC21 + MA® — HO). (4.46) The first bracket is even. Replacing z by —z leaves it unchanged. The second bracket is odd. Replacing z by —2z reverses its sign. Therefore (4.46) must be the polyphase decomposition Heven(2°) + 2! Hoas(2?): (Heven(2") 27" Hoda?) = HLH) + H(-2)_ H@) — H(-2)) ej AAT = tH@) Hal 1 ]. an) This connects polyphase to modulation. For one filter H, and H,, have one row. For a bank of filters H, and H,, are square matrices. A 2-channel analysis bank has equation (4.47) for Ho in the top row, and the same equation for H; in the lower row: Hoeven(z*) Hp oda(2”) [1 a3) Ho@) Ho(-2)]f1 1 (4.48) Hieven(z”) Hy odd(2”) ie 271 Ai) A(-z) JL 1 -1 : Theorem 4.4 The polyphase matrix Hy is connected to the modulation matrix H» by a 2- point DFT and a diagonal delay matrix D(z) = diag(1,z~"). This pattern extends in Section 9.2 to a bank of M filters. Each filter has M phases, and it has M modulations (by multiples of 27/M). The phases are associated with time shifts, and the modulations are frequency shifts. So the DFT connects them. Problem Set 4.3 1, From H(z) = Heven(z?) + 27" Hosa(z”), find the corresponding formula for H(—z). Write the two equations as 7 oof 22 (A@ BH 2)) = [Aeven(z?) Hoaa(z7)] rt a a]: Identify that last mateix and verify that you have inverted equation (4.48). What is H(z) int terms of H(z)? 2. IEHE(2-") H(z) = 20 show from (4.48) that Hi") Hp (2) = 1. y . Find a new example of matrices H(z) and Hp(z) for Problem 2. 4. Write down the equations for M = 4 that are analogous to (4.46)-(4.48). Problems 5-11 develop an important example of biorthogonal filters. $. The upper channel has responses Ho(z) = 1 and Fo(2) = $4271 + $27 = late, Follow the signals x(n) = (—1)" and 6(t) through this channel by plotting (t 2)(] 2)Hox(n).4.3 Efficient Filter Banks 127 6. rh % 10. R. 13. ~n —_—,, + Plot the output wo(n) = Fo(t 2)(1 2)Hox(n) on top of these inputs. Describe the output from this channel with any input x(n): The odd wo(2k + 1) are and the even wo(2k) are ‘The lower channel has responses Hi(2) = Fo(—z) = } — 27" + 427? = $1 — 27)? and F,(z) = —Hy(—z) = —1. Follow the same two inputs (— ay and bin) through this channel by plotting Hyx(n) and w1(n) = Fi (12)(2)Hx(n). Verify that the sum of outputs wo(n)+# (2) is (2) = x(n — 1). In words, the output w (m) from the lower channel is. forn = 2k+1 and for n = 2k for any input x(n). Verify the perfect reconstruction condition on Fo(z)Ho(z) + F(z) (2) for this filter bank, What is the delay €? What is the product filter Py(z)? Find the centered product filter P(2), which is halfband. Find the modulation matrix H,,(z) for the analysis bank. Find the synthesis matrix F,,(z), remembering the transpose. Compute the product ee (2). Reverse the filters in the upper channel by Ho(z) = } + 27! + 427? and Fy(z) = 1. Follow the same signals x(n) = (—1)" and x(n) = 6(n) through this new channel. Construct the corresponding H(z) = Fo(—z) and F(z) = —Ho(—z) and follow the two inputs through the lower channel. Verify that the outputs wo(r)-+-w) (7) reconstruct the impulse x(n). Certainly Fo(z)Ho(2) + Fi(2)Hi(z) isthe same as before (still PR). The halfband filter P(z) = $z+1+ 427! was factored into P(z) 1 and then 1 ¢ P(z). Which filter bank factors P(z) into (1 “beh times (+2/¥27 Big question: Is the more regular filter } + z~" + 42-? better in analysis (Hp) or in synthesis (Fo)? The output wo should be a “good but compressed” copy of x. Show that the linear signal x(n) = n goes entirely through the lowpass channel. The highpass channel has Hix(n) = —_. The constant signal x(n) = 1 also goes through, so this filter bank {still from Problem $) has accuracy p = 2. Explain the equivalence of these representations before downsampling: Hig(2) Jolt) —- x(n) T— Yo) x(n) on z #,(2") AG) Fenn psi) : son eicig | Holz) | [ Hoolz?) Hai(2”) 1 tmatrix moan tis i | Hig) }-[ Hol) Hy (2) 1 | Explain the equivalence of these representations including downsampling: Yo) x(n) 42 LE vain) 2") JHE) y (ny 72 Le128 14, 15. 16, 17. 4.4 Chapter 4 Filter Banks Feven(tt) x(n) 42 vo(n) ct Xega(n — 1) A, (2) 42 v(m) Warning for downsampling by 3. The polyphase components when M = 3 are Co, Cy, Cz and Xp, X\, X2. We want the component Yo of the product, because Y, and ¥> are lost in downsampling. That component Yo does not involve C, times X. Multiplying z~! by z~! does not give z~*. The exponents 0, 3, 6, .., come from C; times X, and C; times X2 (and also Cy times Xq). To get 273 in the product, we multiply z~! by z-? or z" by 273: Yq(z?) = Colz?)Xalz?) + 2° CUA )Xa(z?) + 27°C?) X (2). ‘Then replace z° by z to find the transform of (| 3)y = (J 3)Cx. We save this pattern for a later discussion of (J Mf). It uses “Type 1 polyphase” components Co, C1, Cy. By defining Type 2 components Ro, Ri, Re we could achieve that R; multiplies X,. (The R’s are a permutation of the C’s.) Our immediate interest is restricted to (J 2). Write down the formula for ¥q(z*) that corresponds to equation (4.46). ‘The polyphase com- ponents of C are Cy, Ci. Cp. Show that X¢,es(z) is the z-transform of the downsampled vector (| 2)x. What vector is trans- formed to X.yy(z)? Write H(z) in terms of H(z) for these filters of length N + 1. (@) Ay(2) = Ao(—2) () Hi (2) = 2 Hole!) (e) Ay) = 20" Hy(-27). Find the analysis filters Hy(z) and H(z) for the following polyphase matrices: 2-21 14221427 1427! 1427? 24+z! 1- x e. Hc =[ : ie hazel 0 reer Lge? For each system (a-e) in Problem 17, find the modulation mawix H(z). Polyphase for Upsampling and Reconstruction The reader can anticipate what is coming. The polyphase form above was for the analysis bank, with downsampling. The same ideas apply to the synthesis bank, with upsampling. The ex- panders (+ 2) will again move outside the filters in the polyphase implementation. This time “outside” means after the synthesis filters. We start with one filter called G, and move the upsampler:4.4 Polyphase for Upsampling and Reconstruction 129 (n) Direet v(n) t2 un { w(n) Goas 42 Polyphase v(n) L ony Gorn t2 jue The polyphase form is more efficient, because the half-length filters Go = Geven and G, = Goaa teceive input at the slow rate (low bandwidth), The direct form unnecessarily doubles the rate by (t 2) before the filter. It is always important to verify formulas in the z-domain. In the direct form, upsampling produces U(z) = V(z?), The only powers to enter U (2) are even powers, because upsampling puts zeros in the odd components. Then the filter produces W(z) = G(z)V(z2). In this multi- plication, even powers in G yield even powers in W. The odd phase of G multiplies V(z?) to give the odd phase of W. That is why polyphase works in the synthesis filters. Even times even is separated from odd times even: 2 W@ =U | wes [=n caf cua |ve. Suppose the filter G is a simple delay. Then its Type 1 polyphase components are Geyen = 0 and Gras = identity. The output from this polyphase equation is correct —a delay of t 2v. Example 4.7. Suppose G has four coefficients g(0), g(1). g(2). (3). In the time domain, G is a constant-diagonal matrix. When upsampling acts before the filter to produce G(4 2), it removes columns of G. (Just as downsampling removed rows of C.) The time-domain matrix in direct and inefficient form is G(+t 2): g(0) 1 (0) gl) g(0) 0 ad) Gt2=] g2) gl) gO) 1 =! 82) 9g) g(3) (2) g{i) 80) 0 83) g(I) (4.49) Now comes the polyphase idea. Each row is either all even or all odd. The even rows (not columns as for C) combine intO Geyey. The odd rows combine into Gogg. Each of those is a constant-diagonal matrix (a time-invariant filter!). When they act separately and simultaneously, the process is more efficient: g(0) 82) g(0) al en |- s() 2G) a(l)130 Chapter 4 Filter Banks In this polyphase form, upsampling comes after the two filters. The odd phase is delayed before the even and odd outputs are combined: Goua This is the polyphase form in the time domain. Its transform is the polyphase form in the z- domain. We are using the second Noble Identity on each phase separately: G(t2) becomes [1 detayi(1 2 Coven |. (4.50) direct (t2)Geven(" contains an advance (denoted by z). No problem. Good causal polyphase ma- trices have anticausal inverses, as this one has. The main point is that H;' is a polynomial (FIR45 Lattice Structure 135 not IIR). Add one delay to the whole synthesis bank (multiply by z~') and it becomes causal: oe ewi@=5{ 2 2 | [: ili | (4.59) When you multiply out the factors in H(z), you see the possibilities: 1 -t 4,0) =| atez! b+dz ] a- cz7! b- dz"! (4.60) The top row holds the even and odd phases of the lowpass filter C: CQ) =a+be-! +27? +dz has phases a+ez™! and b+dz7. ‘The second row is the polyphase form of the highpass filter D: D(z) =a +627! —cz~* — dz? has phases a—cz~' and b-—dz7). We design these 4-tap filters by choosing a, b, c, d. They can have linear phase or orthogonality. Our 2 by 2 matrix can be symmetric or it can be orthogonal: Linear phase filters (symmetric-antisymmetric). Choose a = d and b = c. ‘The lowpass filter C has coefficients a, b, b, a. The highpass filter D is antisymmetric, with co- efficients a, b, —b, —a. For the polyphase matrix, just substitute d = a and c = b: -1 1 Linear phase ayo~| ote tee | a-bz' b—az"! 6b) How do you recognize the symmetry of C (or Ho) from its phases in the top row? One phase is the flip of the other phase. Then the whole filter a, b, b, a is the flip of itself, which makes it symmetric. ‘The synthesis filters are also linear phase. The underlying reason is that the inverse of a sym- metric matrix is symmetric. Now change to orthogonal. Orthogonal filter bank Choose d = a and c = —b and normalize. The second row [c d] = {—b a] becomes orthogonal to the first row [a 6]. Both rows and both columns have length /a? + 5. Each of the three factors in Hp(z) is a unitary matrix, after dividing by that length. The inverses come directly from the conjugate transposes: oo 1 a -b -b a ~ @tap| b a [eT Cay am We use the word unitary rather than orthogonal because z is complex. The inverse of a unitary matrix U is 0”. The first matrix is teal, so conjugating had no effect. The second is diagonal, so transposing had no effect. We always do both. Notice that |z| = 1. The inverse of z = e~! equals the conjugate: z~' = e" = Z. Sec- tion 5.1 will introduce the word paraunitary for H p(z), when it is a unitary matrix on the circle Iz] = 1. Our present example is paraunitary.10 Chapter 4 Filter Banks Multiplying the three matrix factors gives H(z), with d = a and c = —b: (4.62) —b-! lt Paraunitary polyphase matrix H(z) = } [ on ri fio | The impulse response shows the difference between linear phase (earlier) and orthogonal (now), The lowpass C(z) comes from the first row, where a and —b give the even coefficients. The coefficients in b + az! go with the odd powers z—! and z73: CQ) =a tbe! — bz 4 az, The highpass response from the second row is D(z) =a + be"! + bz? — az, ‘We lost symmetry and gained orthogonality. One part of orthogonality is that (a, b, —b, a) is perpendicular to (a, b, b, —a). But this is not all. Another part is that (a, b, —b, a, 0, 0) is per- pendicular to (0, 0, a, b, b, —a). We have to consider those double shifts, because they enter the time-domain matrix — two filters C and D with downsampling: H [ G2)c | (4.63) (2D 0 ie | a > Ad a Linear phase and also orthogonal: Not possible for length greater than 2. Later we prove this fact in general. Linear phase FIR pairs that are also orthogonal can have at most two nonzero coefficients in each filter. They are just variations on the average-difference pair, This is a one-step improvement over single filters, which can have only one nonzero coef- ficient. An allpass FIR filter is a delay, with one term C(z) = cz. Lattice Structures The orthogonal bank will now be more general than this example. It will have more coefficients. But H(z) can still be produced from simple factors. They will be cascades of constant matrices and diagonal matrices. The filter bank has a highly efficient lattice structure. We can sce already the form of our a, 6, ¢, d factor in the two important cases of linear phase and orthogonality:45 Lattice Structure 137 Linear Phase k ab 1 =k b =a with k= — eo k ba k 4 . 1 1 Orthogonal k ab 1 =k b a=d, b=-c cen blot ~ -k Ee! ee sa 1 Each lattice involves only one multiplication when implemented properly (Problem 7), There is also a single overall multiplying factor, collected from all factors in the cascade. The numbers k can be design parameters. For any k’s, a cascade of linear phase filters is linear phase. A cascade of orthogonal filters (including —k) is orthogonal. We now study a general lattice built from simple orthogonal factors. Note. The lattice structure gives long orthogonal filters very easily. They can be good filters. But the k’s enter H(z) in a complicated nonlinear way. For the design step, where filter char- acteristics are optimized, most engineers choose simpler parameters like the coefficients A(n). For the implementation step, the lattice has an important advantage. Often the coefficients h(n) will be “quantized” — real numbers are replaced by binary numbers (finitely many digits), In general, this destroys orthogonality. But orthogonality is not lost when implemented as a lat- tice of simple filters. Each orthogonal factor is determined by a real number k (or by an angle 8), When & or @ is quantized, the factor is not exactly correct — but it is still orthogonal. Therefore the whole filter bank remains strictly orthogonal. ‘We now introduce a product of retation matrices and delay matrices. The rotation matrices are constant, exactly as in our examples: cosé sind anak Re| sind cos@ ]=ee[ -k 1 | 4.64) This matrix gives clockwise rotation through the angle 6. When cos@ is factored out, it leaves the number k = tan@. We will think of the rotation matrix (orthogonal matrix) in terms of 6, but we implement it with & to save multiplications. The delay matrices are used to delay the second channel: A(z) | i a ] has determinant 2~'. It is convenient to have an extra factor of —1 in the lower channel, The matrix diag(1,-1), which is also A(—1), accomplishes this. Then the product of £+1 rotations separated by € delays is the polyphase matrix for the whole lattice: Hy(2) = A(-1) Re A(z) Re-1 A(z) +++ Ry A(2) Ro. (4.65)138 Chapter 4 Filter Banks The rotation Ry is through an angle ;. The rotation Ro is through an angle 0. The determi- nant is —z~¢, because of the delays and the sign change from diag(1, —1). Without the delays in between, the product would be a single rotation through the total angle > 6;. With the € de- lays, we have something much more important. It is essentially the most general two-channel orthonormal filter bank with ! delays. H,,(z) is the polyphase matrix of an orthonormal filter bank, because each factor is unitary. We show this analysis bank in its efficient form, with the downsampling operators before the filters. Ro comes first in the structure because it is the right-hand factor in Hp(z). x(n) Figure 4.4: The lattice structure for H,(z) = A(—1) Re A() --- Ry A(z) Ro. The factors cos @ that are removed give a multiplication by cos 4, - - - cos at the end. Note that Haar’s filter bank comes from one rotation with angle 4: if 1 te [saa COs @, sin@ with = V2L 1 -1 {> | sin@ -cose de Actually this is a picture of any polyphase matrix H)(z) at the particular value z = 1, The matrix H,,(1) corresponds to zero frequency, because e° = 1. The response to direct current is always the Haar matrix, when the lowpass filter has a zero at w = x. Theorem 4.6 = if Hy = 0. at w = x (which is z = —1), then the polyphase matrix has 1 1 al aoat{} 1], For an orthonormal filter bank the angles in the lattice structure add to 7. Proof: Hg has a zero at » = m when the sum of odd-numbered coefficients equals the sum of even-numbered coefficients: #(0) — A(1) + &(2) — ... = 0. This first sum rule means that at z = 1, the even phase equals the odd phase: Hoo(1) = h(0) + h(2) + +++ = A(1) +43) +++ = Hoi (1). (4.66) Row | of the polyphase matrix has equal entries at z = 1. Row 2 comes from the highpass filter A. This has the opposite property H; = 0 at w = 0. The odd sum is minus the even sum. With the right signs, H,(1) is the Haar matrix in the Theorem. Since this is exactly what we get in the product of rotations, when z = 1 and the delay ma- trices become I, the total rotation angle must be > 6; = §. The 4-tap Daubechies filter in Section 5.2 has angles } and —75. Those angles add to 3. Looking back at the start of this section, we realize that Haar followed by one rotation is actually45 Lattice Structure 139 useless. That second rotation angle must be zero! The sum rule in the top row of the lowpass filter becomes a + b = —b + a, which forces b = 0, The good angles are % and —%, not + and 0. The Synthesis Lattice The synthesis bank inverts the analysis bank. To invert the lattice, reverse the order of rotations. The inverse A(—1) Re A(z) ... Ri A(z) Ro is the product of inverses in reverse order: (Hp @)7) = REA @RT --- A"@RFAC-1). (4.67) The inverse of each rotation is the transpose: R~! = R7 = rotation through —@. These constant matrices are orthogonal. The inverse of a delay matrix is an advance: " wo=[p fc] ewo-[ 9 7] Since H,p(z) has € delays, its inverse has € advances. Multiplying each advance by z~! puts 1 delays into the upper channel, The synthesis half has upsamplers fast (this is the polyphase form in Figure 4.5). An extra advantage is that we can add or remove rotations without losing cos 6p ~sin % sing 4 cos 8p Figure 4.5: Orthogonal synthesis bank in lattice form. orthogonality, In contrast, the alteration of one filter coefficient (7) is almost certain to destroy orthogonality and also perfect reconstruction. What is the synthesis polyphase matrix? Lattice Coefficients from Filter Coeffi nts We now prove that any 2-channel orthonormal filter bank can be expressed in lattice form, with rotations and delays. Thus the lattice structure is “complete”. Starting with H(z) of degree é, we find a rotation-delay that reduces the degree to £ — 1: _[ cose sine Jf 1 0 ] ae» #0=| sine cos ] [ Ores! | @). (4.68) The new matrix H&~"(z) is still unitary for |z| = 1 and its determinant is +2~¢-, So we reduce the degree again. After £ steps we reach H,(0) whose determinant is +1. It is unitary for |z| = 1 and the only matrices with this property are constants. After properly accounting for the matrix A(—1) = diag(1, ~1), we have the rotation angle 4 that completes the lattice. Theorem 4.7 Every lowpass-highpass orthonormal filter bank has a polyphase matrix H)(z) of the lattice form (4.65).140 Chapter 4 Filter Banks Proof.. The step (4.68) requires an angle ¢ such that cos sine aie eee (€=1) [ —sin® cos®& | Ho =[ oe! |xs (z). (4.69) On both sides, the second row must have no constant terms. Then we can factor z~! from that row and the reduction succeeds — we get the next matrix H-"(z) and continue. If Hy(z) = hy) +--+ + hy (dz, the constant term on the left side of the equation comes from h p(0). The second row on that side must give [ -sin@ cos ]h,(0)=[ 0 0]. (4.70) This row vector that produces zero must exist if (0) is singular. The whole argument rests on the fact that A7 (d)h (0) is the zero matrix. The key equation is H, Hp = E: [AP O) +--+ AE @)z"] [hp (0) + ---+hp@)ze~4] = 1. 4.71) The coefficient ne (d) ky (0) of the highest order term must be zero, When two nonzero ma- trices multiply to give zero, #,(0) and &,(d) are both singular. There exists a vector [~sin@ cos 6] that knocks out the constant term in the second row of (4.69). After steps we reach a final rotation Ro and the lattice is complete. Lattices for Linear Phase Filters For orthonormal filters, Ho and H, have the same (even) length. One is symmetric and the other is antisymmetric. For linear phase filters, those statements are not necessarily true. The great variety of linear phase PR filter banks means a great variety of lattice structures. The equal-length case copies the orthonormal case, but with factors $ = | @ | instead of b a b r=[ 3 2] Theorem 4.8 = Every linear phase perfect reconstruction filter bank with equal (even) length filters has a lattice factorization a, =[ haan ]s:acos.ra@--SiNse, 4.7) The filter bank is a cascade of simple two-tap filters: Le: a & | 1 k& aw =[ 02 | and s-| oe |-4[ co |: The proof is entirely parallel to the orthogonal case (Theorem 4.7). The implementation just has a sign change. The number of lattice sections is half the length of Hp (and H). All factors a; are collected into a single factor a = J] a;, for efficiency. And the two multiplications by k; are further reduced to one multiplication (and three additions) in Problem 7. What is not parallel is that linear phase allows further possibilities: unequal length and symmetric-symmetric pairs. Here are the rules for PR with linear phase:45 Lattice Structure 141 Even length must differ by a multiple of 4: Ho = symm and H, = anti. Odd length must not differ by a multiple of 4: Hy = symm and H; = symm. An example of the second type is in the “Guide to the Book”, with lengths 5 and 3: 4(-1, 2, 6, 2, -1) in Ho and 4(1, -2, 1). For symmetric filters whose lengths differ by 2, the (complete) lattice factorization is L : a lez 1 a oO H(z) = D J]pa22An(z) with Ay(z) = [ 14 Ae ee? bee ] [ o 1 | 4) Dis diagonal, and a practical system can have only 82 4 0. This speeds up the implementation. The FBI 9/7 system is an example that requires only two lattice sections. The general case has a similar (but longer) factorization. The proof is also longer. Our pur- pose has been to establish the main point: the efficiency and the availability of the lattice struc- ture. Perfect Reconstruction with FIR ‘We are interested above all in FIR systems. Then both H, and F,, have a finite number of terms, from the finite number of filter coefficients. The crucial observation is that with well-chosen matrices, the product of two polynomials can be a constant or a monomial: a1 rv oi oe tp ue possible for matrix polynomials. In the scalar case, | + 8z~' is a polynomial but 1/(1 + 627") is not. The reciprocal of a polynomial scalar is not a polynomial. If an ordinary time-invariant filter is FIR, its inverse is IR. The only scalar exceptions are trivial cases (delays). The real exceptions are matrix poly- nomials — which are polyphase matrices of filter banks. That is what this book is about. The inverse of a matrix polynomial can be a matrix polynomial. We ask when. Theorem 4.9 An FIR analysis bank has an FIR synthesis bank that gives perfect reconstruc: tion if and only if the determinant of H p(z) is a monomial cz~! with c # 0. Proof. The entries of (H_p(z))~? are always cofactors divided by the determinant: _ G, i) cofactor of Hy(z) ~ “determinant of H, oz)” ay The cofactor (the determinant without row j and column i) is certainly a polynomial. If the determinant is cz~! (one term only!) then the division leaves a polynomial: the inverse is FIR. When this inverse H7,'(z) is delayed by 2~!, it becomes causal. This is F,(z). It is the Type 2 polyphase matrix of the FIR synthesis bank. The design problem is to maintain a monomial determinant cz~' while building an efficient filter bank. For an M by M matrix the cofactors of H,(z) may have high degrees. The analysis and synthesis filters generally have different lengths. The design problem is harder, but more is possible. Cosine-modulated filter banks are winners.142 Chapter 4 Filter Banks Efficiency of Lattice Structure Consider a symmetric filter bank with length N = 2J. The number of multiplications per unit time is 4/ using the direct form and 2J using polyphase. Additions are the same, with two extra at the 2-pt DFT matrix. On the other hand, there are J lattice sections. Each section requires 2 multiplications and 2 additions. Counting 6; and 2 in the figure brings the total complexity to 2/ multiplications and 2J + 2 additions. The effective complexity (at the input rate) is J multiplications and J +1 additions. Problem 7 discusses an alternative that uses 1 multiplier and 3 adders per lattice section. In summary, the lattice complexity is approximately half of the polyphase complexity. The same is true for orthogonal filters. 8, x(n) x(0) ‘ 42 Fiter with length 12}4 y 2). Total number y J lattice sections of coefficients 4 yee J2H Ba = Polyphase implementation Lattice Structure implementation Problem Set 4.5 1. Show that this matrix gives linear phase. Find a, b, c,d and H'(2z): H,(2) = cos@+z'sin@ -sin@+z27!cos@ POD= 7) cos@—zsin@ =sin@ ~ z-'cos@ 2. Suppose U and V are unitary matrices (constant but possibly complex). This means that u-' =U’ and V-! = V". Show why their product UV is also unitary. Notice the key point: Inverses come in reverse order V-'U—! and so do transposes. 3. For a 6-tap antisymmetric filter @ + b2~! + c2~? — c2~> — bz~4 — az, show that the flip of ‘one phase is minus the other phase. What if the number of taps is odd? 4, Redraw the lattice cascade with the downsampling operators moved to the right, after the but- terfly filters. Be sure to change z~' to z~? (why?). ‘ 5. Find the product H/p(z) when the rotation angles are @ = 0, 6, = 3, and @ = 0. Check the determinant of H,,(z), remembering the ¢ = 2 delays. This is an example in which H(z) contains no terms in z~¢, although the determinant has degree ¢. In the notation of Theorem 4.7,€=2butd=1. ‘The correct definition of degree of H(z) is the Smith-McMillan degree = minimum number of delays to realize the system. For orthonormal filter banks, this equals the degree 1 of the determinant. 6. Find the polyphase matrix H_p(z) for the pair Ho(z) = 1 and H(z) = } — 27) +4277 in Problem Set 4.3. Find the synthesis polyphase matrix Fp(z) for Fo(z) = $+ 27! + 427? and F\(z) = —1. Remember to transpose and reverse rows for F(z), and compute F’,(z)H ,(z). ™ We can multiply by S = 7 e | with one multiplication by ¢ = ash (and one by Sad that can be collected with others at the end of the cascade): si allo )it =)45 Lattice Structure 143 Write S with 3 adds using the figure. Find a “one multiplication per rotation R”. 1-k k pe. Y=k/(1-k) 2 multipliers 1 muttiplier 17k 2 adders 3 adders 8. Find the polyphase matrix Hf,(z) for the analysis pait Ho(z) = Land Hi(z) = 4(-1+ 92°? — 162-3 + 92-4 — 2-6). Find F,p(z), transposed and row reversed, for the synthesis pair Je (— 1492774 16279 492-4 2-8) and F(z) = 1. Verify that Fy(2)Hp(z) = 2-4. Land ¢ = 3. 9. What is the synthesis polyphase matrix F for the lattice structure in the last figure? 10. The rules for linear phase with PR were stated separately for even and odd lengths. Combine them into one rule for the degrees of Ho(z) and H(z): No + Ni +2 must be a multiple of 4. Change a given PR filter bank by choosing H(z) = H(z) and Fiy(z) = 2-?Hy(z). What is the relation between H’,(z) and H(z)? Is this new filter bank PR? 12. Find the analysis filters and the relation between Hy(z) and Hy(z): we-[27][e 2][4 3][. 2][4 7] =" 11. 13. Let Hp(z aa) : Jpoere (z)is a polynomial, Find H(z) and H(z). Find F(z) for a PR system. What are the synthesis filters? L -1 z 0 "i 14, Let, (2) = I [ fe) 1 iL Find F(z) and all the filters for a PR system. 15, What is the lowpass orthogonal filter with @, = $, 6 = —, and 6; = 4? 16. What are the symmetric even-length analysis filters with ky = 2, k, = 0,k3 = —7,andk, = 5? What are the PR synthesis filters? 17. If we impose orthogonality on the symmetric factors S; in (4.72), show that the filters only have two nonzero coefficients. This yields another proof of Theorem 5.3 that symmetry prevents orthogonality. 18. Show nat [ 14 Ie ‘ Js euivatent co [ eee Il ta ] This form of the last two blocks in (4.72) reduces the lattice computation time.Chapter 5 Orthogonal Filter Banks 5.1 Paraunitary Matrices For an orthogonal matrix, the inverse is the transpose. When the matrix is 2 by 2, this imposes a tremendous constraint. Suppose we choose just one entry of the matrix, in the upper left corner. Our choice should not exceed 1 in absolute value, and we call it cos 0: oa The other entries are almost completely determined by this choice. Below cos @ we need sin@ or —sin@, to make the first column a unit vector. The second column must be a unit vector orthogonal to the first one, and we select cos? = —sin® a= sind cos i; 6.) H rotates every plane vector x by @. The length is preserved: \\Hx|| = |[x]l. This is not the only orthogonal matrix that starts with cos@. We could multiply the second column or the second row by ~1. Then the rotation matrix in (5.1) becomes a reflection ma- trix; its determinant changes to —1. If these are complex matrices, we could multiply any col- umn and any row by numbers on the unit circle |z| = 1. This yields every unitary 2 by 2 ma- trix. Essentially, one entry determines the whole matrix. The inverse is the conjugate transpose: H' =F’. An analysis bank is represented by an infinite matrix (in the time domain). But in the fre- quency domain or z-domain, the matrix is 2 by 2 (from M = 2 channels and M = 2 phases). This matrix depends on a parameter w or z. Therefore we stretch the definition from unitary to paraunitary: Definition 5.1. The matrix H(z) is paraunitary if it is unitary for all |z| = 1: HT (e~/*)H(e!) (5.2) This extends to all z #0 by Ae = H(z"), Then paraunitary matrix has BH (")HQ) = A@ A(z) =f forall z. (5.3) When the coefficients h(k) are complex, they are conjugated in H(z).5.1 Paraunitary Matrices 145 The matrix H need not be 2 by 2. If itis 1 by 1, then |H(e/)| = 1. The corresponding filter is allpass. The best allpass examples are ratios of polynomials coming from IIR filters — since only trivial polynomials z~' can have |H(e/”)] = 1. If H(z) is M x M, it could come from an M-channel filter bank. It might be the polyphase matrix H(z) or the modulation matrix H,,(z) (divided by /2). We will show that the filter bank is orthogonal if these matrices are paraunitary. That is the important connection for this book. Equation (5.2) gives the inverse matrix by transposing and conjugating the original. The synthesis bank comes by “reversing” the analysis bank. Note that for a square matrix, H(z) is paraunitary when H(z) and H™ (z) and F(z) are paraunitary. And notice especially what equa- tion (5.3) says about the determinants of these matrices: (det (2) (det H(z) = 1. (5.4) The determinants are 1 by 1 allpass! Theorem 5.1 {fa square paraunitary matrix H(z) is FIR (= polynomial), then its determinant must be a delay: det H(z) = t27!. (5.5) The determinant of H(z) is also a delay for any bi-orthogonal filter bank. Orthogonality re- quires more; the polyphase matrix H)(z) must be paraunitary. If H(z) is rectangular, say M by r, then H@H(2) = Tis still possible. The identity matrix is now r by r (and necessarily r < M, since the rank cannot exceed M), The matrix is still called paraunitary. There is no inverse matrix H'(z) in the rectangular case, but A (z) is a left-inverse. ‘We hope and expect that H(z) can be completed to a square paraunitary matrix, by adding M—r more columns. In the applications, we are starting with r filters and creating M — r additional filters — while preserving orthogonality. When Fi(z)H(z) = di with d > 0, we could still use the word paraunitary. The chief exam- ple is the modulation matrix, which has d = 2. Some authors keep this flexibility. Our chief interest is in the case M = 2. A 2 by 2 paraunitary matrix is essentially determined by one entry. For a paraunitary polyphase matrix, the even phase Hoo(z) of the lowpass filter essentially determines the whole orthogonal filter bank. For filter banks with four taps, there are two free parameters — which can be ¢(0) and c(2), The design problem is greatly reduced when the filter bank is orthogonal and its polyphase matrix is paraunitary. 2 by 2 Paraunitary Examples The first example is H(z) = I. It corresponds to a “lazy filter bank” without filters. The poly- phase matrix is H, = I when the filters are Ho(z) = 1 and Hy(z) = z7'. The bank just splits the input vector x into odd and even phases, without filtering it. The next paraunitary matrix is H(z) = 3[} _}]- This is still constant. It is the polyphase matrix of the Haar average-difference filter bank. We move on to more interesting examples. Suppose Ro,..., R; are constant rotation matrices as in equation (5.1). The rotation angles are 6, ...,. If we multiply those matrices, we get a single rotation through the total angle 14;. But if we introduce a diagonal matrix A(z) = diag (1, 2~") between those rotations, we get something much more general and important: A(z) = Ry AQ) Ris A@)-+-Ri AG) Ro. (5.6)146 Chapter5 Orthogonal Filter Banks This matrix is paraunitary, because it is the product of paraunitary factors. Its determinant is Za (the product of determinants). The filter bank can be realized as a lattice structure involving / delays. Section 4.5 proved that this is the most general 2 by 2 paraunitary matrix of degree 1. The only subtle point is the meaning of the word “degree.” Here are two small examples to make that point, both with Ro = R2 = I. Then H(z) is A(z)R1 A(z). The only difference is in Ry: w= [' w]e[irt el-al+ 2] ee le | Both have determinant z~?, Both have degree / = 2! But the power z~?, which appears in the first, does not appear in the second, The correct definition of degree is the Smith-McMillan degree — the minimum number of delays required to realize the system. For paraunitary systems, but not for ail systems, the degree is revealed by the determinant. For lowpass-highpass filters with four taps, the even and odd phases have two terms each. If H(z) is paraunitary, it fits the general form (5.6) with / = 1. There are two parameters 6 and 6; to be chosen. Again we have two design parameters, for 4-tap orthogonal filters, but this time they are angles. ‘This chapter constructs a family of maxflat filters. Then Chapter 6 shows that these filters give the Daubechies wavelets, We mention the connection now, so you will attach importance to maxflat filters. They don’t have a sharp transition band — they don’t have the quickest transition from passband to stopband — but they produce the most vanishing moments. Problem Set 5.1 1. Suppose H(z) = (0) + A(z") + «+ + A(N)z~" is paraunitary with N > 0. Show that (0) A(N) = zero matrix. Deduce that 4(0) and &(N) are both singular matrices. 2. Multiply out H(z) = Ri A(z)Ro for angles 6, and 69. If this is a polyphase matrix, what filter coefficients d(K) come from the even and odd phases in the lower sow of H(z)? If that is a highpass filter, with D = 0 at @ = 0 (which is z = 1), show that 6, + 6) = —% + m7. (Haar has 8 = —% and 0, = ) 3. Complete this polynomial matrix to have det H(z) = 1. Is it paraunitary? Dice ] A= [ 4, When can a row be the first row of a paraunitary matrix? 5. Show that H = J—2vv" is aunitary matrix, where v7» = 1. Compute H~! and the determinant of H. Construct an example. 6. Show that H = I — pv? + z~'vv? is a paraunitary matrix for a unit-norm vector p. Compute. its inverse and determinant. The factorization of H,,(z) using these Householder matrices is in(V].5.2 Orthonormal Filter Banks 147 5.2 Orthonormal Filter Banks This section brings together the requirements for an orthonormal filter bank. We will see those requirements in the time domain and the polyphase domain and the modulation domain. These tequirements are conditions on the filter coefficients c(k) and d(k). Then equation (5.19) indi- cates a simple choice of the d’s coming from the c’s. If the lowpass filter meets the orthogonality requirements, it is easy to construct a highpass filter to go with it. The discussion is in terms of a 2-channel FIR filter bank, M = 2. But the conditions extend immediately to any M. The polyphase matrix and the modulation matrix must be paraunitary. Tn the M-channel case, the lowpass filter does not immediately determine the Mf — 1 remaining filters (which are bandpass). There is some freedom in their construction, and we come back in Chapter 9 to M > 2. c 12} t2H4 ct input x — A output x =x D 12} f2K5 dB Figure 5.1: An orthogonal filter bank has synthesis bank = transpose of analysis bank. Figure 5.1 shows the structure of an orthogonal filter bank. We intend to achieve ¥ = x, with synthesis filters C7 and DT that are time-reversals of the analysis filters: = C and Gn) =c(-n) 67) = D and din) =d(-n) (5.8) so As it stands, C and D are anticausal. At the end we make them causal by N delays. The output in) is equally delayed; it is x(x — N), But the algebra is easiest for C7 and D” with no delays. This special structure imposes special conditions on the c’s and d's for perfect reconstruction. We will call the requirements Condition O (for orthogonality). This section finds four equivalent forms: Condition O on the infinite matrix, on the lowpass coefficients, and on the polyphase and modulation matrices Hy and H,,. We look first at the infinite matrices in the time domain. Time Domain: Condition O and the Alternating Flip The key matrix in the time domain is H,. It represents the direct form of the analysis bank, with downsampling. The lowpass part L = (| 2)C comes above the highpass part B = (} 2)D. We display this infinite matrix for filters of length four: 3) ¢(2) (1) @) (3) ¢(2) e(1)_—e(0) L -[5 ]- 43) dQ) al) d@) eG dQ) dQ) dl) d(0) The shifts by 2 were created by downsampling, which removed the odd-numbered rows.148 Chapter 5 Orthogonal Filter Banks With orthogonality, the synthesis filters are to be time-teversals of the analysis filters. The infinite synthesis matrix contains the transposes of (J 2)C and (J 2)B. Those transposes are C7 (42) and D? (2), since upsampling is the transpose of downsampling: (3) 4(3) €Q) 42) =| eC) ¢@) dl) d@) Me en el co) aon woy | Cae ely - dl) - (0) - d() The shifts by 2 in the columns are created by upsampling, which removes every other column. We require ¥ = x. This means that H7H, = 1. The matrix H, is requited to be an orthogo- nat matrix. \ts columns are orthonormal and so are its rows: HyH? = I. We can express this Condition O in matrix form, and in block form, and in coefficient form. Condition O An orthogonal filter bank comes from an orthogonal matrix: YH, =1 and H,HT = 1. (5.11) Tn block form this means that (7 BT] [| =L'L+BB=I (5.12) - L ur it 10 T BT) a = [p]e w|=[ i i |-[3 - 6.13) For the coefficients e(k) and d(k), equation (5.13) becomes orthogonality to double shifts: LL’ =1: Ye(n)e(n — 2k) = 5k) (5.14) LB™=0: Yve(n)d(n— 2k) =0 (5.15) BB’ =1: Yd(n)d(n — 2k) = 6(). (5.16) Because of (5.14)-(5.16), we refer to Condition O as double-shift orthogonality. Its equivalent in the frequency domain is presented below, This double-shift orthogonality immediately rules out odd length filters! If the length is N + 1 = 5, a shift by 4 gives an inner product that cannot be zero: (€@), e(1), €(2), €(3), e(4)) - 0, 0,0, 0, (0) = e()e(4) #0. N cannot be even (the filter length cannot be odd) because g double shifts would give a shift by N — and the inner product c(0)e(N) is not zero. So N is odd. The degree 2N of the halfband product filter P is 2,6, 10, ... The clearest examples have N = 3 and 2N = 6. Example. Condition O in (5.14) imposes two constraints on four coefficients: (0)? + (1)? +. (2)? +.¢(3)? = 1 and c(0)e(2) + e(1)e(3) = 0. (5.17) Equation (5.16) is an identical condition on the d’s, from BB? = I. Equation (5.15) is the or- thogonality of the rows of L to the rows of B. Those are the fundamental design constraints on an orthogonal filter bank.5.2 Orthonormal Filter Banks 149 The conditions on the c’s and d’s are independent, but something good happens. If the c’s Satisfy equation (5.14), it is easy to choose the d's, We display a choice of d’s that automatically gives orthogonality: eG) (2) e(1)_— (0) 3) e(2) el) 0) L [5 ]- ~c(0) e(1) -e(2) €@) - 18) -c(0) e(1) -e(2) &(3) This is the alternating flip. The c’s are reversed in order and alternated in sign, to produce the d’s. We start with lowpass coefficients ¢(0), ...,¢(N), where N is odd. The highpass coefficients are - : (5.19) ‘The essential point can be checked by eye, in the infinite matrix (5.18). If the top rows are or- thogonal to each other, then all rows are orthogonal. The zero dot products in LB" are: —¢(3)e(0) + e(2)e(1) — e(1)e(2) + e(0)e(3) —e(1)e(0) + e(O)e(1) 0 0. (5.20) a Furthermore, the @’s are orthonormal within themselves (BB’ = 1) because the c’s are. Equa- tions (5.17) hold for the d’s, when they are constructed by flipping the c’s. Minus signs cancel in d(3)d(1) which is (—¢(0))(—e(2)). Our four-tap example has N = 3. The alternating flip gives LB’ = 0 for every odd N. ‘The top rows of H, in (5.18) are always orthogonal to the bottom rows. Also (5.16) for the d's follows from (5.14) for the c’s. Thus the alternating flip reduces orthogonality to (5.14): Condition O on the coefficients: Lewmea — 2k) = 6(k). Polyphase Domain: Condition O and the Alternating Flip The polyphase form separates (| 2)C and ({ 2)D into even phase and odd phase. In the time domain, we are rearranging the columns of the matrix H,. All even columns come before the odd columns. The matrix goes into the 2 by 2 block form of Section 4.4, with time-invariant filters as the blocks: Coven Coad ] : Deven Dosa In the z-domain, we are rearranging the response functions C(z) and D(z): Flock = Det = Covenl2-) + 27" Coga(2).. 5.21) Those phase responses are written Hyo(z) and Hoy(z) when C(z) is Ho(z). The highpass re- sponse D(z) = H(z) decomposes in the same way. The polyphase matrix is #,)=| Coren(2) — Coaa(2) ]-[ Hoo(2)— Hoi(2) |. Dewen() Daiat2) |= | Hil) Hutz) eee)150 Chapter 5 Orthogonal Filter Banks Now Condition O translates directly into a requirement on H,(z). A filter bank is orthogonal when its polyphase matrix is paraunitary: Hy (e~)H,(e!*) = T for all w and H)(z)Hp(z) =I for all 2, (5.23) The inverse of Hp(z) is the synthesis polyphase matrix. The matrix and its inverse can be mul- tiplied in either order —~ here is analysis times synthesis: Coren(z) Coase) 1[ Coven(2™') Deven(z7!) ] _ (5.24) Dewen(2) Deaatz) JL Cousle) Doss") : On the unit circle, where z~! is Z, row 1 times column 1 becomes © Coven 2)? + |Coaa(z)I? = 1 when fel = (5.25) ‘This is the essence of Condition O in the polyphase domain, For the example with four coefficients, it is helpful to multiply out equation (5.25): (€(O) + e(2)27") (€(0) + €(2)z) + (e(1) + €(3)27') (e(1) + €(3)z) = (0) + €(2)? + e(1)? + (3)? + fe(O)e(2) + e(1)e(3)] (2-! +z) = 1. Thus (5.25) is equivalent to the explicit statement (5.17). The sum of squares is I and the dot product e(O)e(2) + ¢(1)e(3) is zero. : ‘The other multiplications in (5.24) give answers 1 or 0 in the same way. All these require- ments on the @’s are automatically satisfied by the flip construction! We write that choice d(k) = (-1)te(W — &) in the z-domain: Dede = Pe w ~ by(-2)* = Peem(-ar". This relation between highpass and lowpass is an alternating flip: D(a) = (-2) *C(-z"!). (5.26) The number N is odd. Because of (—z)-%, the even and odd phases in C are reversed to odd and even phases in D. We take N = 3 as typical. An alternating flip of c(0), (1), ¢(2), €(3) yields D@ = e(3)—e(2)27! + ez — ez (6.27) (c@) + e(1)z-?) - 27! (€(2) + e(@)z7?). With this flip in row 2, the multiplication Hy (2)H7 (2) becomes (0) + ¢(2)z7* e(t) +¢(3)27? e(0) + ¢(2)z ¢3)+e(I)z ]_ 1 (3) -Fe(1)z“!— -e(2) - e@)zt e(1)+eG)z -e(2)-e@z J The off-diagonal entries of the product are automatically zero. The alternating flip achieves LB" = 0 with or without orthogonality. The 2,2 entry of the product is the same as the 1, 1 entry, when {z| = 1. The orthogonality requirement (5.25) makes the 1, 1 entry equal to 1.§.2 Orthonormal Filter Banks 151 Modulation Domain: Condition O and the Alternating Flip ‘The function that arises from modulating C(z) is C(—z). The frequency in z = e/” changes by 7 to produce —z = e/°+*), This modulation takes C(@) to C(w +7). The frequency response graph is shifted by 7. Our goal is to relate C(z) to C(—z) and D(z) to D(—z) for an orthogonal filter bank. We already know Condition O on the coefficients: (0)? +.¢(1)? +€(2)? +¢3)? = 1 and e(O)e(2) + c(1)e(3) = Watch how 1 and 0 appear in |C(z)/?. Stay on the unit circle where Z-! = z: (€(O) +.e(1)27! + €(2)27? + €(3)2~) (0) + e(1)z t+ €(2)2? + €(3)27) = 1+ [e(O)e(1) + e(1)e(2) + ¢(2)e(3)] (e7! + z) + e(O)e(3) (2-3 + 2’). Now change z to —z. The odd powers z and z? change sign. When we add, those odd powers cancel: coe + Ie -)te ICl@)l? 4 + Ictw + m= =2 forall z= ei. (5.28) This is the haifband condition, also called the Nyquist condition: IC@ Pi isa ‘palfband filter. Those filters we can design! In other words, the lowpass analysis filter C(z) is a spectral factor of a halfband filter, Condition O on H(z). The modulation matrix of an orthogonal filter bank is a paraunitary matrix times 2: An (2)Hn (2) = 20 for alll z. (5.29) On the circle z = e/, the modulation matrix is a unitary matrix times V2: | 28 Ctr) [2 De) |- a Hb Dw) Dw+z) || C@rm Dor 0 2 6.30) The 1, 1 entry of this matrix product is [C(@)|? +|C(@ + 2)? = 2 by (5.28). The other entries, when we multiply them out, follow immediately from (5.15) and (5.16). Thus Condition O on the coefficients is equivalent to Condition O on the modulation matrix H,,. It is also equivalent to Condition O on Hp. It is the statement that the analysis bank followed by its transpose gives perfect reconstruction. We summarize: Theorem 5.2 For an orthogonal filter bank the lowpass coefficients must satisfy Condition O (we give four equivalent forms). Matrix form LL = (}. 2)CC7(t 2) =1 Coefficient form. Le(nde(n ~ 2k) = 5(k) Polyphase form [Ceven(e#?)? + ICoaate)?? = 1 Modulation form = |C(w)/? + [Cw +m)? =2 By an alternating flip, LB” = 0 and BB’ = J follow immediately from the lowpass part LL" =1. The real problem is the design of the lowpass filter.152 Chapter Orthogonal Filter Banks Symmetry Prevents Orthogonality Itis natural to want two good properties at once, Symmetry is good for the eye, and orthogonality is good for the algorithm. But the only filters with both properties are averaging filters (Haar filters) with two coefficients. We are forced to use IIR filters, or M channels, or multifilters with matrix coefficients (Section 7.5). Extra computation is unavoidable, because the next theorem rules out a perfect filter. Theorem 5.3 A symmetric orthogonal FIR filter can only have two nonzero coefficients. Proof. N is odd for orthogonality. The filter length must be even. With N = 5 a symmetric filter of length 6 has the form (c(0), ¢(1), e(2), e(2), e(1), e(0)). This vector must be orthogonal to all its double shifts. The inner product with its shift by four must be 2¢(0)e(1) = 0. Therefore e(1) = 0. Then the inner product with its shift by nvo gives 2 c(O)e(2) = 0. The only nonzero coefficient is ¢(0) at both ends of the filter. This completes the proof. By convention c(0) is the first nonzero coefficient. Shift the filter if necessary to achieve this. The only symmetric orthogonal possibilities are ¢ = (1, 1)//2 and (1,0, 0, 1)/2 and (,0,...,0, 1/72. Only the Haar coefficients (1, 1)//2 will lead to orthogonal wavelets. Symmetry really conflicts with orthogonality. A second proof observes that the odd phase is the flip of the even phase: (€(O), (4), (2), (2), €(4), c(0)) has [Ceven(z)I? = ICoua(z) I. Condition O is |Ceven(z)|? + |Coaa(z)|? = 2. With symmetry this separates into |Ceven(z)|? = 1 and |Coag(z)I? = 1. The even phase is an allpass filter | So is the odd phase. But FIR allpass filters can only have one nonzero coefficient, which completes the second proof. A third proof is based on the zeros of C(z). This is in Section 5.4 below. Problem Set 5.2 1. (a) Show that the alternating flip with odd N gives Dla) = —e"*C(w +m). () Then D@ +) = e°C(w). Verify that C(w) D@) + Cw +2)Dl@ Fm) = 0. (©) Also |C(@) |? + |C(@ + 2)? = 2 implies that |D(w)? + [D(w +m)? =2 2. For any four coefficients A(0), ..., #(3), verify that Veen? + [Hou = 4 (IH@)P + 1H (2). Then Condition O for polyphase equals Condition O for modulation. 3. Find the flaw in this construction of the modulation matrix H(z). Start with an arbitrary upper left entry C(z). Complete the 2 by 2 matrix to be paraunitary (times /2), ‘Then the filter bank is orthogonal. 4, Find d by an alternating flip of ¢ = (c(0), ...,.¢(5)). Verify equation (5.15) directly to show that ¢ is double-shift orthogonal to d. 5. Verify thate = 25 (1 + 73,3 + /3,3 — V3, 1 — V3) satisfies Condition © (Daubechies). 6. If two lowpass filters € and H satisfy Condition O, does their product satisfy Condition 0? 7, If two polyphase matrices H,(z) and Kp(z) satisfy Condition © (they are paraunitary), does their product satisfy Condition 0?5.3 Halfband Filters 153 8. Iftwo modulation matrices H,,(z) and K,,(z) satisfy Condition O, show that Hy(2Kn(2)/V2 is also paraunitary. What is the lowpass filter in the product? 9. Why does orthogonality require an alternating flip between the lowpass filter C’ and the high- pass filter D? Explain why the pataunitary matrix =] Cowal) — Coase) Dewalt = [CauC2dl 4,0)=[ Doveo(2) Doartz) | ™ Pipe = CoenlDh Go further to show that Deven = 2~* (flip of Coyy) and Dogg = £277 (flip Of —Ceven). This gives the alternating flip with any even delay 2-7". 40, Find Ho(z) and H,(z) and H,,(z). Is this paraunitary? Find the PR synthesis filters. What is Fy)? x(n) U. For H,(z)=1—w" +27? withy=1[ 2-1 1 3 J’, find the four PR filters. 12, Let H(z) = Ry A(z)Ro where & = % and @ = —. What are the analysis and synthesis filters? Plot the frequency responses of H,(z). 5.3 Halfband Filters Out of all the equations in the previous section, we would like to emphasize one. It came at the end, It applied first of all to the lowpass filter C(z), and then by the alternating flip also to D(z). It was equation (5.28), that the frequency response C(w) = > c(k)e~/* satisfies IC(@)P + IC + xP =2. (6.31) ‘The key question is, what does equation (5.31) say about |C(w)|? itself? We assign the symbol P(w) to this important quantity [C(@)|. It is the power spectral re- sponse. Because C(w) multiplies C(@), the filter with this response P(@) is symmetric. It is the “autocorrelation filter”: N N é N Pow) =D rpinyeni™ = (Yretkye™) (D> etkye”). (5.32) —N 0 0 The function P(w) = |C(w)}? is real and nonnegative. It equals its complex conjugate. This verifies the symmetry p(n) = p(—n) that was expected (with real coefficients), To repeat: When " c(k)e~/** multiplies its conjugate F e(!)e/”, we watch forn =k —1 which is ! = k ~ n. The coefficient p(n) is the sum of e(k) times e(k —n): p(n) = Liewee — n) = autocorrelation of the sequence c(k). (5.33) Autocorrelation is p = ¢ * ¢7. This is the convolution of ¢ = (€O), e(1), ¢(2), ...) with its time reversal e7 = (...,¢(2),¢(1), ¢(0)). Replacing —n by x in (5.33) brings no change in Bp.154 Chapter 5 Orthogonal Filter Banks The reason for the surprising notation ¢” is that multiplying C(@) by C(@) corresponds ex- actly to multiplying the infinite filter matrix C by C7: (2) E s : e(1) s P=CCT = : (0) (0) : : (5.34) 7 (i) . 7 (2) P is a symmetric positive definite (or semidefinite) Toeplitz matrix. The transpose of CCT is CC". The diagonal p(0) = ¢(0)? + e(1)? + ¢(2)? + - -- is certainly positive. Equation (5.31) says that p(0) = 1, when the matrix C comes from an orthonormal filter bank. What does equation (5.31) say about the other coefficients p(n)? In a word, it says nothing about the odd coefficients and it assigns zero to the even coefficients. C is the start of an orthog- onal filter bank if and only if the autocorrelation filter P is a halfband filter: P(z) + P(—z) =2. (5.35) The even coefficients with n = 2m must be Sm): 1 ifm=0 Halfband filter p(2m) = Se(k)e(k -— | 0 itm #0. (5.36) The odd coefficients are not ey zero! They: cancel ‘automatically i in equation (5.35). To require CC’ = , with odd coefficients zero, would make C an allpass filter. It could only be a delay. The requirement P(z) + P(—z) = 2 is much weaker than P(z) = 1. Perfect reconstruc- tion is possible for an FIR filter bank, but essentially impossible for one FIR filter. The highpass response D(w) leads similarly to the autocorrelation P;(w) = {D(@)?. Then DD? must also be a normalized halfband filter. That result is automatic with the alternating flip, which gives P\(w) = P(@ +2). Then p, (2m) = p(2m) = 6(m) for the even coefficients. The odd coefficients change sign, P|(z) = P(—z), but the halfband condition P,(z) + P\(—z) =2 remains true. The sum |C()/? + |C(w + 2)? is P(w) + P(w + 7). For a halfband filter, this sum is a constant. The graph of P(w) shows a special symmetry with respect to the halfband frequency w = %—hence the name. Notice what happens in downsampling — the even coefficients yield the identity filter: (12)P =I when P is normalized halfband. (6.37) Example 5.1. ‘The symmetric filter with p(0) = 1 and p(1) = p(-1) = 5 is a halfband filter. Its response is P(z) = 1+ 2 and equivalently P(w) = 1 + cosw. In the z-domain, P(z) + P(—z) = 2. The odd powers cancel and there is no 2” term. In the @- domain | + cosw + 1 — cos = 2. The odd frequency cancels. Notice that P(w) = 1+ cosw5.3 Halfband Filters 155 is never negative! It does reach zero at the highest frequency @ = 2, corresponding to z = —1. When P(w) touches zero, its spectral factor C(w) must also touch zero— since P = |C|*. This leads us to C(w) = (1 + e™)//2: P(w) = |C()? = (1+ 7) (1+e) /2= 1+ cosa. (5.38) These coefficients ¢(0) = e(1) = uv come from the familiar averaging filter. The division by V2 gives an orthonormal filter. Figure 5.2 shows the lowpass halfband filter P with response 1+ cos@. Added to P(w + 7) it gives the constant 2. | Pca] |cco| 005 01 O16 G2 025 09 025 04 045 O85 Figure 5.2: The orthonormal filter C(w) has P(w) = |C(w)|?. The normalized frequency 0.5 is w=n, The requirement P(w) > 0 is crucial. Otherwise we could not factor P(«w) into [C(w)|?. The halfband filter with coefficients p(—1) = p(0) = p(1) = 1 could never be |C(w)}?. Its response P(w) = e“” + 1 +e! is negative at w = 7. Example 5.2. The Daubechies 4-tap filter picks out C(w) from P = |C?? when P(@) = (1+ cosw)? (i ~ $cosw). (5.39) Note the double zero at @ = 2, coming from (1 + cos w)?. If we keep only that factor, this P(«) would be the square of the previous example. Its factor would be the square of the previous C(w), namely $ +e“ + 4e-%, Those coefficients 5, 1, $ are important —they will lead to the hat function, when we study wavelets. But P(@) = (1 + cos)” does not by itself yield a halfband filter, so the lowpass C' with coefficients 3, 1, $ cannot go into an orthogonal filter bank. The hat function is not orthogonal to its translates. To repeat: (1 + cos)” includes the term cos w. This produces an even frequency cos 2w. In the z-domain we are squaring 1 + $ (z~' + z), which produces the even power 2~?. This is not halfband! Daubechies’ extra factor | — 4 cos w must be included, to cancel the 2w term in P(w) and the z? term in P(z). We will have orthogonality, thanks to that factor. To see that P(w) is halfband, multiply it out. The cos 2« term is missing: P(@) = (1+ 2cosw + cos w) (1 — }cos@) = 1+ Fcosw — }cos*w.156 Chapter 5 Orthogonal Filter Banks In the z-domain, the z? term is missing. Its coefficient p(2) is zero: PQ) =—ho + Zet lt Zot ke. (5.40) We know that P(w) = 0 because 1 — $cosw > 0. Therefore it can be factored into |C(w)| (spectral factorization). This is not a trivial calculation. It is made easier by the fact that we already know the factor for 1 + cos, from the first example. This leaves only the linear piece Q(w) = 1 — } cose or Oz) = 1 - $ (c~! +2). We factor Q in three steps: ~$@' +2) = 6O+427) GO +00 (6.1) 1=8(0)+(1)? and —$ = 5(0)b(1) (5.42) Solving the quadratic equations gives b(0) = (1 + V3)/V8 and &(1) = (1 — V3)/V%. The solutions are real because 1 — } cos w is safely positive. Another approach, basically the same, is to multiply (5.41) by z to get an ordinary quadratic. The quadratic formula gives its roots as 2 + V3. Since Q = 0 at these roots, we have BO) + (1) (2 + V3)"! =0. (5.43) The previous solution is correct because | + /3-+(1—/3)(2— V3)~! = 0. This is the minimum phase solution, from the root 2 — 3 inside the unit circle. There is another solution from 2 + 3, in which b(0) and b(1) are exchanged. This is maximum phase. Two more solutions come from reversing signs to —b(0) and —b(1). We are seeing the limited number of possible spectral factors in |C(w)|*. The general rule for higher degree polynomials and longer filters is the same: Minimum phase: — Choose roots of 2 P(z) that are on or inside |z| = 1. Maximum phase: Choose roots of 2” P(z) that are on or outside |z| = 1. Mixed phase is also possible, choosing some roots inside and some outside, That can bring us to linear phase. We will show how linear phase factors of the Daubechies polynomials lead to biorthogonal filter banks which are among the current favorites. Completion of Example. We factored 1 — }cosw and 1 + cosw. Multiply to obtain ce = gy ite") (04+ v3+00- vac") (5.44) = aA +9) + 6+ VBe1 + - Ve +1 - VIE" ], Those are the four coefficients c(0),...,c(3) of the famous Daubechies filter D4. In our present normalization, they are divided by 32 = 44/2. In other normalizations they are divided by 4. Remember their two key properties: 1. The halfband filter has P(z) + P(—z) = 2. The factor C(z) goes into an orthonormal filter bank. D(z) comes from C(z) by an alternating flip. 2. The response C(z) has a double zero at z = —1. In frequency, C(w) has a double zero at w = 7. The response is flat at m because of (1 + cosw)’. The double zero at w = 2 will produce two vanishing moments for the Daubechies wavelets in Chapter 6.5.4 Spectral Factorization 187 0 01 02 03 04 0s Figure 5.3: A halfband filter P (w) and its mirror image P(w +). Their sum is constant. Figure 5.3 shows a graph of |C(w){*, so you can see the halfband Property that produces orthogonality. ‘The highpass response in that figure is |C(w + 2)|?, which equals |D(w)|2. The sum of the two is constant, so there is no amplitude distortion. The flatness gives great accuracy hear @ = 0 and w = x (not so great in the middle). The filter bank gives perfect reconstruction. Problem Set 5.3 1. Solve equations (5.42) for x = 6(0)*. Confirm the factorization. 2. Whatis the 2 by 2 polyphase matrix. H(z) from the Daubechies C(z) and the alternating flip? 3. What is the 2 by 2 modulation matrix H,,(z) for that four-tap Daubechies example? Verify that Ay H, = 2. 4. If a linear phase halfband filter satisfies G(z) + G(—z) = z and N? Can G(z) be an antisymmetric? + What is the relation between / 5.4 Spectral Factorization Inan orthonormal filter bank, C(z) = /2H (2) is a spectral factor of a symmetric halfband filter P(z). The factorization is P(z) = C(z~')C(z) and the halfband property is P(z) + P(~z) =2. In frequency, P(w) = |C(w)|? achieves the orthogonality condition |C(co)[?+ IC@+z)P =2. In the reverse direction, P(z) is the autocorrelation of C(z). This intimate relation of spectral factor C(z) and its autocorrelation P(z) is fundamental throughout signal processing. ‘Two questions arise immediately: 1. (Theory) Can every polynomial with P(«w) > 0 be factored into |C(w)[?? 2. (Practice) How is this spectral factorization actually done? The answer to Question 1 is yes. This is the Féjer-Riesz Theorem. The answer to Question 2 is not so quick. There are many competing algorithms for spectral factorization. Short filters offer158 Chapter 5 Orthogonal Filter Banks no serious difficulty, but with 100 or even 50 coefficients the weaker algorithms become slow and/or unreliable. When C(w) is only approximate, the reconstruction is not perfect. The trigonometric polynomials P(w) and C(w) are both of degree N: N ; : H 2 Vereen = 1c(e!*y? = [etme —N 0 P(z) has symmetric coefficients p(n) = p(—n). There are N + 1 independent coefficients in P and the same number in C, They are linked by quadratic equations, when we solve P(w) = tC(@)/?. Those equations are solvable if and only if P(w) > 0 for all w. As an aside, note that matrix spectral factorization is also possible where P(w) is symmet- tic positive definite, Both 1 and 2, theory and practice, are nontrivial. The Riccati equation is involved. We indicate four factorization methods. Three are actually used; Method C is for conversa- tion only. The first method begins by finding the zeros of a polynomial (by a good algorithm!). This proves that spectral factorization is possible, by doing it. Method A (zeros of a polynomial), With real symmetric coefficients p(n), we have P(z) = P(1/2). Ifz isa root, so is 1/z;. When z; is inside the unit circle, 1 /z; is outside. The roots zon the unit circle must have even multiplicity, by the crucial assumption that P(w) > 0. Therefore the polynomial 2” P(z) of degree 2, with leading coefficient p(N) % 0, must have these 2V factors: NM Me 2" P@) =p) |] @- 2) (c- +) TI @-2)- (5.45) i=l cadear This contains the key point, but we know more. Real coefficients ensure that the complex con- jugate Z is a root when z is a root. The complex roots off the unit circle actually come four at a time: 2 and Z; inside, 1/z; and 1/2; outside. The complex roots on the circle also come four at atime: z, twice and Z; twice. Real roots on the circle come two at a time (even multiplicity). Now construct C(z) by taking aif the roots z; (including Z;) inside the circle, and also take ‘one out of every double root z; on the circle: Me NM Cw) = pl? ] @-2) [] &-a)- (5.46) il j=l This is the “minimum phase spectral factor.” It has no roots outside the circle. The coefficients of C(z) are still real, because the complex roots are automatically in conjugate pairs: Z; and Z) came with z; and z;. Example 5.3. The 4-tap Daubechies filter in the previous section led to zeros at z, = 2 — V3 and z7' = 2+ 3. The other four roots of 2 P(z) are at z; = —I (on the unit circle and again teal). Two of those roots go into the spectral factor (5.46). Thus z>C(z) is a cubic polynomial with roots 2 — 3, —1, and —1. It is minimum phase. Every factorization of P(z) into F(z) H(z) must put some roots into H(z) and the remaining roots into F(z). The rules for this separation of roots of P(z) are: « For F and H to be real filters, z and Z must stay together. For F and H to be symmetric filters, z and z~! must stay together.5.4 Spectral Factorization 159 For F to be the transpose of H, z and z~' must go separately. This is the spectral factor- ization C(z) C(z~") that gives an orthogonal filter bank when P is halfband, Figure 5.4a shows a partition of the zeros into circles and squares that makes both factors symmetric. The splitting in Figure 5.4b makes one factor the transpose (coefficients reversed) of the other factor. To achieve both properties at the same time, all zeros of P(z) ~ not just the zeros on the unit circle — must be of even multiplicity. We now show that this is impossible for a halfband filter. This gives another proof of Theorem 5.3, that orthogonality conflicts with sym- metry. Each factor is symmetric © factor = transpose of 0 Figure 5.4: Twelve zeros of a halfband filter separate into analysis (©) and synthesis (CD. Theorem 5.4 A symmetric orthogonal FIR C(z) can only have two nonzero coefficients. Proof. An FIR filter is symmetric when z” C(z) = C(z7'). If z; is a zero of C(2), so is z7'. Then P(z) = C(z~")C(z) has a double root at z;. More precisely, all roots of the polynomial 2% P{z) have even multiplicity. This polynomial is a perfect square [R(z)]. if the filter is also orthogonal, P(z) must be halfband: 2" P(z) = [r(0) +++» +r(N)z* Phas only one odd power z*, The first odd power in R(z) produces (when it multiplies r(0)) an odd power in [R(z)]*. The last even power in R(z) also produces (when it multiplies r(N)z” ) an odd power in [R(z)]?. But our halfband filter has only one odd power. We cannot allow any even powers or any odd powers in the terms - -- indicated by the three dots, The polynomial R(z) has only two terms and P(z) has three terms. Then C(z) only has two nonzero coefficients. All symmetric orthogonal FIR filters have C(z) = (1-+2~")/-/2 with odd N. The halfband product filter is P(z) = 32" +14 427%. Example5.4. For symmetric filters, the roots z; = 2—/3 and z7! = 2+/3 must stay together when we factor P(z). The four roots at z = —I can be split between H(z) and F(z). One sym- metric splitting is HZ) = (1 + z7!)-1 + 427! — 279)/2V2 and FQ) = (1+27")3/4,/2. There are several symmetric factorizations, but none of them can be orthogonal.160 Chapter 5 Orthogonal Filter Banks Now we return to computation of zeros of polynomials. For long filters, a good algorithm is needed to find the zeros z; and z; of P(z). We quote from the 1994 abstract by Lang and Frenzel [La,Fr]: Finding polynomial roots rapidly and accurately is an important problem in many areas of signal processing. We use Miller's method for computing a root of the deflated polynomial. This esti- mate is improved by applying Newton’s method to the original polynomial, Furthermore we give a simple approach to improve the accuracy for spectral factorization when there are double roots on the unit circle. Milller’s method uses three previous estimates of the root of z P(z) to find the next estimate. ‘The parabola that interpolates at the three old points has a root at the new point. Since parabolas can have complex roots, Miiller’s algorithm can find complex roots from a real start — while Newton can only move chaotically on the real line. Newton’s method uses the most recent estimate z,. For real roots, the tangent line at z, to the graph of 2% P(z) crosses zero at the new point zz 41. This is the outstanding method for solv- ing nonlinear equations, provided zo is close enough — which is the task of Miiller’s method. The polynomial 2° — 1 was one of the tests (not the only one!). The code is on ftp from cml-rice.edu under directory pub/sofiware. MATLAB uses an eigenvalue method. Its subroutine roots is effective up to quite large degree. The roots of a polynomial z” +--+ are the eigenvalues of its N x N companion matrix, which has I’s down a diagonal and minus the polynomial coefficients along a row, For example, z> — 222 — 5z ~- 9 is specified by the vector y = (1 —2 —5 —9]. ThecommandM = compan(v) Dee oee@: produces the matrix =| 1 0 0 | with det(2z¥ —M) = 2? ~ 227 — 5z—9. MATLAB O20 finds the eigenvalues (by the QR method) which are the roots of the polynomial. ‘The next section mentions how rescaling y to 4y allowed us to compute Daubechies filters of twice the length achievable without this scaling. The coefficients in P(z) were better controlled. Linear phase filters with extremely good stopband attenuation have many zeros on or near the unit circle. These are the hardest zeros to compute. Method B {solve quadratic equations). We are looking for N + 1 numbers c(0),...,¢(N). The N + I equations are of second degree, involving c’s times c’s. The equations come from matching powers of e in C(@)C(w) = P(w): (o ctkye™) (Sethe) = Spine, (5.47) ° ° WW One way to make those equations explicit is in matrix form: 20) e¢(1) ¢2) + e(N) ¢(0) pO) ¢(0) e(1) ¢(2) ; (1) p(l) e(0) ¢(1) (2) e2) |=] p) |. (5.48) ©) e(1) : : (0) e(N) P(N) The first equation is ¢(0)? + --- + e(N)? = p(0). This gives the constant term in (5.47). The second equation gives the e~ term. The last equation is c(O)e(N)e*® = p(N) efN@,5.4 Spectral Factorization 161 Equation (5.48) is not a linear system! It is quadratic, in fact homogeneous of degree 2. It has a real solution if and only if P(w) > 0 for all w. This is not an easy condition to verify on the coefficients p(n). If P(w) = 0 is not true—in which case our solution methods must fail —we can add enough to the DC term p(0) to make it true. For orthogonality, the p(n) come from a halfband filter. The even coefficients p(2), p(4), -.- are all zero. But our discussion is not in any way limited to this halfband case. Spectral factorization applies to all filters with P(w) > 0. It even applies to IIR filters, but those lead to infinitely many equations. Example 5.5. The previous section factored P(w) = 1 — 4 (ef + e!). Comparing coeffi- cients of 1 and e~' led us to 1 e® 1) ][ @ ])_f 1 id 0 Il e(!) -[ -3 ]: 2 (0)? + (1)? e(O)e(1) This is our system (5.48) for that particular example with N = 1. Eliminating ¢(1) gave a single quadratic equation. The unknown was x = (0)? and the equation was x + {= = 1 or 16x? — 16x + 1 = 0. For N > 1 we cannot reduce the N + 1 quadratic equations to a single equation for ¢(0), An approximate solution by method A, B, C, or D (or another method E) is the best we can expect. To use a nonlinear equation solver, write the kth quadratic equation as } TF O(k)e = p(k). The symmetric matrix Q(k) has 1’s along its kth subdiagonal and superdiagonal (and Q(0) = 22). Here is Q(2) with N = 3: oo 10 (0) $[ e@) (1) ¢@) ¢@) J : H ot =pQ). (5.50) 0 100 re) This is c(O)e(2) + e(1)e(3) = p(2) from matching the e~%* terms in |C(w)|? = P(w). The partial derivatives of that left side L(2) are in the gradient vector O(2)e: aL(2)/acO) = (2) 001 0 (0) aL(2)/ae(l) = ¢(3) : 000 1 (1) aL@yec@) = eo SSH) 1 g 9 9 || e@) aLQ)/ae3) = ell) 0 1 0 0 4L ce) The kth quadratic function is L(k) = eT Q(k)e, and its gradient is O(k)e. The second derivatives are also desired by nonlinear subroutines, and also readily avail- able. They are in the constant matrix Q(k). One successful program using gradients and sec- ond derivatives has been the Quadratic Constrained Least Squares (QCLS) optimization code by anonymous ftp from eceserv0.ece.wisc.edu under the directory pub/nguyen/software/QCLS. We use it in Chapter 9 to design M-band filter banks. Method C (matrix factorization). Equation (5.48) is an attractive form for the quadratic equa- tions. But there is a more symmetric form. If you like infinite matrices, you will enjoy this. Instead of a finite matrix times a vector, it has an infinite constant-diagonal matrix C times its162 Chapter5 Orthogonal Fitter Banks transpose: (0) e) el)» e(N) Palle, a Tc. e(0) cl) + e(N) : wle ce 2) (= ew) [POP A Do PP oe e(N) - The columns of P have entries p(—N), ..., p(0), ...,, P(N). In this matrix form of |C(@)? = P(w), the symmetric matrix P factors into upper triangular C™ times lower triangular C. This is only possible if P is positive semi-definite — which is exactly our condition P(w) > 0. Matrices are decomposed into triangular factors every day. This is the matrix statement of ordinary Gaussian elimination. The factorization is usually written A = LU. It gives lower tri- angular times upper triangular. Fortunately, infinite constant-diagonal matrices commute; our equation is also CC? = P. The harder problem is to factor infinite matrices in finite time. Approximate method: take a finite section of P. Keep R rows and columns, where R is larger (perhaps much larger) than N. This finite piece Pp is still symmetric and positive defi- nite. Therefore Px can be factored into CgC, where Cp is lower triangular. That is the Cholesky factorization of Pg. It is a symmetrized form of A = LU, available because Pp is symmetric positive definite. The finite matrix Cx does not contain the exact c(k). It is not even true that Cr has constant diagonals (although Pz has). The reduction to a finite matrix has chopped the tail ends of the row-column multiplications, either in C7C or in CC™. But the rows of the computed factor Cz do approach the rows of C. The correct (minimum-phase) coefficients ¢(k) appear in the limit as R ~> 00, We demonstrate with P(w) = 1 ~ feoseo. This Daubechies example was solved exactly fore) = (1+ V3)/-V8 = 0.9659 and c(1) = (1 — V3)/V8 = —0.2588. Take R = 4 and use chol in MATLAB to factor Pg into CC?: 1-025 025 1° 025 ie 025 1 ~o25 | amd 025 1 1 cu| 7025 0.9682 = 0.2582 0.9661 —0.2588 0.9659 This matrix has ¢(0) and ¢(1) correct to four places in the last row. But with long filters this method is very slow. Method D (Cepstral method: Take logarithms). The idea is to convert the multiplication P(z) = C(z™')C() into addition. Formally, log (5 p(n)z~*) is easily separated into positive and negative powers of z. The symmetry p(n) = p(—n) and the positivity P(w) > 0 yield a logarithm L(z) with coefficients In) = [(—n): log P(z) = ) nz”5.4 Spectral Factorization 163 These are infinite series. The logarithm of a polynomial is not a polynomial. This means that our finite computations can only be approximate. The easy separation into log C(z~!) + log C(z) is the key advantage of the method. The sequence i(n) is the complex cepstrum of p(n), The series for Lz) converges in an annulus |z;| < |z| < 1/|z;| of the complex plane. Here z; is the largest root of P(z) inside the unit circle, and 1/z; is the smallest root outside. (The method is in trouble with roots z; on the circle. Best to remove those first. Otherwise L(z) = log P(z) will be infinite at z; and the series cannot converge.) The computation of the ¢’s requires an inverse Fourier transform of log P(z). Then the ¢’s are computed from the /’s. A detailed treatment of the cepstrum {(m) is given by Oppenheim and Schafer [OS]. The con- stant terms give e(0) = exp HH). The next term ¢(1) is interesting because the 2~! terms only involve £(0) and #(1). The recursion for n = 1, 2, ... turns out to be o(n) = Knc(0) + Un — Ne(1) +--+ Lie(n — 1). We need only NV +- 1 coefficients /(0), ...,4(N) in the logarithm to find all N + 1 coefficients (0), ..., e(N) in the spectral factor. To find those i(n) from the given p(n), we use a large-size FFT in the z-domain, A typical size is 8N, for acceptable accuracy. This cepstral method does not compute zeros of polynomials. So it doesn’t find symmet- Tic filters. It is a good way to find orthogonal filters. A code is available by anonymous ftp at eceserv0.ece.wisc.edu under directory pub/nguyen/software/CEPSTRAL. Very optional comment. Spectral factorization also solves singly infinite constant-diagonat systems. (This is the genuine Toeplitz problem. Doubly infinite matrices could be named after Laurent — but mostly we still say Toeplitz.) The coefficient matrix P, has entries p(i — j) only fori > Oand j > 0: PO) pl) Pp) - x(0) 50) Pax, =, is PO) p@) pil) - x) |_| bd) | 651) p2) ply pO) - x(2) 52) This corresponds in continuous time t > 0 to a Wiener-Hopf integral equation: oo f p(s — 1)x(1) dt = b(s) for s > 0. (5.52) o P,, does not have constant-diagonal factors in P, = LU. Lower triangular times upper destroys the time-invariant pattern. (Starting at time zero is responsible.) The beautiful Wiener-Hopf idea is that upper times lower succeeds perfectly: e(0) e(1) (2) - e(0) P,=ClC, = 20) ) + |] cl) at e(0) - e(2) e(1} (0) Wiener and Hopf computed this spectral factorization of Py by Method D, One of Norbert Wiener’s great theorems is that 37 |{(1)| < oo when }[p(n)| < 00. With no zeros of P(z) on the unit circle, he could take the logarithm even for IIR filters. The solution is x4 = (Pi 1b = (Cy CI). (5.54)164 Chapter 5 Orthogonal Filter Banks The inverses of C, and ct are constant-diagonal. So Wiener-Hopf can compute the spectral factorization P(w) = |C(w)|* and transform back to the time domain. Finite constant-diagonal matrices don’t have constant-diagonal factors and spectral factor- ization no longer succeeds. Nevertheless Pyxw = by can be solved quickly by the Levinson algorithm or a “superfast” algorithm or by preconditioned conjugate gradients (ChSt). We added these comments because transform methods are so central to signal processing. This is the whole underpinning of filter theory. Problem Set 5.4 - Suppose P(z) has six zeros at z = —1 and four other real zeros at z =a, a-', b, b-', Draw the complex plane and indicate which zeros go into the minimum phase spectral factor C(z). 2. For the same ten zeros, indicate a set of zeros that produces a symmetric (linear phase) C(z). Also indicate a second possibility. 3. Why must all roots of P(z) on the unit circle have even multiplicity, to allow P(z) = C(z) x Ce!) and Pw) = ICP? 4. The coefficients of the Daubechies polynomials C(z) up to order 12 are tabulated at the end of this chapter. Find the zeros using roots in MATLAB or another algorithm. What were the roots of the halfband polynomial P(z)? Show that P(z) = 2-"C(z)C(z~") must be symmetric. If C is lowpass with passband and stopband cutoff frequencies a», and w, and errors 8, and 6,, what are the cutoff frequencies and the errors of P? - 5.5 Maxflat (Daubechies) Filters This section is about an important family of filters, which will lead to an outstanding family of wavelets. The same construction yields both. Wavelets come from filters with special properties. Historically, their close relation was not immediately seen — now it is the subject of Chapter 6. The importance of this special construction is in its combination of two key Properties: 1. These particular filters (and wavelets) are orthogonal. 2. The frequency responses have maximum flatness at w = 0 and w = x. The lowpass filters will have p = 1,2,3,4,... zeros at 7, They have 2p = 2,4,6,8,... coefficients, so that N = 2p — 1. We use boldface p for the coefficients of P(w) = |C(w)|? and lightface p to count the zeros of C(w) at = x. The highpass coefficients d(k) come from an alternating flip. The first member of this family was the subject of Chapter 1: c(0) = e(1) = 1/2. Note the normalization c(0)? + e(1)? = 1. These numbers go into a unitary matrix. For each p = 1,2,3,4,... the filter bank is orthonormal. The product filters have degree 2N = 4p —2: lez 2 y Po(z) = ( ) ” Qop-2(z) will be halfband by special choice of Q. In the literature on filters, this family is described as maxflat. The coefficients were given by [Herrmann]. They were already in formulas for interpolation, described below. In the history5.5 Maxflat (Daubechies) Filters 165, of wavelets, we are reproducing the great 1988 discovery by Ingrid Daubechies. The filters are FIR with 2p coefficients. The wavelets are supported on the interval (0, V] = (0, 2p — 1). As P increases, the filters are increasingly “regular” and the wavelets are increasingly “smooth.” This section concentrates on filter properties and coefficients. We give a simple derivation of P(z), and new facts about its zeros. The next chapters will concentrate on wavelets and the step into continuous time. Condition O and Condition A, Before starting, it is helpful to count the requirements we must impose. There are 2p numbers to bechosen. These can be the coefficients ¢(0), ...,¢(2p— 1) in the lowpass filter, with frequency response C(«w). They could equally well be the coefficients p(0), ..., p(2p — 1) of the centered (even) polynomial P(w) = |C(w)|*. ‘The e’s come from the p's by spectral factorization. The nonnegative polynomial P(w) is factored by the methods of the previous section: 2p-1 : 2p-1 7 P(o) = Y° pinde™* equals [C(w)? = | > e(nyerinal (5.55) ip 0 Our formulas yield the numbers p(n) = p(—n). Except for the first few filters in the family, there are no simple formulas for ¢(7). These 2p numbers are determined by p conditions for orthogonality from Condition O, and Pconditions for a flat response from Condition A. More precisely, the requirement is “Condition A,” —the subscript indicates the order of flatness at @ = m (and w = 0). Here are the p+ p conditions: Condition =P =([C/* is a normalized halfband filter: “ pO) = 1 and p(2) = p(4 = p(2p —2) =0. (5.56) Condition Ap — C(w) has a zero of order p at w = 21: w= crn) =0. (5.57) The equation C(x) = 0 says that }°c(n)(—1)" = 0, The odd-numbered coefficients have the same sum as the even-numbered coefficients: Condition Ay one(n): JS" e(n) = > e(n). (5.58) oddn evena ‘This is the first of the “sum rules.” Altogether we can impose the pth order zero in (5.57) as p sum rules on the coefficients: =I Condition A, on e(n): . (-1)"nte(n) =0 fork =0,1,. -aprl (5.59) n=0 The factor n* comes from the kth derivative of > ¢(n)e"*. Then (—1)" comes from substitut- ing w = a. The convention for n° is 1.166 Chapter 5 Orthogonal Filter Banks Note on C(O) = V2: The sum rule (5.58) also applies to the coefficients p(n), because P() = |C(@)/* also vanishes at @ = 7. The odd sum must be I, since the only nonzero even- numbered coefficient is p(0) = 1: Lew = Yo p@) =p =1. (5.60) oddn event The sum over all n is P(O) = 2. Then P(w) = |C(w)|? yields C(0) = +V2. We always choose the plus sign for a lowpass filter, so the DC term at w = 0 is not reversed in sign: Yew = co) = /PO = v2. (5.61) alla The p zeros at 1 mean that C (co) has a factor (1 + e**)?: 1+e@ 2 R(w) has degree p — 1, to bring the total degree of C(w) to 2p — 1. You could say that the pth order flatness is accounted for by (1 + e~/)”. Then the p coefficients in R(w) are chosen to satisfy the p equations of Condition O. To repeat: p equations for orthogonality and p equations for flatness. Condition O is ap- plied to P(w); it must be halfband. Condition A, is applied to C(w); it must have the fac- tor (1 + e7)?. This is easily converted to a condition on P(w) = [C(w)|?, when we use [1 +e)? /2 = (1 +c0sw): Condition Ay on Cw): Cw) = ( Rw). (5.62) (5.63) 1 Condition Ap on P(w): P(w) has a factor (ASsy’. 2 Formulas for P(w) We intend to give two formulas for P(w) = |C(@) |. The one associated with Ingrid Daubechies has (1 + cos @)? times a sum of p terms. The formula associated with Yves Meyer gives the derivative of P(w) as —c(sinw)*?-!. Then integration determines ¢ and P(w). The best starting point is the ordinary polynomial B,(y). This has degree p — 1, with p coefficients. It is the binomial series for (1 — y)~?, truncated after p terms: By(y) = 1+ py + perda eet are) P= (1—y)? + O(y"). (5.64) The coefficient of y* is ?*£-"). The remainder has order y? because this is the first term to be dropped. The complex zeros of this polynomial B,(y) will be all-important for the Daubechies filters, ‘We combine B,(y) with the factor (1 — y)? that has p zeros at y = 1. The variable y on 0, 1] will correspond to the frequency « on (0, xr]. The product P(y) = 2(1 — y)?B,(y) has exactly the flatness we want at y = 0: 2(1 = yy Bp(y) = 2(1 — y)P((L — y)? + O(y?)} = 2+ OLy?). (3.65)5.5 Maxflat (Daubechies) Filters 167 This is a polynomial of degree 2p — 1. It is the unique polynomial with 2p coefficients that satisfies p conditions at each endpoint: Biyy and its first p — | derivatives are zero at y = Qand y = 1, except P(0) =2. Two more properties fotlow quickly. First, the derivative has p — 1 zeros at both end points. It is a polynomial of degree 2p — 2 and with those zeros it must be P'(y) = —Cy?""(1 = y)P-! for some C. (5.66) The second property comes when we add P(y) to P(1 — y). The sum equals 2 at both ends and is still flat. Its 2p coefficients are uniquely determined — it must be the constant polynomial 2: Py) + Py) =2. (5.67) Aty = 4 this gives P(4) = 1. Figure 5.5 shows how P(y) is odd around its middle value. This “Hermite interpolating polynomial” drops from 2 to 0 with flatness at the ends. Here are the polynomials for p = 2 and p = 3: Bo{y) = 1+ 2y and P(y) =2(1 - y)*(1 +2y) = 2-6? +4y? By(y) = 1+3y + 6y? and PCy) = 2(1 — y)?Ba(y) = 2 — 20y? + 30y4 — 12y5. SS 2 ° nef 6 Figure 5.5: P(y) on the left and P(w) on the right, for p = 2 and p =3. Now we go from ordinary polynomials in y to trigonometric polynomials in . The degree stays at 2p — 1, The change that takes 0 < y < lintoO ( \eSsy. (5.69)168 Chapter 5 Orthogonal Filter Banks This satisfies Conditions O and Ap. Its Meyer form, by integrating P'(w) and choosing c to give P(x) =0, is . oe f (sina)? “de. (5.70) 0 For p = 1,2, 3 the Daubechies and Meyer forms are Pw@)= 1+cosw=2- fl sinwdw P@)= (1+cosw)*(1 — }cosw) = 2 — 3 ff’ sin? wdw P@)= (1+ cosw)>(1 — 3 cose + 3cos*@) = 2~ !5 f’ sin’ ode Most authors emphasize the Daubechies form, with its highly visible factor (1 + cos w)?. That immediately ensures a pth order zero for the factors at = x, Spectral factorization is speeded up, because only a lower-degree polynomial remains. It may not be so clear that (5.69) is a half- band filter. The even powers like cos? w and cos‘ w must disappear and they do. In the explicit formula for p = 2, multiplication produces P(w) = 1+ }.cosw — } cos? w. The halfband property is P(w) + P(w +7) = 2. This addition cancels the odd powers of cosw, and the even powers are not present (except the constant term 1). This identity follows immediately from (5.67) because | ~ y = 1#s0se =, 1-sonloen)- By) + PC — y) =2 becomes P(w)+ P(o+) = 2. (5.71) The reader recognizes this “Condition O” as |C(w)/? + |C(w + x) |? = 2. The halfband property is immediate in the Meyer form, with absolutely no calculations. Re- place y by (1 — cos w)/2 in (5.66) to find P’(w) dw: = ~ —1 =cyP" — yyP-ldy = -c(! Sosy! ¢: tose sing ae (6.72) This is —c(1 — cos w)?-' sin w dw, which is also —c(sin w)??-! dw. Its integral is -f {1 - cos? )! sin» dw = odd powers of cosa. The only even frequency is a constant of integration. The filter is halfband. The flatness condition requires first of all that P(t) = 0. The constant c makes this true. The derivative P’(w) = —c(sin w)??-' has a zero of order 2p — 1 atw = x. Then P itself has a zero of order 2p. Its factor C has a zero of order p. Condition Ap is satisfied and Meyer's formula is confirmed. Note that P(w) decreases monotonically from P(0) = 2 to P(t) = 0. Its derivative ~c(sin «w)*?-" is everywhere negative between 0 and 7. There are no ripples in Figure 5.5. Therefore P(w) > 0 for all «w, and a factorization into |C(w)/ is assured, The transition from passband (low frequencies) to stopband thigh frequencies) — steeper and sharper as p increases. The slope at the midpoint w = % is ~c (sin £) PP" which is —c. We will show that c increases asymptotically like ./p as p — oo. Thus the transition band has width of order 1/,/P.§.5 Maxflat (Daubechies) Filters 169 The Halfband Filter P(z) Now we change from y and @ to the complex variable z. This will produce the filter coeffi- cients in P(z). That polynomial will be halfband and centered. The shifted polynomial Po(z) = 2 P(z) = z'~??P(z) will be halfband and causal. The change of variables comes from zeae: =cosw = 1—2y. (5.73) 2 = 0 and w = 0 give z = 1. Similarly y = | and» = givez = —1. Thus y that the midpoints y = 4 and w = 5 give z = +i. There are two 2's for each y, fom t+z7! = 2—4y. (This is a quadratic equation for z.) One z is inside the unit circle, the other is I/z outside, This “Joukowski transformation” is also central in fluid flow. The endpoints z = 1 and z = —1 are really double roots of z+ z~! = 2 andz+z7) = -2. The change of variable gives 1 — y and y in factored form: I-y= Ltgese a) (4) a ia = cso eye 1 ). 6.74) Substituting in Poy), the maxflat filter in the z-domain becomes P(z): ee pa = (#2) (Ey sa Ve (SEY. oa = This factors into P(z) = C(z)C(z~!) when P(w) factors into }C(w)|?. The p zeros at y = 1 and w = m are now 2p zeros at z = —1. Half of them go into C(z). The p — 1 complex zeros of the other factor Bp(y) become 2p — 2 zeros of P(z). Half of those (the p — 1 zeros inside the circle |z| = 1, if we want minimum phase) also go into C(z). So the spectral factor C(z) can be computed in two steps: 1. Find the p — I zeros of B,(y) and the p — 1 corresponding z’s with |z| < 1, 2. Include p zeros at z = —1. Then C(z) has these 2p — 1 zeros. Example. p = 2 leading to Daubechies Dg from B2(y) = 1 + 2y, which is $(-z +4-274). The zero is at y = —4. Therefore z+z~! = 4. This quadratic equation has roots z = 24/3. Then the 2p ~ 1 roots of C(z) are —1, —1,2 — V3. The coefficients of Dg are approximately 0.4830, 0.8365, 0.2241, and —0.1294: CQ =a +21 V0 - @- V3!) =[C4V5 +6 + VI 4 B- VIF + VDE] / AVE Example. p = 70 leading to Daubechies Dj 49 from Byo(y). From p = 2to p = 70is quite a jump! Figure 5.6 displays the 69 zeros of B7o(y). They are close to a limiting curve in the complex y-plane. The equation [4y(1 — y)] = 1 of that curve is discussed Below. In the z-plane the limiting curve is moon-shaped, with the beautiful formula le-z"!| = 2. It consists of two circles! The roots of the minimum phase factor C(z) are close170 Chapter 5 Orthogonal Filter Banks Im(z] im [yl \z-a=42 Wl=4 a Figure 5.6: The 138 zeros of P(z) and the zeros of Bp(y) up to p = 60. to the inner circle, To those p — 1 = 69 inner zeros we add p = 70 zeros at z = —1. Then the spectral factor of P(z) has 139 roots, and the constant 6 makes C(1) = v2: C@ =B ¢: ss 3 ne -2Z;). (5.76) t Same Coefficients in Interpolation The coefficients in — 4 + 32-2 +273 + 22-4 — kz appear in many places. This is a typical example, with p = 2, of a maxflat halfband filter. Its factor (1 + z~') yields 2p = 4 zeros at z = —I. The halfband property means that P is an interpolating filter: Px keeps the even- numbered coefficients of x = (...,x(0), 0, x(2),0,...). Four zeros at 7 mean that the four polynomials 1, 2, 7, 1° are correctly interpolated in the odd components: gives § Pe=(....1,1.1,11, gives Pr=(...,0,1,2,3.4, gives Px 0,1,4,9, 16, gives 0, 1,8, 27, 64 Expressed differently, % (e(1)+x(—1)) — 7g (3) +4(—3)) is fourth-order accurate at the mid- point ¢ = 0. This links wavelet theory to recursive subdivision (interpolation to create smooth curves). Starting with equally-spaced values x(2n), P puts new values at the halfway points 1 =n. Then P produces new values at t = 3. The established values do not change. The limit is a smooth curve through the original values. The monograph [CDM] develops this important application, with references. The stability of recursive interpolation is controlled by Condition E applied to P.5.5 Maxtlat (Daubechies) Filters 171 Asymptotics of the Daubechies Filters Figure 5.6 practically requires us to study the zeros Y and Z as p — oo. The first steps were taken by (LeKa] and [ShSt}. The truncated binomial series Bp(y) has degree p — 1. Its p — 1 zeros yield 2p — 2 zeros in the z-plane, from Z + Z~! = 2 — 4Y. The main facts proved so far are: 1, All the zeros have |¥! < } and Re Z > 0. 2. In the y-plane, the zeros are all outside the limiting curve jw| = |4y(1 — y)| = 1. 3. In the z-plane, the zeros are all outside the limiting curve |z — z~){| = 2. 4, The zeros are near a uniform distribution along the circle of radius 1+ log (4xp)/2p in the w-plane. 5. The far left zero is Z =i — W/ /p — iW?/2p + O(p~*/) where erf(W) = 1. Note that if z lies on that moon-shaped limiting curve |z — z~!| = 2, so do Z and z~! and Z The complex roots in the z-plane come four at a time, for finite p and in the limit p = oo. The moon consists of two circles of radius V2 (Problem ). The outer circle is |z — 1] = V2 with center at 1, The inner circle is (z + 1] = 2 with center at —1. The circles meet at z = ti, and the far left zeros go slowly toward these two points. Figure 5.6 shows the zeros in the y-plane up to p = 60. Their approach to the limiting curve is fascinating. The application of long filters, with large p, is still to be developed. It can use the lattice structure of Section 4.5 Computing the Spectral Factor C(z) For large p, computing the zeros of Bp(y) and the spectral factorization P(z) = C(z)C(z"') are challenging tasks. They are related but not identical. After we find the zeros, the filter co- efficients in C(z) are fully determined — but not necessarily in a well-conditioned way. The direct multiplication of 69 or 139 linear factors is not safe, The cepstral method (Section 5.4) is competitive because it goes directly to C(z), without the zeros. It is based on splitting log P(z) into log C(z)+ log C (21). Our experience with the zeros, using MATLAB and also Lang’s code from Rice, showed the importance of a simple weighting: The zeros of Bp(y) were correct up to p = 34. The codes fail for p = 35. With weighted variable 4y, the zeros became correct up to p = 80. The reason for the breakup at p = 35 is the wide dynamic range of coefficients in Bp(y). The constant term is 1. The highest term y?~! has coefficient Qp-2! | V2n@p=D @p-2yr? gre! (P-DIP-DI~ In D (p—D? ~ Yap—p by Stirling’s formula. The leading term 4°~! multiplying y?~' suggests that 4y is a better vari- able than y. This was strongly confirmed by experiment. The recursion (5.77) b(p) =1; forp—1:1:1 b@)=bG+1)*Qp-i-D/4a(p—-i172 Chapter 5 Orthogonal Filter Banks produces the coefficients in MATLAB order for the command Y = reots (b)/4. The zeros are needed in a linear phase factorization Po(z) = Ho(z) Fo(z), which does not come from the cepstral method. In this case all four zeros z,Z,2~', 27! go into the same fac- tor (two from inside the unit circle, two from outside). One possibility is to put those quartets alternately in analysis and synthesis, Ho and Fo, to give filters of nearly equal length. Both get Pp zeros at» = m. Another possibility is for Ho(z) to be very short, like Haar. Then Fo(z) comes from an exact division Po(z)/ Ho(z). No zeros are needed! Experiment will show which linear-phase factors give the best compression of signals and images. Transition Band for Maxflat Filters ‘The equiripple filters from the Remez-Parks-McCleilan algorithm have a sharp transition from lowpass to highpass. Their transition band has width of order +. Their slope at the midpoint w = 5 is of order N. They minimize the maximum error, giving the best pointwise approximation to 1 in the passband and 0 in the stopband. But they only have one zero (at most) at w = 7. The Daubechies filters, with many zeros at , have no ripples. P(w) and |C(w){ decrease monotonically between 0 and 2. We now show that the slope at » = 3 is much smaller, of order VN instead of N = 2p — 1. Theorem 5.6 The maxflat filter has center slope proportional to /N. The transition from P(w) = 0.98 to P(@) = 0.02 is over an interval of length 4/./N. Proof. The constant c in Meyer’s form is fixed by c fy (sino) dw = 2. This definite integral is known to be a ratio of Gamma functions (which are factorials (n+ 1) = n!), We use Stirling’s formula to estimate the integral: FED = a(MG)* Gt (FO a The slope of P(w) at w = 5 is-—c ~ — ‘x in Meyer’s form. The transition bandwidth is therefore O(1//N) and we make this more precise. Between 2 — Gq and & + Jy the drop in P(@) is the integral of c(sinw)%. Shift by ¥ to center the integral, replacing sin — @) by cos: aid ey Type drop = (cosw)"d =f 1-2 dox |= f “249. (5.78 aE es El ow) ce oy Here 6 = wJ/N. Thus 95% of the drop in P(w) comes with o = 2 (within two standard de- viations of the mean, for the normal distribution in statistics). This transition interval has width Aw = 4//N, as the theorem predicts. That rule was found experimentally by Kaiser and Reed in 1977, at the beginning of the triumph of digital filters. Problem Set 5.5 1. The halfband filter P(w) = 1+ D p(n)e~"® (odd n only) satisfies P(r) = 0. Deduce directly that P(0) = 2. 2. Find the zeros of B3(y) and the Daubechies 6-tap filter Dg.5.5 Maxflat (Daubechies) Filters 173 3. The definite integral ff (sinw)" dw equals 5135" for odd N. Express this in factorials and use Stirling's formula to rederive the estimate /2%/7N. Then the slope at the center frequency Eis O(N). 4. The points z = 1 + V2e'® are on the circle |¢ — 1] = /2. Substitute for z to show that this circle is one part of our limiting curve:Chapter 6 Multiresolution 6.1 The Idea of Multiresolution Our main approach to wavelets is through 2-channel filter banks. Everything develops from the filter coefficients. All constructions are concrete and highly explicit. Choose good coefficients and you get good wavelets. The heart of the theory is to see how conditions on the numbers A(K) and e(k) and d(k) determine properties of @(t) and w(t)—the scaling function and the basic wavelet. Then the problem is to design filters that achieve those properties. By iterating the filter bank, Section 6.2 reaches the dilation equation for @(t) and the wavelet equation for w(t). Sections 6.3 and 6.4 study those equations in the time domain and frequency domain. Conditions O and A lead to orthogonality and approximation accuracy. The Daubechies wavelets are “optimal” with respect to those two properties. But these orthogonal wavelets are not and cannot be symmetric (except for Haar). Also the transition from passband to stopband is not sharp. So the design problem is still open. Better wavelets remain to be constructed. This opening section aims for an overview that brings out the key ideas. Before the con- struction using discrete time, we describe what is wanted in continuous time. The goal is a de- composition of the whole function space into subspaces. That implies a decomposition of each function — there is a piece of f(t) in each subspace. Those pieces (or projections) give finer and finer details of f(¢). The signal is “resolved” at scales At = 1, 1/2,..., (1/2)/, For audio signals, these scales are essentially octaves. They represent higher and higher fre- quencies. For images and indeed for all signals, the simultaneous appearance of multiple scales is known as multiresolution. Multiresolution will be described first for subspaces V; and W;. The scaling spaces V, are increasing. The wavelet space W; is the difference between V; and Vj41. The sum of V; and W; is Vj41. Then these extra conditions involving dilation to 2t and translation to t — k define a genuine multiresolution: If f(4) isin V, then f(¢) and f(2t) and all f(¢ —k) and fOr &) are aves In the end, one wavelet generates a whole basis. The functions w(2/1 —k) come by dilation and translation (all j and all k). There are six steps toward this goal, and we take them one at a time: 1. An increasing sequence of subspaces V; (complete in L?)6.1 The Idea of Multiresolution 175 2. The wavelet subspace W; that gives Vj + Wy = Vj4t 3. The dilation requirement from f(t) in V; to f(2t) in Visi 4, The basis ¢(t — k) for Vo and w(t — k) for Wo 5. The basis 6(2/t — k) for V; and w(2/t ~ k) for Ww; 6. The basis of all wavelets w(2/t — k) for the whole space L?. A shortcut to multiresolution. Before those six steps, may I mention one shortcut step that starts with the filter coefficients h(k). That step is to solve the dilation equation for the scaling function $(t): HE mimi $(t) =D 2A(&) o(2r -k). The first requirements on the coefficients are 7 h(k) = | and (-1)' h(k) = 0. The full tequirement is Condition E in Section 7.2. When this is satisfied, #(¢) can be computed. Then {$(2/t — k)} isa basis for V,. These spaces are automatically increasing and complete and shift- invariant and connected by dilation. Thus multiresolution is achieved. A Scale of Subspaces Each V; is contained in the next subspace V;4,. A function in one subspace is in all the higher (finer) subspaces: VeCV CCV CV Cee A function f(¢) in the whole space has a piece in each subspace. Those pieces contain more and more of the full information in f(t). The piece in V; is f;(t). One requirement on the sequence of subspaces is completeness: fi > fO as fj > oO, The first example will not have the dilation feature required for multiresolution: Example 6.1. V; contains all trigonometric polynomials of degree < j. Certainly V; is contained in Vj,1. The spaces are growing. (Since Daubechies uses — Jj where we use j, her subspaces are decreasing. Most authors now use an increasing sequence, for simpler numbering.) The piece of f(r) in V; is the partial sum f;(¢) of its Fourier series: F(t) = Y cee! is the piece in V;. wis; This is the projection of f(t) onto V;. The exponentials e* are orthogonal, so the energy in Jj(t) is the sum of |cx|? over low frequencies |k| < j. The energy in f(t) — fj(t) is the sum over high frequencies |k| > j. This approaches zero as j > oo. Therefore the sequence V; is complete in the whole 27-periodic space L?. Now we identify the second family of subspaces. Wy contains the new information AFi(0) = fj41(t) — f)(). This is the “detail” at level j. From the viewpoint of individual functions, ’ SLO + AGO = fin. § 6.1)6.1 The Idea of Multiresolution 177 With orthogonality of each piece f;(t) to the next detail A f;(2), these subspaces are orthogonal. But we emphasize now that orthogonality is not essential. A nonorthogonal example comes directly from any nonorthogonal basis Bo(t), bi(t), ... The piece f;(t) includes all the terms through 5;(¢): Sum up to j: fit) = Vo cede) isin Vy Next term : Afj(t) = cjyibjy1(t) isin Wj. ‘The pattern is not lost, just the orthogonality. The new space V;4, is still the “direct sum" of V; and W;, which intersect only at the zero vector. The angle between subspaces can be less than 90°, as long as every fj41 in Vj41 has exactly one splitting into f; + Af;: Vj AW; = {0} and Vj +W; = Via” (66) This nonorthogonal situation applies to biorthogonal filters and wavelets (Section 6.5). There W, is orthogonal to a different subspace V). The extra freedom can be put to good use. Figure 6.1: An orthogonal sum and a direct sum. Both written V;@W, = V;,1 and both allowed. The Dilatlon Requirement So far we have an increasing and complete scale of spaces. Each V;, is contained in the next V,41. For multiresolution, the crucial word scale carries an additional meaning. V;41 consists of all rescaled functions in V;: Dilation: f(thisinV; <=> f(2t)isin Vj41. ‘The graph of f(2r) changes twice as fast as the graph of f(t). On a map, the scale is doubled. At3,000,000 :1 the state of Utah fills a page. At 6,000,000 :1 its height is a half page. The length that represents a mile is cut in half. This length is At or Ax or h. The example using f(t) = c_je" + --- + cje'/* does not meet this rescaling requirement. The highest frequency only increases by one, between V; and V;,;. But when ¢ is changed to 27, the highest frequency becomes 2j. The frequencies must double. The new space V;4, isrequired to contain all those new frequencies. To satisfy the scaling requirement, the partial sums go an octave at atime. The sum for f; should stop at frequency 2/ instead of j. Then Af; comains all frequencies between 2/ and 2/+': Multiresolution example : fit) = Yoege* for lk < 2/ Nextdetail: Afj(t) = Yicgel™ for 2/ < fkl < 2/+1.178 Chapter 6 Multiresolution This is a genuine multiresolution, in which V; and W, have roughly the same dimension. It is the Littlewood-Paley decomposition of a Fourier series, into octaves instead of single terms. This is a chief part of the mathematical background. To fit the requirements precisely, when F(t) is defined on the whole line oo < ¢ < 00, we should use all frequencies w and not just integers: sal Flw)el, SO= im f Swede. Now the spaces V; go down the scale toward j = —oo, as well as up the scale. The continuous frequency w can be halved as well as doubled. The basis functions become sinc functions, by the sampling theorem. Continuous frequency but discrete basis, as is normal for L?. And the nested spaces include j < 0: CV CVC CV CV Cee (6.7) In addition to completeness as j > 00, we require emptiness as j > —oo: V; ={0) and V; = whole space. (6.8) d i Emptiness means that || f(t)|| -> 0 as j —oo. Completeness still means that f)(t) > f(¢) as j > 00. The detail Af; = fj+1 — f; belongs to W, and we still have ¥, DW; = Via. 69) This can be an orthogonal sum, with Af; orthogonal to fj. ‘Tt must be a direct sum, with V, (1 W; = {0}. The reconstruction of /(¢) from its details Af; can start at j = Oas before, or it can start at j = —00: FO = folt+ AF) or f= As(o. oO oO The sum of subspaces can start at j = Oor j = the subspace V,41: oo. When the sum stops at J > 0, we have i i View =Vo+ OW) or Vin = > Wj. s=0 ia The left sum includes the scaling functions in Vo. The sum on the right involves only the wave- lets. That form includes all the very large time scales At = 2~/ as j + —oo, In practice we use the first sum. Our calculations begin at some unit scale. ‘The scaling func- tions at j = 0 and the wavelets with j > 0 are the basis. I suppose the scaling functions at level Jj = J and the wavelets with j > J are another basis. The Translation Requirement and the Basis Instead of rescaling f(t), we now shift its graph, This is translation, and it leads to the funda- mental requirement of time-invariance in signal processing. The subspaces are shift-invariant: If f,(0) is in V| then so are all its translates f(t — k).6.1 The Idea of Multiresolution 179 Suppose f(¢) is in Vo. Then f(2t) is in Vi and so is f(2t — k). By induction, f(2/r) is in V; and so is f (2’t — k). Dilation and translation are now built in. With translation we are committed to working on the whole line —oo < f < 00, or to pe- tiodicity. A particular f(t) may have compact support, but the whole space Vp (all functions together) is shift-invariant, For finite intervals, the requirements have to be (and can be) ad- justed. Dilation and translation operate freely on the whole line, and can be studied by Fourier transform. The final requirement for multiresolution concerns a basis for each space V;. If we choose one function $(¢) in Vo, its translates #(¢ — &) may be independent. These translates may span the whole space Vp. They may even be orthonormal. The starting assumption, to be weakened later, is that Vo contains such a function: There exists $(t) so that {p(t — k)} is an orthonormal basis for Vo. When the functions ¢(¢ — k) are an orthonormal basis for Vo, the rescaled functions 2p (2t—k) will bean orthonormal basis for V;. Atscaling level j, the basis functions ¢(2/1—k) are normalized by 2//2, We collect all the requirements in one place: Multiresolution Analysis The subspaces V; satisfy requirements 1 to 4: 1. Vj Vjat and ()V; = (0} and (JV; = L? (completeness). 2, Scale invariance: f(t) € Vj <=> f Qt) € Vix. 3, Shift invariance: f(t) € Vo —> f(t —k) € Vo. 4, Shift-invariant basis: Vo has an orthonormal basis {f(t ~ k)}- 4! Shift-invariant basis: Vo has a stable basis (Riesz basis) {¢(¢ — k)}. 4.and 4! are interchangeable. A stable basis can be orthogonalized in a shift-invariant way. This isin Section 6.4, together with the definition: stable = Riesz = uniformly independent. In practice we choose a convenient basis, orthogonal or not. Then V; has the basis @j.(t) = 2/2 (2/t —k): oo Fi) = > ajudyalt) is the piece in V;. k=—00 In the orthogonal case, the energy in this piece is (6.10) Shift-invariance and scale-invariance are built in through the basis {2//76(2/t — k)}. This basis combines requirements 2, 3, and 4! We have at least three ways to construct or describe a multiresolution: , 1. By the spaces V;180 Chapter 6 Multiresolution 2. By the scaling function ¢(f) 3. By the coefficients 24(k) in the dilation equation, Our next examples use the spaces V; and their bases. Then we move to description 3 and the dilation equation. Section 6.4 will orthogonalize the basis. The result will be the orthonormal 6(t — k) that multiresolution originally asks for. What we really need is a good shift-invariant basis. Itis also possible to allow several scaling functions $1, ...,¢,, when one function (with its translates) cannot produce the whole space Vo. This occurs in Example 3 below. It corresponds to “multiwavelets”. ‘The framework for multiresolution is set by the dilation-translation requirement. Examples come first. Then we study the dilation equation, and construct wavelets. ee f(t) in Vj f(t-1) in Vj £20 in Viet Figure 6.2: Translation stays in V). Dilation moves into Vj. Why is f(2t) — f(t) not in W;? Examples of Multiresolution 1. Piecewise constant functions. Vp contains all functions in L? that are constant on unit in- tervals n <1 00. Here ¢® = $") and the box function is a “fixed point” of the iteration, When we filter and rescale $(r) we get back (1), because the sum of two half-! -length boxes is the original box: p(2t) + 6 (6.29) The z-domain. coca isa Ba crude ba built from H@)=$4 4271. Please notice that we do not square this function, H(z) is H (z2)H(z): HOG =GFt Gt seo abe de tg dete led (6.30) After i iterations, H(z) will have 2! coefficients all equal to 2-. After rescaling, this still corresponds to the box function. Now use three filter coefficients h = (1, 4, 4). The box $(¢) produces three half-boxes in OOO = 392 + gM Ct - 1) + $92 — 2), Then there are seven quarter-boxes in $)(t). Rescaling prevents the support interval from be- coming long. The limiting interval is 0 <1 < 2. HO gO) da) + » +++ + 1 2 3 1 2 2 1 2 Figure 6.5: The cascade algorithm for 1, 1, ! converges to the hat function, A reasonable guess for the limiting function $(¢) is the hat function. This is piecewise linear, going up to $(1) = 1 and down to ¢(2) = 0. We verify that the hat function is a fixed point of the iteration. Filtering and rescaling leaves this scaling function (1) unchanged: (0) = $(22) + o(2r — 1) + 3 9(2r-2). (6.31) Notice how the coefficients 4, 3, } are doubled. The hat function is a combination of three nar- rower hats. For future reference, we note the different properties of these examples: 1. H(z) = 3+ 4271 is not zero at z = —1, corresponding to @ = 7. The iterations fail to converge. 2. H(z) =} + 327! is zero at z = —1. The iterations converge. The filter H@H (zy is halfband: no even powers except the constant term. The box function is orthogonal to its translates.6.2 Wavelets from Filters 191 3. H@ = + }27! + 42°? is zero (twice) at z = ~1. The iterations converge. The filter H(Z)H(z"') is not halfband. It contains the even powers z? and z~2. The hat function $(t) is not orthogonal to #(t — 1). ‘We must quickly emphasize that a zero at z = —1 (which is @ = 7 in the frequency domain) does not guarantee the convergence of (1). But without that zero in the filter response, strong convergence has no chance. Similarly, a halfband filter does not guarantee that $(t) is orthogonal to its translates. But without that halfband property of H(z)H(z7'), orthogonality has no chance. Section 7.2 will further indicate those connections; they are not quite two-way implications: Convergence of (t) to #(¢) needs H =O atz = —1 Orthogonality of ¢(t — k) needs H(z)H(z7') to be halfband Orthogonal Functions from Orthogonal Filters When the filter bank is orthonormal in discrete time, we hope for orthogonal basis functions in continuous time. All wavelets w(2/t) should be orthogonal to the scaling functions $(t — k). Furthermore, the wavelets w(2/t — k) should be mutually orthogonal and the scaling functions $(¢ — k) should be mutually orthogonal. Note that () is not orthogonal to $(2t). Theorem 6.4 Assume that the cascade algorithm converges: $(t) > (t) uniformly in t. Ifthe coefficients e(k) and d(k) come from an orthonormal filter bank, so they have double-shift orthogonality, then 1. The scaling functions $(t — n) are orthonormal to each other: five —n)p(t — m)dt = 6(m — n). 00 2. The scaling functions are orthogonal to the wavelets: 20 f o(t — myw(t —n)dt =0. -00 3. The wavelets wjx(t) = 2//?w(2/t — k) at all scales are orthonormal: 20 i wyt)wsx(t) dt = 6(j — J)6(k — K). 00 Proof of 1: The box functions 6 (¢ —X) are certainly orthonormal (because nonoverlapping). We will show that when @(¢ — &) are orthogonal, the next iterates 6+) (¢ — k) are also or- thornormal. Then the limits ¢(¢ — k) are orthonormal. The induction'step from i to i + 1 assumes that the o(t — k) are orthonormal, and sets f=m-n:192 Chapter 6 Multiresolution forre = mgr —n)at 2 i (Yelk p(2t — 2m — &)) (Sh elkye(2t — 2m — ky) dt (6.32) f (Xl )6 (24 — 2m — &) (elk — 2196 2r — 2m =k) 2ae = Lekoe = 2) = 6D = 6m —n). The crucial step came in the last line, when we used the orthogonality of the row {e(0)---¢(V)] to its double shifts. These are rows of L = (1 2)C. The orthogonality is in the statement LL? = 1. Equivalently, it is in the statement that |DetkyeHol? is a normalized halfband filter: no even powers except the constant term 1. Note the important point! Orthogonality of wavelets came from orthogonality of filters. When the infinite iterations converge, the limits retain orthogonality. This holds at each scale level j. In f 6(2/t — m)p(2/t — n) dt, we replace 2/t by T. Orthogonality does not hold be- tween scaling functions at different levels. Certainly, ¢(¢) is not orthogonal to all @(2t — n), or the dilation equation would require ¢ = 0. Proof of 2: Repeat the integration steps above for ¢ times w: foe —m)w(t—n)dt [cw V2 p(2t — 2m - &) (Yd) V2.9 2t - 2n —h)) at = = Vek - 2 <0. Always, ! = m—n. The last step uses the orthogonality of the rows of L = (| 2)C to the tows of B = (| 2)D. Again the double shift is essential. It is false that aff rows of C and D are orthogonal. The matrix form of this double-shift orthogonality is LB? = 0. Itcomes from the alternating flip. That choice always produces double-shift orthogonality of d's to c’s, but it does not by itself make w(t) orthogonal to $(t). To reach the end of part 2, we needed part 1— orthogonality between the ¢’s. Proof of 3: The orthogonality of wavelets w z(t) at the same scale level (the same j) is proved as in parts I and 2: ia w(t — m)w(t —n)dt Load ~ 2 = 60 = 6(m — 2). Again continuous time orthogonality follows from discrete time orthogonality. This is not DD" =1. Itis BB" = 1, with double shifts in the rows of B = ({2)D. The orthogonality of wavelets at different scale levels (different j) is immediate from the rules of multiresolution. Suppose j < J. Then W, is orthogonal to Vy by part 2. But W, is contained in V,41 and therefore in V,. So W, is orthogonal to W,. This proves the orthogonality theorem. Final note: It was convenient to start from the box function ¢® (¢), which is orthogonal to its translates. Then an orthogonal filter bank maintains this orthogonality to translates. Other start- ing functions will lead to the same fixed point @(f), or at least to a multiple c(t) — ff strong convergence holds.6.3 Computing the Scaling Function by Recursion 193 In general, convergence can be “weak” or “strong”. For weak convergence, the functions ¢( can oscillate faster and faster. You would not call this convergence. But the integral of ¢(#) converges to the integral of $(t), on every fixed interval [0, 7]. (Integration controls the oscillations.) In the convergence that we assumed, 6 (t) approaches $(t) at every point. There is a better starting function ¢ (¢) than the box. The constant value ¢(n) on each interval x < ¢ < n+ 1 can be the correct $(n). The values of ¢ are filled in at half-integers and quarter-integers by the iterations @“(r) and @(t). The graph of $(t) appears, 2! points at atime. We stop when we have enough points for the printer to connect into a continuous graph. The next section explains how to start with the correct values of $(n) at the integers. Problem Set 6.2 1. For the filter with (0) = 4(1) = 5 and any $(t), describe and draw $(r). 2, If H(2) is a polynomial of degree N, what is the degree of H(z?) H(z)? What is the degree of HQ) = [icp HG")? Rescaling will replace z by z'/?. After# steps, the degree is divided by 2'. Show that the degree of H(z!/) approaches N as i + 00. 3. With coefficients (0), ....A(N), the support interval of $(1) grows to {0, N]. What hap- pens if $(¢) is a box on [0, 2N]? 4. The unit area of the box is preserved if and only if A(0) + --- + ACN) = 1. Are negative coefficients allowed? 5, Suppose the filter coefficients h(k) are 4, 0, 0, 3. Starting from the box function, take one step of the cascade algorithm and draw $‘(¢). Then take a second step and draw ¢°'(1). Describe. ¢®(#)—on what fraction of the interval [0, 3] does $ (2) = 1? 6. Suppose the only filter coefficient is h(0) = 1. Starting from the box function @® (+), draw the graphs of (2) and $(¢). In what sense does ¢"(t) converge to the delta function 5(t)? ‘To verify the dilation equation 5(t) = 28(2r), multiply by any smooth f(r) and compare the integrals of both sides. 7. Suppose (2) is a stretched box of unit area: (2) = 1/2 for0 <¢ < 2. Draw the graphs of 6(4) and ¢®() when A(0) = &(1) = 1/2. On what interval is (1) nonzero? What is the limit ¢(¢)? 8. Suppose ¢(¢) is the Haar wavelet with zero area: 9) = 1 for O<# < 1/2 and $%) =-1 for 1/2<¢<1. With h(0) = A(1) = 1/2, draw the graphs of $°(6) and $(¢). The sequence (7) con- verges “weakly” to what multiple p(t)? 6.3 Computing the Scaling Function by Recursion The main point of this section can be stated in three sentences. Then you can follow through on the details, or look ahead for the matrices m(0) and m(1): The dilation equation gives (0), (1), ... as the eigenvector of a matrix (0). ‘Then $(t),at ¢ = half-integers comes from multiplying by a matrix m(1). Then @(t) at every dyadic t comes by recursion. Each step uses mm(0) or m(1).194 Chapter 6 Multiresolution ‘The scaling function is created recursively. This section gives the rule. ‘The dilation equation is easiest with only two coefficients (N = 1). Then m(0) = 2h(0) and m(1) = 2h(1) are scalars not matrices. The two. om dilation equation is : we m(0) (21) + m(l) oer -1). (6.33) The solution oat be zero outside the interval 0 = t ¢(n) = 1. This is the “unit area” requirement that we impose on the function $ (¢) at the start of the iterations. Then the scaling function has {[o@de = | at the end. Corollary The sum S- $(t + k) is identically |. Proof. Multiply the vector dilation equation @(t) = m(0) (2) + m(1) (2 — 1) on the left by e. Use the fact that e m(0) = e ande m(1) =e: e D(t) =e O(2r) +e O(2¢ - 1). This is a dilation equation for ¢ b(t) and its solution is the box function! Thus e (t) = 1. The “periodized scaling function” 5° @(t + &) is identically one. Example 6.3, The coefficients 2h(k) = 4, 1, } lead to the hat function, The 2 by 2 eigenvalue problem for (0) gives the correct values of (0) and (1): + © [fT [ $) : (0) =0 2 = [ 4 ll (1) ol) ] aes gy 1. The sum of all hat functions ¢(t-+2) is identically one. Notice that the first row of the eigenvalue equation is always 2h(0)(0) = $(0). Then (0) is zero, apart from the: exceptional case h(0) = $ which occurs for the box function. This means that the scaling function ¢(¢) is zero up to and including t = 0. The box function starts with a jump at ¢ = 0, because h(0) = 3. Example 6.4. The Daubechies coefficients have 8h(k) = 14 V3, 3+/3, 3~ V3, and 1—J3. Dividing by 4 we have 2h(k), the numbers that enter m(0): 1 14+¥3 0 0 (0) (0) q| 373 343 143 oa) = ou) |. 0 1-V3 3-7 (2) $(2)6.3 Computing the Scaling Function by Recursion 199 ‘The eigenvector gives (0), @(1), and (2): $0) =0 9) =4 (143) 4) =4 (I~ v3). ‘We now know the Daubechies scaling function $(¢) at the integers. The only nonzeros in the fixed-point eigenvector ® are (1) and (2). From these values att = 1 and = 2, the recursion produces (t) at any dyadic point. Practical conclusion Every ${n/2') comes via m(0) and m(1). Theoretical conclusion Those dyadic values have a uniform bound if and only if all products of m(0) and mz(1) in all orders have a uniform upper bound. When this holds, the dilation equa- tion has a bounded solution (r) for all r. We can propose sufficient conditions so that all products of m(0) and m( 1) havea uniform bound. We can also propose necessary conditions. It is not known how to verify the necessary and suf- ficient condition (Section 7.3). Derivatives of the Dilation Equation While working in the time domain, we might as well take the derivative of #(t). The result is highly interesting and not fully understood, Part of the problem is that the derivative $'(t) may not exist. ‘The plan is to differentiate each term in the dilation equation for (£): oO = 4D AGO¢'2r-h. This is another dilation equation, with every coefficient doubled, The equation (1) = M@(2r) has led to @'() = 2M'(2r). At ¢ = 0 this yields the fixed-point equation ©'(0) = 2M©'(0). The eigenvector ’(0) contains the derivatives ¢'(n) at the integers t =n. To solve ®'(0) = M&(0), we have a new requirement. The number . = } must also be an eigenvalue of M. Again this applies to the N x N matrices m(0) and m(1). This new requirement on the entries is stated as Condition A2 in the following theorem, Theorem 6.7 The matrices M and m(0) and m(1) have eigenvalues 1 and } ifand only if the filter coefficients satisfy Condition Az which includes A\: N N Condition Ay: Y(—Ikh(k) = 0 and (—1)kkh(k) = 0. 0 0 The eigenvector for . = 1 is ®(0), containing the values $(0), ... » P(N ~ 1). The eigenvector for i = 5 is ®'(0), containing the derivatives $'(0),..., @'(N = 1). In this case H(w) has a double zero at w = 2. This beautiful pattern extends onward to Condition Ay. The matrices have eigenvalues 4 = 1,4,..., q@y if and only if the filter coefficients satisfy p sum rules: ’ Condition Ap: Y(-1)k"h(kK) =O form =0,...,p —1.200 Chapter6 Multiresolution The eigenvector for A = (4)” contains values of the m' derivative of #() at the integers. For- mally, (0) = 2"40" (0) comes from differentiating the dilation equation m times at the integers. We mention the frequency domain equivalent: Condition Ay: The frequency response H (w) has a zero of order p at» = 7. We will see this again! And we also begin to uncover the crucial role of the left eigenvectors (which are row vectors). Those tell how to produce polynomials from combinations of the trans- lates g(t — k). Under Condition A, these low order polynomials 1, ¢,..., £°-} are in the low- pass space Vo. They are the keys to approximation of a function f(t) by functions in Vo. The letters A, indicate “approximation of order p.” The theorem above, with its extension from 2 to p, is absolutely basic to the algebra of downsampled filters, These eigenvalues and eigenvectors control everything in Chapter 7. ‘You will see that the derivative ¢’(t) is often one-sided. Derivatives of (2) may not exist in the usual sense. This subject still contains some mysteries. Example 6.5. The hat function coefficients 2h(k) = 4, he 4 satisfy Condition A: First sum rule: $-1+}=0 Second sum rule: 0(4) — 1(1) +2(4) =0. Therefore m(0) will have eigenvalues 1 and }: _f %#® 0 J_ 0 mo) =[ mea my |=}. 4 |: The eigenvector for ) = 1 has components 0 and 1. They agree with ¢(0) and (1), the hat function at the integers. The eigenvector for A = 4 has components 1 and —1. They are ¢/, (0) and @, (1), the slopes ¢’(+) of the hat function in the two intervals. These are derivatives from the right at the points t = 0 and t = 1. The slopes on the left side of those points are different because the hat function has corners. The matrix m(1) must also have eigenvalues | and 4: 2h(1) 2h ee | ene oe ele. | 2h(2) 0 The eigenvector for 4 = 1 has components 1 and 0. Those agree with the hat function at the shifted points ¢ = 1 and t = 2. The eigenvector for 4 = 4 has components I and —1. Those agree with the slopes of the hat function from the left att = 1 and t = 2. Remember that m(0) is involved at the start of an interval and m(1) is involved at the end of an interval. Condition Az is not satisfied for the hat function. There is no eigenvalue 4 = + The hat function has no second derivatives at the integers. vie ee Problem Set 6.3 1, If the filter H(z) is halfband, show that the eigenvector in m(0)® = © is an impulse 6(n). What are the values of (t) at the integers?6.4 Infinite Product Formula 201 2. If @:(¢) and g2(t) satisfy dilation equations, does their product P(t) = $1(1e2(s) satisfy a dilation equation? 3. Show that the convolution $j (+) *42(¢) does satisfy a dilation equation with coefficients from hy hy. 4. Find a specific function f(¢) that does not satisfy any dilation equation. 6.4 Infinite Product Formula The scaling function ¢(t) comes from the dilation equation N a) = 2 AWG. =O Thus ¢(t) is the fixed point, or fixed function, when we iterate with H and rescale. In the time domain, the matrix that filters and rescales is M = (| 2)2H. Now we intend to find the Fourier transform oo) in the frequency domain. Just as the time-domain solution involved products of m(0) and m(1), the frequency-domain solution will involve an infinite product of H(w)’s. It is quite remarkable that two-scale equations received so little attention for so long, His- torically, t and 22 were not often seen in the same equation. They began to appear prominently for fractals, which are self-similar. Now, multiple scales seem to be everywhere. We meet them in this book through multirate filters — with two scales. Then the iteration leads to all scales. If the 2’s were removed, the dilation equation would be an ordinary difference equation. The coefficients are constant, so we look for pure exponential solutions e“””. When you make. that substitution, you are effectively taking the Fourier transform of the equation. The trans- form turns difference equations and differential equations (and dilation equations) into algebraic equations. We do that now for the two-scale equation, and we watch how 2 leads to w/2. The dilation equation becomes $(w) = H () og). This leads recursively to an infinite product for ¢(w). This transform must be a sinc function when h(0) = h(1) = 1/2 — because the time-domain solution $(f) is a box function, That sinc function must be orthogonal to its modulations by e~*, because the box function is orthogonal to its translates ¢(r — k). We have to study orthogonality and also approximation, which is controlled by “zeros at 7.” These properties are now studied in the frequency domain. 1, Condition O for orthogonality: |H@)P?+|H(@+2)P = Lin the frequency domain 2D AK )aK — 28) = — 6(¢) inthe time domain This is double-shift orthogonality of the lowpass filter coefficients. It connects to orthogonality of the scaling functions ¢(t — k). We use the word “connects” rather than “implies,” because a further condition is eventually needed to insure orthogonality in the limit of the iteration. The step from discrete time to continuous time seldom goes wrong, but it can. This orthogonality will appear in Theorem 6.10 as a neat statement about the Fourier trans- forms of ¢ and w: SY Bo4 20m) =1 and S°$(w+ 2nn)ii@+ tan) = 0. 4 ee202 Chapter 6 Multiresolution The other side of the theory is about approximation. This imposes a very different condition on the A(k) and the polynomial H(@) = > h(k)e“#, 2. Condition Ap: H(w) = ¢: te)’ o). This factor (1 +e)” means that H() has a zero of order p at w = m. We will prove that this puts the polynomials 1, ¢,...,t?-' in the scaling subspace Vo. They are combinations of $(t — n) and they are orthogonal to w(t — n). In the frequency domain, there is again a neat statement about the Fourier transforms of ¢ and w: has a zero of order p at every w = 2nn,n ¢ 0 i has a zero of order p at zero frequency. The wavelet coefficients of a smooth function f(t) = > bj,wje(t) decrease faster when pis larger. The estimate is |bj¢| = O (27/?). This is valuable for compression. This section does the frequency-domain algebra, to solve the dilation equation and to explain Condition O and Condition Ap. Transform and Solution of the Dilation Equation To transform the dilation equation, multiply by e~!*. Integrate with respect to t: OO Ne oo i i oe ids = 2) he) i (2 — kye™ dt, 00 ka 20 The left side is $(@). In the integral on the right, set « = 2¢ — k and t = (u +k)/2: 00 60 oe f $(2t — ke 2dt = f due Or? dy = eth? F(a), (6.43) co 00 Tnstead of ¢ and 2t, the transform involves w and w/2. The dilation equation becomes $0) = (Saye?) $(8) = H(3)3(8). (6.44) This is the result of filtering and rescaling. Filtering multiplies $(w) by H(w). Rescaling changes w to @/2. The scaling function (which is unique up to a constant multiple C —this is still to be proved) comes out unchanged: Now iterate this equation. It connects w to w/2 and therefore it connects @/2 to w/4: o) = H(g)[4(2)9(8)).- After N iterations, this becomes (w) = H(3) H()--- H() (Be). In the limit as N — 00, we have a formula for the solution $(w). Note that 0/2” is approaching zero, and $(0) = f (t)d¢ is the area under the graph of (1). This equals one. We impose6.4 Infinite Product Formula 203 the normalization $0) = 1 in the frequency domain, just as we required unit area in the time domain. Then the formal limit of the iteration leads to the famous infinite product for ¢: ie 4 6G) = iH H(¥). (6.45) int We note the minimum requirement for ¢ eoaver ace fof this or any infinite product: the factors H(c/2/) must approach 1 as j — 0. Thus we need H(0) = 1. By periodicity H(2x) = H(4z) = 1. Then the equation (a) = H (#) #($) has a remarkable consequence. The values ¢Qz), (4m), $(8x), are all equal. If H(x) = 0, those values equal zero because $n) = H(x) b(n). = This “zero at 7” is a natural requirement on H(w) in order that (w) may decay and $(t) may be a reasonable function. The infinite product converges for every w and every H(w). We have an explicit formula for $). ‘Whether any function ¢ (¢) has this Fourier transform is another matter! Convergence follows from a rough bound on H(w) in terms of C = max | H’(w)|: 1H(@)| = [1+ Hw) — HO)| < 1+ Col eM". Then the product (w) has the same upper bound: [o(w)} = |H(S)] |A(g)| --- < e€ lov? coo... = eM, This is a wild overestimate of $o), as almost any example will show. Box example. The coefficients are h(0) = h(1) = 3. Then H(w) = 3 (1 +e). The product of the first N factors contains 2% terms. Looked at correctly, those terms are the first 2 powers of e~#/2": HW) 7 (1+?) (14 eM). (1 . evn") 7 * Fun (geometric series) i= je . = POs (sum of 2% terms). (6.46) Now let NV > 00, The denominator has 1—e~ = 1—(1-i0 + -»-) = i9 +: -- with@ = w/2%, The limit of 2" 76 is ico. Therefore the limit of the partial product is the infinite product ge = TT (G + Jeter?!) =(1-e) fio. (647) This sinc function is the transform of the box function. The integral of e~'“’ from 0 to 1 agrees with $(@). Instead of increasing like e“!!, as allowed by the general estimate, the transform Fw) actually decreases to zero as w becomes large. Compare the construction of #(f) with $(). In Section 6.2, we assumed that $"(t) con- verged uniformly to $(f). Then we studied its properties. In this section, the convergence of the infinite product is Cheap (for each separate w), What we need is sufficient decay of Go) as204 Chapter 6 Multiresolution |e] > oo. Our precise assumption will be continuity of the function A(w) in Theorem 6.10 be- low. Then we can safely study $l) i in the frequency domain. Note first that for real frequencies, the growth of lo) i is at most polynomial: Theorem 6.8 [6(w)| < el for complex w and |f(w)| < c(1 + |w!") for real wv. Brief reason: H (w) is periodic. Ithas a maximum value 2. The equation $Qw) = H(w)$(w) says that $ grows by at most 2" when w is doubled. The bound |o|™ has this growth rate. A constant is included to make c(1 + ||) correct for small [w}. The example H (w) = 3+ 4e7'” is bounded by | for real w and by e!#! for complex w. Then M = 0. The transform Hw) a the box function has those same bounds: for real =~ o o 1 ol = Jn Jn 7S { ell? elo’... = ell for complex w. Section 6.1 showed that the support, interval for @(t) is [0, N]. This can be proved in the frequency domain too. Our bounds on $(w) show two fundamental facts about P(t): Theorem 6.9 Any dilation equation with h(0)-+--++h(N) = | has a unique and compactly supported solution (1). This solution may be a distribution. Compact support comes from |G(w)| < ell, The Paley-Wiener Theorem implies that (J) is supported on the interval [—C, C]. With more care (D, p.176) we could find again the exact support interval [0, NJ. Uniqueness comes from our formula for the solution! The infinite product converges to $(w), which is continuous because $(¢) has compact support: B() = (HI H(w/2!)) 8/2) approaches $e) = (i Hiw/2)) BO). 1 y= In the IIR case, suitable hypotheses will again give uniqueness (of course not compact support). At the other extreme, note how the fazy filter with A(0) = 1 leads to @(r) = delta function. The dilation equation (1) = 26 (21) is solved by $(t) = 8(1): In frequency: H(w) = 1 s0 bw) = 1. Cascade algorithm: g(t) = box function on {0, 2~] with height 2'. Verify directly: 5(1) = 25(2t) from f f(NS(@)dt = FO) = f f(SQN2aP. Alt these methods show that 4(0) = I produces the best-known distribution (1) = 6(1). Orthogonality in the Frequency Domain ‘The product formula for #() applies with or without Condition O. When that condition holds, we expect orthogonality of the translates ¢(¢ — k). To establish this orthogonality in the tre- quency domain, we need to know that the equivalent statement is A(@) = 1. The function A(m) enters naturally into this discussion. It is the transform Lake" of the vector of inner prod- ucts of @{t) with @(¢ — k):6.4 Infinite Product Formula 205 Theorem 6.10 The inner products a(k) are the Fourier coefficients of the 27-periodic func- tion A(w): 00 co atk) =| O(NO —kK)dt transformsto A(w) = oe |oo+ 2nn)/. 00 “oo The translates $ (1 — k) are orthonormal if and only if A(w) = 1. Proof. An inner product in the time domain equals an inner product in the frequency domain, by Parseval’s identity. The inner product in the time domain is between ¢(f) and ¢(t — k). The transforms of these functions are $(w) and e~!*$(w), Each inner product integrates one func- tion times the complex conjugate of the other: at) = f oe@e=Bar = =f Bwrpwe de fs Ee 1 f®Qn oF = [ SY bet 22m) ede. (6.48) ‘0 co The last integral split (—o0, 00) into an infinite number of 27-pieces, using the periodicity of et, This integral defines the k Fourier coefficient of A(w). Thus A(w) = Ya(k)e. For an FIR filter, $(f) = 0 outside the interval [0, NJ. The inner products are a(k) = 0 for |k| > N, because @(t) and (t — k) have no overlap. The function A(w) = Sra(k)e' is a trigonometric polynomial of degree N, which is not obvious from > \lo + 2xn)?. In Section 7.3 we will compute a(k) directly from the coefficients k(n). This is always a main point of the theory, to return every calculation to those numbers h(). When the translates are orthonormal, all inner products a(k) are zero except for a(0) = 1. The function with those coefficients is the constant function A(w) ce O(¢—%) are orthonormal <= > |$(@+2nn)f? We now apply Condition O in the frequency domain to deduce this orthogonality of ¢(t—k). We are repeating in the frequency domain the result of Section 6.2 in the time domain. I believe this is worthwhile! The arguments in the two domains look quite different. Recall the condition on the frequency response H(@) to produce an orthonormal filter bank: Condition O: |H@)P+|H@+z)P = 1. This function H (w) leads to $(c2) which leads to A(w). Somehow, Condition © must imply that A(w) = 1. The steps are typical of computations in the frequency domain. Theorem 6.11 If A(w) is continuous, A(w) = | is equivalent to Condition O. Proof. We use a very important two-scale identity, proved below: AQo) = lH)? A@)+lH@+x))? A@tn). (6.49) If A@) = 1, this iminediately gives that |H (co)? + |H(w+ x) = 1.Zuo Chapter 6 Multiresolution For the converse, suppose that Condition © holds. The identity says that A(2c) is a weighted average of A(w) and A(w + 7). At the point wo where A(2w) reaches its maximum, A(@) and A(wo + 2) must also reach that maximum. Now repeat the argument at «/2, to show that A(wo/2) shares this same maximum with A (wo). Continuing, the maximum of A(w) is achieved al @p/4 and wo/8 and eventually (by continuity) at @ = 0. By a similar argument, which is due to Tchamitchian, the minimum of A(q) is also attained at w = 0. Therefore A(w) is constant, We verify below that the constant is one: A(w) = 1. It only remains to prove (6.49). This valuable identity for A(2w) has a nice proof. It uses the dilation equation $20) = H(o)$(). At the points 2w + 2zn, this splits into separate cases for even n and odd n: GQwo+2nn) = Hlwot+an)dw+an) — [ H@ $+ 2) ifn = 2k ~ H@+n)$@+(2k+)x) ifa=2+1. Now square both sides. Sum from —oo to 00 on n and therefore on k. The sum of squares is our function A(2«) in the desired identity; 2 co AQo) = 1H)? > |G + 2b) + 1Hlo+ PD [Got x +20)? Ed Ss = JH)? Alo) + |H(ot mM? A@+n). (6.50) The final step is to confirm that A(0) = 1. This comes from our other condition on the lowpass filter, not yet used in the frequency domain. Condition A; is H(r) = 0. In the time domain, this first sum rule guaranteed an eigenvalue 2 = | for the matrices M and m(0) and m(1). The fixed-point equation 6 (n) = ¢(n) at the integers could be solved. Condition AL is equally essential in the frequency domain. Here we use it to pin down the value A(0) = 1. Theorem 6.12 If H(x) = 0 then $(2xn) = 0 for all n #0. Therefore co AO) = >> |G2xn)) = [GO =1. (651) bad Proof. The infinite product for $(2m) = A(x) H (x /2)--- starts with the factor H (Gr). Imme- diately this product is zero. For any higher value n > 1, write » = 2/m with odd m. Then the G+ 1)* factor in the infinite product is zero when @ = 2xn: $Qnn) = Hanya (=*) (3) . ae because H(xn/2/) = H(xm). By periodicity this is H(r) = 0. The only nonzero term is (gO). But $0) = (0)H#(0)H(0)--- which is 1. Orthogonalization of the Basis The condition for an orthonormal basis is A(w) = 1. When this is not satisfied, there is an easy way to make it satisfied. In other words: when the translates $ (t — n) are not orthonormal, there6.4 Infinite Product Formula 207 is an easy way to make them orthonormal. Divide $(@) by the given A() (or rather, its square root) to get the new orthogonalized function Ponth(@): 3 HO) donh) Ton This immediately gives orthogonality of the new basis {¢orn(t — #)}: _ yo |G@+2an) Aw) _ Aonn(o) = 2 Tatas A@) ‘That succeeds if A(w) is never zero. This is the condition for a Riesz basis. Theorem 6.13 The upper and lower bounds on A(w) are the Riesz constants B and A for the basis (@(t — k)} of Vo. Thus A(w) > A > 0 gives a stable basis, and dividing $(w) by A(w) gives an orthonormal basis. When $(r) comes from a dilation equation — this is our normal! situation — Condition E in Chapter 7 gives an equivalent test for a Riesz basis (in terms of eigenvalues). If this test is passed, the wavelets w,,(f) are a Riesz basis for LR). Proof. To test the linear independence of the functions ¢(¢ — X), form the matrix A from their inner products. The entries are Aj; = (@(¢ — i), 6(¢ — j)). That number is a(j — 7), and A is a Toeplitz matrix! It is the matrix TT” in Section 2.5. In the frequency domain it becomes multiplication by A(@). The upper and lower bounds on A(w) determine whether {(¢ — k)} is a Riesz basis. Orthogonalization is always a basic step in linear algebra. There it is done by the Gram- Schmidt algorithm. We start with independent vectors and produce orthonormal vectors (or functions). This algorithm is not successful here, because it is not time-invariant. The orthog- onalized functions will certainly not be translates — when the Gram-Schmidt algorithm works on functions in a definite order like #(t), $(t— 1), 6@ +1), .... To keep a shift-invariant basis, we needed to orthogonalize all these translates at once. The division by /A(@) did it. In matrix language, Moq,M2.4), = J. In the improved factorization by Fourier methods, all tows of Marth come from the zeroth row by double shifts, In other words, Moth comes from a filter, One problem with dividing by A(@). This destroys the finite response of the original filter H. The new filter Horth is IIR, not FIR. The new scaling function @opy(t) that corresponds to $(w)//A(@) does not have compact support. Vetterli and Herley noticed that this is not as bad as it seems. Since $(t) is zero outside the interval (0, NJ, the inner products a(k) = J ¢(OC@ — © dt are zero for |k| > N. The function A(w) with these Fourier coefficients is a real non-negative trigonometric polynomial of degree N. Its square root G(w) = g(kye io, by spectral factorization, is also a polynomial of degree N. The frequency response of the new orthogonalized filter is a ratio of polynomials H@) Ge) ‘The input-output equation y(k) = S> Aogh (k)x(" — ) is an implicit difference equation, from an autoregressive moving average filter: Honn(@) = N N 1 Yis@ye- = Yorba —%. 7 9208 Chapter 6 Multiresolution ‘The new filter is IER but it only involves 2N + 2 parameters &(k) and h(k). Therefore it can be physically realized, and now the basis has been orthogonalized. Problem Set 6.4 1. Use the identity sin 20 = 2sin6 cos@ to show that ey); @ ®)_ 1 sinw sing (0085) (cos 3) (cos 37) “OF Sin ® sing * Cancel sines and let N ~> 00 to find a great infinite product: yoy _ 1 []oos(5) = x sine. 2. The Haar filter has H(w) = 3(1 +7") = e~“/t cos. Use # in Problem 1 to give anew proof for the infinite product (6.47) of H(co/2!): (ert cos) (et cos 3) 2 (ewe = sin @ 3. Suppose H(w) = 1(1 + e7')?, Find $(w) and $(4). 4. If H(w) has p zeros at w = x, show that #(w) has p zeros at «@ = 2n for each n ¥ 0. 6.5 Biorthogonal Wavelets This chapter has concentrated on orthogonal wavelets, coming from an orthogonal filter bank. The synthesis filters are transposes of the analysis filters. One multiresolution is all we need. The synthesis wavelets are the same, in this self-orthogonal case, as the analysis wavelets, But from biorthogonal filters we must expect biorthogonal wavelets. We now meet a new scaling function $(¢). Its translates $(t ~ k) will span a new lowpass space Vo—different from Vo. There is also a wavelet &(t), The translates (1 — k) span a complementary highpass space Wo. The sum of those spaces will be ¥, = Vo + Wo, the next space in the second multiresolution. To a large extent, the theory is achieved by inserting a tilde where appropriaté. We want to indicate why this second scale of spaces v, is needed, Biorthogonality comes automatically with inverse matrices. The rows of a2 x 2 matrix and the columns of its inverse are biorthogonal: row 1 column column ]_[ 1 0 row 2 1 2 Oe Notice something pleasant. The product in the other order is still J. The Tight-inverse is also the left-inverse. This order involves columns times rows, which are full matrices: [ column It oy i+[ a lt row 2 Jad Those two column-row products are projections, and they add to I. These simple facts about 2 x 2 matrices have important parallels for biorthogonal filters and biorthogonal wavelets.6.5 Biorthogonal Wavelets 209 Filter banks display those parallels immediately. The analysis bank has filters (J 2)Ho and (J 2)H). Those are rows 1 and 2. The synthesis bank has expanders before filters, Fo (t 2) and F; (+ 2). Those are columns | and 2. In the orthogonal case Fo is H} and Fy is HT. In the biorthogonal case we don't have transposes but we still have inverses. To understand the pattern for wavelets, we absolutely must return to multiresolution, One scale of spaces Vo C Vi C +++ © VY) is too limited. We need two hierarchies of spaces, V; in synthesis and ¥, in analysis. Tilde Notation Does the tilde go on the analysis functions or the synthesis functions? Both conventions are equally possible. We hope to agree with other authors! More and more, the (ilde is going on the analysis functions. Then f(¢) is expanded in synthesis functions, which have no tilde. But the coefficients come from the analysis functions and have tildes: fot) = Laude —k) isin Vo, with m= ffinde-Hdt — (6.52) £0) = CLDawaz isin 2, with by = [SOB de (6.53) What does this mean for the filter banks that process the coefficients? Those filters use the letter H in analysis and F in synthesis. We will stay with H and F (rather than C and D) when dis- cussing biorthogonal filters. An important result in this section is the Fast Wavelet Transform in equation (6.70), and its inverse (the biorthogonal IFWT) in Theorem 6.16. Biorthogonal Multiresolution This chapter began with orthogonal bases {(¢ — k)} for Vo and (w(t —k)) for Wo. The equation Vo ® Wo = V, started a multiresolution, Wj was the orthogonal (!) complement of V; inside Via1. Allis well if 6(r) and w(t) come from an orthogonal bank of FIR filters. Their translates are all orthogonal. They span perpendicular spaces and we have an orthogonal multiresolution. All is not so well if $(¢) and w(t) fail to have compact support. The filters fail to be FIR. Often this means that we have asked for too much! Instead of orthogonal bases, we should be content with stable bases. An outstanding example is the space of piecewise linear functions. The stable basis consists of the hat function $(t) and its translates, That basis is not orthogonal. When the basis is not orthogonal, there is no reason to insist that Wo must be orthogonal to Vo. If we do, the multiresolution is called semi-orthogonal in Section 7.4, and we have “pre- wavelets”. But the important property is a stable basis {w(t — k)]. The highpass coefficients will construct w(t). Remember the pattern for perfect reconstruction. Coefficients are chosen so that Fo(z) Ho(z) is halfoand. When #(¢) is the hat function from Fo(z) = (44+)’, the other factor Ho(z) needs five coefficients. This means N = 2 but N = 4. The wavelet has 3-interval support. Then o(¢ — &) and w(t — k) span Vo and Wo, without orthogonality. The new analysis multiresolution is the point of this section. The coefficients from H(z) g0 into a different dilation equation, whose solution is the analyzing function $(¢): i Analysis Dilation Equation: $(t) = > 2ho(k) d(2t — &). (6.54) o The coefficients hg(k) add to 1 as before. The new multiresolution obeys the same conditions as before; just add a tilde. Vo is spanned by (#(¢ — &)}. The space V; is spanned by {(2/1 — k)).210 Chapter 6 Multiresolution They are clearly shift-invariant. The dilation equation (6.54) says that % C Vj. Then also ¥; c Vj41. The highpass coefficients produce the wavelet: ” Analysis Wavelet Equation: i5(1) = ) > 2h1(k) $(2t — k). (6.55) 7 This wavelet is supported on (0, €], where 26 = N + NV. That sum N + WV is the degree of the product filters Fo(z) Ho(z) and F(z) H(z). Those are symmetric halfband filters, and in the hat function example the degree is N + WN = 244. Then é = 3isodd, The four functions $(¢), w(t), (0), C(t) are graphed in Figure 6.6. ‘Analysis Scaling Function Syothesis Scaling Fanction 04 03] on 02] 0.05 on : o| O41 02! ° 1 2 3 4 ° 05 1 18 2 Analysis Wavelet Function Synthesis Wavelet Function 02| 0.05 o| o| 0.05 0.2! O41 con 0.15 ° 08 1 15 2 ° 1 2 3 4 Figure 6.6: Biorthogonal scaling functions and wavelets from a 5/3 PR filter bank. Biorthogonality in Continuous Time Our construction of #(2), w(t), O(¢), ®(Z) starts with biorthogonal filters. The lowpass analysis coefficients 4g(k) are not double-shift orthogonal to themselves. They are double-shift biorthog- onal to the lowpass synthesis coefficients f(k): 2D holbeyfolk + 2n) = b(n). (6.56) ‘This means that Fo(z) Ho(z) is halfband. Similarly the highpass filters give Fi(z) H1(z) = half- band: 2S" ha (of (k + 2n) = 6(n). (6.57)65 Biorthogonal Wavelets 21 The other key relation is biorthogonality of highpass to lowpass: Dito + 22) =O and Shy (kYolk + 2n) = 0. (6.58) The reader knows that all these equations restate perfect reconstruction: Fo(z) = A(z), F(z) = —Ho(—z), Fo(z) Holz) is a halfband filter, Our question is, how does biorthogonality appear in continuous time? The functions $(t) and ot) come from it iterating the lowpass filters Fo and Ho (with rescaling!). Start the cascade of iterations from ¢(t) = $(t) = box function on [0, 1]. Their translates have biorthogonality. The box $(t—k) has no overlap with ¢ (t—€) when k # €. This biorthogonality is preserved at every iteration step, when we use equation (6.56). This is exactly Paralle! to the earlier proof (6.25) that orthogonality is preserved at each iteration. When $(¢) and $(t) converge in L? to the scaling functions (1) and 3), those limit functions inherit the same biorthogonality: f $t — OO — O dt = Hk - 2). (6.59) 20 With i — oo in the cascade algorithm, these limits (+) and $(¢) solve the synthesis and analysis dilation equations. Now bring in the wavelet equations: fl omvma= f 7 (Srwe2r-H) (LH wAwser—H)ar. 6.60) That right side is zero because of (6.58) and (6.59). And biorthogonality extends to the translates for the same reason: . il O(t — BE — Oat =O forall k and e. (661) 7 Finally the wavelets w and i are biorthogonal from the wavelet equations and (6.57): . f w(t — WBE — Hat = 6k—. (6.62) ES Allthis is routine, provided the cascade algorithms for @(t) and $(t) both converge in L?, Wave- let theory gives the requirements in Section 7.2, as tests on eigenvalues of two matrices T and F. Suppose those tests are passed (not at all automatic!), The basis functions are biorthogonal, What does this say about the subspaces they span? Theorem 6.14 — Suppose the filter coefficients satisfy (6.56){6.58) and also Condition E (for L? convergence of the cascade algorithm). Then the synthesis functions @(t — k), w(t — k) are biorthogonal to the analysis functions $(t — €), i(t — €) as in (6.59)(6.62). Each scaling space is orthogonal to the dual wavelet space: V,LW, and Wii,. (6.63) Vp and Wo are perpendicular because their bases @(¢ — k) and t(t — k) are perpendicular. When tis replaced by 2/y, the zero inner products are still zero. (Change variables back to T = 2/t.) So at each scale j we have perpendicular subspaces.212 Chapter 6 Multiresolution ye s L SY Figure 6.7: V; is perpendicular to W;, while V, is perpendicular to W;. The two multiresolutions are intertwining. As always we have View and V4.0; = Far. (6.64) These are direct sums but generally not orthogonal sums, The subspaces V; and W, have zero intersection, but they are not perpendicular. Instead V; is perpendicular to W;, All the subspaces Wj-1, Wj-2, ... are then perpendicular to W;. Similarly all the subspaces Wj_1, Wj-2, ... are perpendicular to W;. Therefore we have biorthogonal bases (dual bases): Corollary — The wavelets wjx(t) = 2//>w(2/t—k) and Bx (t) = 2//2H(2/t-k) are biorthog- onal bases for L?: 00 f wylt)Bye(t) dt = 6) — JK K). (6.65) co Representing f(t) in a Wavelet Series If we have a, wavelet basis, we have a wavelet series. Any square-integrable (finite energy) func- tion f(t) can be expanded in wavelets: oo fO=V VL Fawpaw. (6.66) =e oo The synthesis wavelets are used to synthesize the function (of course). But the coefficients Bj, come from inner products with the analysis wavelets. This is why b,x has a tilde. Multiply (6.66) by the analysis wavelet i) (¢) and integrate over , Biorthogonality yields ox = | FO) Bx Oat. (6.67) ee Equations (6.66) and (6.67) are the biorthogonal wavelet transform and its inverse. The trans- form takes function to coefficients, the inverse transform synthesizes the function. We show be- low how the coefficients can be computed recursively (or pyramidally). This is the fast wavelet transform.6.5 Biorthogonal Wavelets 213 Note that Parseval’s equality between f | f(¢)? dt and > Y [bjs P is not true. That required orthogonality —- and not biorthogonality. When we square and integrate the series for F(t), non- zero inner products come from the products wj4(t)w, x(t). We do have an inequality 00 oe oo A i, If@P dt < OY? bal’ s B f (F@P at. (6.68) co 00 With B > A > Othis says we have a stable basis or Riesz basis. This is true for wavelets, subject to the same Condition E [Cohen-Daubechies]. Fast Biorthogonal Wavelet Transform The reader knows that in practice the wavelet expansion cannot go all the way back to j = —oo. We do not use arbitrarily low frequencies (longer and longer wavelets). A more practical expan- sion starts with Vo and Vo, at a scale normalized to Ar = 1, and goes to V; and v,, where the finer scale is 2~’. Enough high-frequency details are included to reproduce an accurate signal. Thus we work primarily with the subspaces Vy = Vo + Wo +-+- + Wy_1. There are two important bases for V;. One is @yg(t) = 2//*(2/t — k) for —00 < k < 00. These scaling functions are at level J. The other basis consists of g(t) from Vo and wy(t) forO 2). The 2/6 biorthogonal transform with 42 =[1 I] is reversible and very useful. Lowpass components yx; come first: Yai = Xai + Xr41 and then yois1 = ai41 ~ Ly2i/2} + L212 — Yoi42)/16]. The inverse also has “even = even + f(odd)” and “odd = odd + g(even)”: Xa = Yu —Xri41 and then x41 = yoint + Ly/2] — WOrw-2 — ya42)/16). Majani has shown that the new binary 9/7 and 13/7 transforms have reversible forms (lossless in integers). The normalized DCT is not reversible for integer data. Note 3 The maxflat Daubechies filters with 4p — 1 coefficients and 2 )p zeros are also known as Deslauriers-Dubuc filters [DesDub, CDM]. They interpolate because they are halfband. They leave the values x(n) unchanged and produce midpoint values x(n + 4). Section 5.5 confirmed that all polynomials of degree less than 2p are interpolated exactly. Recursive subdivision start- ing from x(1) = 6(n) converges to the scaling function (+) by the cascade algorithm! Problem Set 6.5 ee Double-shift orthogonality of lowpass filters is 2D ho(kYfo(k + 2n) = 6(n). Show that in frequency this becomes Ho(@) Fo(w) + How + 1) Fo(w + x) = 1. ‘Write the same equation in the z-domain. 2, Problem 1 involves a row and column of the modulation matrices F,, and Hy»: Faotate) = [ Fo) Fiz) Il Holz) Ha(-2) Fo{-z) Fi(-z) AM) M(-2 J Which row-column multiplications correspond to which equations (6.56)-(6.58)? 3, Suppose Ho(w) Fo(w) + How + 7) Faw + 1) = 1 as required, By alternating flip Hw) =e" Fut) and F\(w) = -e" Hy(w + 7). Show that the other entries of F,,(z)H,(z) = I are then correct. 4, What wavelets come from the biorthogonal filters with Hy = 1, Fo = get l + gL = 32-14 42-!, Fy = 17 Recognize the delta and hat: $6) = BAH and $0) = $2141) +42N +4608 -D.6.5 Biorthogonal Wavelets 5. 9 Then construct wavelets from H(¢) = —36(2r + 1) + $(2t) — 3G(2t — 1) and w(t) = 26 (21 — 1). Check the biorthogonality conditions f HOE — Kade 5k) and f we — kde fowau —k)dt fFowe ~kadt 0. Lifting from the Haar filters H(z) = $(1-+27') and F(z) = 1427! show that all PR solutions have the form (6.75). The difference D = H* — H must satisfy D(z)F(z) = D(—2)F(-z) from (6.74). Substitute D(z) = a + bz! + cz? + dz + --- to show that it has the form F(-2)S(2?). . Lifting (H, F) to (H", F) does not change the scaling function #(¢). Show that the new wave- let is w*(t) = w(t) ~ D s(k) ote — k). The fast wavelet transform is subband filtering of the inner products aj, = (f (1), djx(#)). The highpass channel produces bj, = (f (0), j«(£)} by (6.70): ay = SCAG = 2Rajyir and by = Dhue - Ways. Suppose H is lifted to H*. Show that the lifted at, are at, = ay + Ss(l — k)by. The fast inverse transform unlifts by —s(/ — k) in the same way. Then it inverts the (H/, H) transform as usual by aj. = Sk — Way + DF k - Wb jr. }. From the input x(7) = ay, compute even samples an, = ai.% and odd bo = a)2x41 — (aoe + @oe+3)- Then lift to af, = aoe + (by4-1 + bu)/4. Combine to recognize the 5/3 filter bank, computed more efficiently and in place —no auxiliary memory. Which filter A gives linear interpolation at each step of recursive subdivision? What is $(¢)? 219Chapter 7 Wavelet Theory This chapter follows through on important questions about scaling functions and wavelets. We study their smoothness, their inner products, their accuracy of approximation, and the number of vanishing moments. Even more basic are the existence and the construction of $(t). Does the dilation equation have a solution $(t) with finite energy? Does the cascade algorithm con- verge to this solution? We answer those questions in terms of the filter coefficients 4(k) (and the answers are not always yes). Our overall aim is understanding, with a minimum of technical details, We point immedi- ately to two fundamental operators in wavelet theory. They both have a double shift between rows coming from (|, 2): M=(12)2H and T=({ 2)2HH". (CAD) Apart from its extra factor 2, M is completely familiar: filter by H and then downsample. This is the decimated lowpass analysis filter, with coefficients A(k) adding to 1. In the time domain, those are on the diagonals of H. The symmetric product HH” is also a Toeplitz matrix. Its entries are the coefficients in |H(@)[?. The rows are double-shifted by (|. 2). In frequency, downsampling produces an alias- ing term from modulation by 7: (Mf) = HG) (9) + HG +m) 44m) (TH)(o) = |HS)P FG) + AS +P FE +m) ‘The properties of M and T hold the answers to our questions, Iterating the lowpass filter, with subsampling, involves powers of matrices. The convergence depends on the eigenvalues. That will be the message in this chapter: Watch the eigenvalues. ‘The “transition operator” or “transfer operator” T turns out to be simpler than M — because |H(w)?? > Oand HHT is symmetric positive definite. T enters when we compute inner products and energies (L* norms). After / iterations of M in the cascade algorithm, starting from 6 (t) and reaching (r), the inner products are (7.2) co a@(k) = f 99H GC + bat. 73) The key point will be that Za“ (k) gives the inner products a“+)(k). The powers of T (and therefore the eigenvalues of T!) decide whether the cascade algorithm converges, when we use7.1 Accuracy of Approximation 221 the L? energy norm. The properties of M give pointwise answers, and the properties of T give mean square answers, We can summarize in a few lines approximately what those answers are: 1, Combinations of ¢(¢ — k) can exactly produce the polynomials 1,7,..., 2°" if M has eigenvalues 1,4,...,(1)’"" ; Thas eigenvalues 1, 4,...,(4)°?’. 2. The wavelets are orthogonal to 1, ..., #?~' so they have p vanishing moments. The func- tions #(¢ — k) give pth order approximation from the space Vo. 3. The cascade algorithm converges to (t) in L? if the other eigenvalues of T satisfy |4| < 1. 4, $(¢) and w(t) have s derivatives in L? if the other eigenvalues of T satisfy |Al < 47°. In 1, the special eigenvalues (powers of » give anew form of Condition A». It is equivalent to p zeros at 7, from a factor (1 + z~!)? in H(z). In 3, the requirement |A| < 1 on the other eigenvalues will be called Condition E. This is new to the book. It gives convergence of the cascade algorithm to a Riesz basis {¢(t ~ k)}. Condition E is the key to iteration of filters, and thus to wavelets. In4, the smoothness s is very likely not an integer (but we work with integers for simplicity). The number Smax of derivatives of ¢(t) is never greater than the number p of vanishing moments. The smoothness of #(+) is important in image processing, and p is more important. This chapter begins with the accuracy of approximation. That has the most direct answer. The approximation order is p when the frequency response H(w) has a pth order zero at w = x. Zeros at x are the heart of wavelet theory! This multiple zero is produced by the factor (1+e7‘)? that appears throughout the design of filters. T is associated with | H (w) |? for which the order of the zero becomes 2p. This is reflected in the 2p special eigenvalues of T. Sections 7.2-7.3 analyze the cascade algorithm $+) (¢) = D> 2h(k) $(2# — k). Section 7.4 explains splines and spline wavelets. Section 7.5 introduces multiwavelets. We note here the MATLAB commands to construct M and T from the vector h: function M = down(h) n = length(#); M = zeros(n,n); for i = iin, for j if (0<2* 4 - J) & (2 * i - J) <= n) M(i,j) = end end, end autocor = conv(h, fliplr(h)); T = down (autocor) = 1:n, 2*h(2*i- 5); 7.1 Accuracy of Approximation In applications of wavelets, a function f(t) is projected onto a scaling space V;. This index j gives the time scale At = 2~/ in the calculation. The scaling functions are 2//7¢(2/t) and its translates by kAt. Those represent one basis for the space V). The projection f;(t) is the piece of f(t) in that subspace, so it is a combination of those basis functions: a Foreach j, f)(t)= > ajp2/?g(2/t -k). (7.4)