Cambridge University Press

978-1-107-60104-8 - The Mathematics of Signal Processing

Steven B. Damelin and Willard Miller
Frontmatter
More information

The Mathematics of Signal Processing

Arising from courses taught by the authors, this largely self-contained treatment is
ideal for mathematicians who are interested in applications or for students from
applied fields who want to understand the mathematics behind their subject.
Early chapters cover Fourier analysis, functional analysis, probability and linear
algebra, all of which have been chosen to prepare the reader for the applications to
come. The book includes rigorous proofs of core results in compressive sensing and
wavelet convergence. Fundamental is the treatment of the linear system y = Φx in
both finite and infinite dimensions. There are three possibilities: the system is
determined, overdetermined or underdetermined, each with different aspects.
The authors assume only basic familiarity with advanced calculus, linear algebra and
matrix theory, and modest familiarity with signal processing, so the book is accessible
to students from the advanced undergraduate level. Many exercises are also included.

steven b. damelin is Full Professor of Mathematics and Director for the Unit of
Advances in Mathematics and its Applications, USA.

willard miller, jr. is Professor Emeritus in the School of Mathematics at the

University of Minnesota.

© in this web service Cambridge University Press www.cambridge.org

Cambridge University Press
978-1-107-60104-8 - The Mathematics of Signal Processing
Steven B. Damelin and Willard Miller
Frontmatter
More information

Cambridge Texts in Applied Mathematics

All titles listed below can be obtained from good booksellers or from
Cambridge University Press. For a complete series listing, visit
www.cambridge.org/mathematics.

Nonlinear Dispersive Waves

MARK J. ABLOWITZ
Complex Variables: Introduction and Applications (2nd Edition)
MARK J. ABLOWITZ & ATHANASSIOS S. FOKAS
Scaling
G. I. R. BARENBLATT

Introduction to Symmetry Analysis

BRIAN J. CANTWELL
Hydrodynamic Instabilities
FRANÇOIS CHARRU

A First Course in Continuum Mechanics

OSCAR GONZALEZ & ANDREW M. STUART

Theory of Vortex Sound

M. S. HOWE
Applied Solid Mechanics
PETER HOWELL, GREGORY KOZYREFF & JOHN OCKENDON
Practical Applied Mathematics: Modelling, Analysis, Approximation
SAM HOWISON
A First Course in the Numerical Analysis of Differential Equations (2nd Edition)
ARIEH ISERLES

A First Course in Combinatorial Optimization

JON LEE
An Introduction to Parallel and Vector Scientific Computation
RONALD W. SHONKWILER & LEW LEFTON

© in this web service Cambridge University Press www.cambridge.org

Cambridge University Press
978-1-107-60104-8 - The Mathematics of Signal Processing
Steven B. Damelin and Willard Miller
Frontmatter
More information

The Mathematics of Signal Processing

S T E V E N B. D A M E L I N
Unit of Advances in Mathematics and its Applications

W I L L A R D M I L L E R, J R.
University of Minnesota

© in this web service Cambridge University Press www.cambridge.org

Cambridge University Press
978-1-107-60104-8 - The Mathematics of Signal Processing
Steven B. Damelin and Willard Miller
Frontmatter
More information

cambridge university press

Cambridge, New York, Melbourne, Madrid, Cape Town,
Singapore, São Paulo, Delhi, Mexico City
Cambridge University Press
The Edinburgh Building, Cambridge CB2 8RU, UK

Published in the United States of America by Cambridge University Press, New York

www.cambridge.org
Information on this title: www.cambridge.org/9781107013223


c S. B. Damelin and W. Miller, Jr. 2012

This publication is in copyright. Subject to statutory exception

and to the provisions of relevant collective licensing agreements,
no reproduction of any part may take place without the written
permission of Cambridge University Press.

First published 2012

Printed in the United Kingdom at the University Press, Cambridge

A catalog record for this publication is available from the British Library

ISBN 978-1-107-01322-3 Hardback

ISBN 978-1-107-60104-8 Paperback

Cambridge University Press has no responsibility for the persistence or

accuracy of URLs for external or third-party internet websites referred to
in this publication, and does not guarantee that any content on such
websites is, or will remain, accurate or appropriate.

© in this web service Cambridge University Press www.cambridge.org

Cambridge University Press
978-1-107-60104-8 - The Mathematics of Signal Processing
Steven B. Damelin and Willard Miller
Frontmatter
More information

Dedicated to our parents, children and partners

© in this web service Cambridge University Press www.cambridge.org

Cambridge University Press
978-1-107-60104-8 - The Mathematics of Signal Processing
Steven B. Damelin and Willard Miller
Frontmatter
More information

© in this web service Cambridge University Press www.cambridge.org

Cambridge University Press
978-1-107-60104-8 - The Mathematics of Signal Processing
Steven B. Damelin and Willard Miller
Frontmatter
More information

Contents

Preface page xi

Introduction 1
1 Normed vector spaces 3
1.1 Definitions 4
1.2 Inner products and norms 10
1.3 Finite-dimensional `p spaces 14
1.4 Digging deeper: completion of inner product spaces 20
1.5 Hilbert spaces, L2 and `2 25
1.6 Orthogonal projections, Gram–Schmidt
orthogonalization 39
1.7 Linear operators and matrices, LS approximations 46
1.8 Additional exercises 64
2 Analytic tools 73
2.1 Improper integrals 73
2.2 The gamma functions and beta functions 78
2.3 The sinc function and its improper relatives 79
2.4 Infinite products 83
2.5 Additional exercises 86
3 Fourier series 92
3.1 Definitions, real Fourier series and complex Fourier
series 92
3.2 Examples 96
3.3 Intervals of varying length, odd and even functions 97
3.4 Convergence results 99
3.5 More on pointwise convergence, Gibbs phenomena 107
3.6 Further properties of Fourier series 113

© in this web service Cambridge University Press www.cambridge.org

Cambridge University Press
978-1-107-60104-8 - The Mathematics of Signal Processing
Steven B. Damelin and Willard Miller
Frontmatter
More information

viii Contents

3.7 Digging deeper: arithmetic summability and Fejér’s

theorem 116
3.8 Additional exercises 123
4 The Fourier transform 127
4.1 Fourier transforms as integrals 127
4.2 The transform as a limit of Fourier series 129
4.3 L2 convergence of the Fourier transform 135
4.4 The Riemann–Lebesgue lemma and pointwise
convergence 140
4.5 Relations between Fourier series and integrals:
sampling 146
4.6 Fourier series and Fourier integrals: periodization 152
4.7 The Fourier integral and the uncertainty principle 154
4.8 Digging deeper 157
4.9 Additional exercises 161
5 Compressive sampling 164
5.1 Introduction 164
5.2 Algebraic theory of compressive sampling 168
5.3 Analytic theory of compressive sampling 172
5.4 Probabilistic theory of compressive sampling 183
5.5 Discussion and practical implementation 201
5.6 Additional exercises 206
6 Discrete transforms 208
6.1 Z transforms 208
6.2 Inverse Z transforms 211
6.3 Difference equations 213
6.4 Discrete Fourier transform and relations to Fourier
series 214
6.5 Fast Fourier transform (FFT) 222
6.6 Approximation to the Fourier transform 223
6.7 Additional exercises 224
7 Linear filters 230
7.1 Discrete linear filters 230
7.2 Continuous filters 233
7.3 Discrete filters in the frequency domain 235
7.4 Other operations on discrete signals 238
7.5 Additional exercises 240
8 Windowed Fourier and continuous
wavelet transforms. Frames 242
8.1 The windowed Fourier transform 243

© in this web service Cambridge University Press www.cambridge.org

Cambridge University Press
978-1-107-60104-8 - The Mathematics of Signal Processing
Steven B. Damelin and Willard Miller
Frontmatter
More information

Contents ix

8.2 Bases and frames, windowed frames 251

8.3 Affine frames 268
8.4 Additional exercises 270
9 Multiresolution analysis 272
9.1 Haar wavelets 272
9.2 The multiresolution structure 284
9.3 Filter banks and reconstruction of signals 296
9.4 The unitary two-channel filter bank system 304
9.5 A perfect reconstruction filter bank with N = 1 306
9.6 Perfect reconstruction for two-channel filter banks 307
9.7 Halfband filters and spectral factorization 309
9.8 Maxflat filters 312
9.9 Low pass iteration and the cascade algorithm 317
9.10 Scaling functions by recursion: dyadic points 320
9.11 The cascade algorithm in the frequency domain 329
9.12 Some technical results 332
9.13 Additional exercises 335
10 Discrete wavelet theory 341
10.1 L2 convergence 345
10.2 Accuracy of approximation 354
10.3 Smoothness of scaling functions and wavelets 359
10.4 Additional exercises 365
11 Biorthogonal filters and wavelets 367
11.1 Resumé of basic facts on biorthogonal filters 367
11.2 Biorthogonal wavelets: multiresolution structure 370
11.3 Splines 382
11.4 Generalizations of filter banks and wavelets 390
11.5 Finite length signals 395
11.6 Circulant matrices 397
11.7 Additional exercises 400
12 Parsimonious representation of data 401
12.1 The nature of digital images 402
12.2 Pattern recognition and clustering 418
12.3 Image representation of data 426
12.4 Image compression 429
12.5 Additional exercises 433

References 437
Index 443

© in this web service Cambridge University Press www.cambridge.org

Cambridge University Press
978-1-107-60104-8 - The Mathematics of Signal Processing
Steven B. Damelin and Willard Miller
Frontmatter
More information

© in this web service Cambridge University Press www.cambridge.org

Cambridge University Press
978-1-107-60104-8 - The Mathematics of Signal Processing
Steven B. Damelin and Willard Miller
Frontmatter
More information

Preface

Basically, this is a book about mathematics, pitched at the advanced

undergraduate/beginning graduate level, where ideas from signal pro-
cessing are used to motivate much of the material, and applications
of the theory to signal processing are featured. It is meant for math
students who are interested in potential applications of mathematical
structures and for students from the fields of application who want to
understand the mathematical foundations of their subject. The first few
chapters cover rather standard material in Fourier analysis, functional
analysis, probability theory and linear algebra, but the topics are care-
fully chosen to prepare the student for the more technical applications
to come. The mathematical core is the treatment of the linear system
y = Φx in both finite-dimensional and infinite-dimensional cases. This
breaks up naturally into three categories in which the system is deter-
mined, overdetermined or underdetermined. Each has different mathe-
matical aspects and leads to different types of application. There are a
number of books with some overlap in coverage with this volume, e.g.,
[11, 15, 17, 19, 53, 69, 71, 72, 73, 82, 84, 95, 99, 101], and we have
profited from them. However, our text has a number of features, includ-
ing its coverage of subject matter, that together make it unique. An
important aspect of this book on the interface between fields is that it
is largely self-contained. Many such books continually refer the reader
elsewhere for essential background material. We have tried to avoid this.
We assume the reader has a basic familiarity with advanced calculus and
with linear algebra and matrix theory up through the diagonalization
of symmetric or self-adjoint matrices. Most of the remaining develop-
ment of topics is self-contained. When we do need to call on technical
results not proved in the text, we try to be specific. Little in the way
of formal knowledge about signal processing is assumed. Thus while

© in this web service Cambridge University Press www.cambridge.org

Cambridge University Press
978-1-107-60104-8 - The Mathematics of Signal Processing
Steven B. Damelin and Willard Miller
Frontmatter
More information

xii Preface

this means that many interesting topics cannot be covered in a text of

modest size, the topics that are treated have a logical coherence, and
the reader is not continually distracted by appeals to other books and
papers. There are many exercises. In most of the volume the logic of the
mathematical topics predominates, but in a few chapters, particularly
for compressive sensing and for parsimonious representation of data, the
issues in the area of application predominate and mathematical topics
are introduced as appropriate to tackle the applied problems. Some of
the sections, designated by “Digging deeper” are more technical and
can be mostly skipped on a first reading. We usually give a nontechnical
description of the principal results of these sections. The book is suffi-
ciently flexible to provide relatively easy access to new ideas for students
or instructors who wish to skip around, while filling in the background
details for others. We include a large list of references for the reader who
wants to “dig deeper.” In particular, this is the case in the chapter on
the parsimonious representation of data.
This book arose from courses we have both taught and from ongoing
research. The idea of writing the book originated while the first author
was a New Directions Professor of Imaging at the Institute for Mathe-
matics and its Applications, The University of Minnesota during the 05–
06 academic year. The authors acknowledge support from the National
Science Foundation; the Centre for High Performance Computing, Cape
Town; the Institute for Mathematics and its Applications, University
of Minnesota; the School of Computational and Applied Mathematics,
the University of the Witwatersrand, Johannesburg; Georgia Southern
University; and the United States Office of Airforce Research. We are in-
debted to a large number of colleagues and students who have provided
valuable feedback on this project, particularly Li Lin and Peter Mueller
who tested the compressive sensing algorithms. All figures in this book
were generated by us from open source programs such as CVX, Maple
or MATLAB, or from licensed MATLAB wavelet and signal processing
toolboxes.
In closing, we thank the staff at Cambridge University Press, especially
David Tranah and Jon Billam, for their support and cooperation during
the preparation of this volume and we look forward to working with
them on future projects.

© in this web service Cambridge University Press www.cambridge.org