SUPER-RESOLUTION

IMAGING
SUPER-RESOLUTION
IMAGING

Edited by
Peyman Milanfar

Boca Raton London New York

CRC Press is an imprint of the

Taylor & Francis Group, an informa business
CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742
© 2011 by Taylor & Francis Group, LLC
CRC Press is an imprint of Taylor & Francis Group, an Informa business

No claim to original U.S. Government works

Version Date: 20140514

International Standard Book Number-13: 978-1-4398-1931-9 (eBook - PDF)

This book contains information obtained from authentic and highly regarded sources. Reasonable
efforts have been made to publish reliable data and information, but the author and publisher cannot
assume responsibility for the validity of all materials or the consequences of their use. The authors and
publishers have attempted to trace the copyright holders of all material reproduced in this publication
and apologize to copyright holders if permission to publish in this form has not been obtained. If any
copyright material has not been acknowledged please write and let us know so we may rectify in any
future reprint.

Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced,
transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or
hereafter invented, including photocopying, microfilming, and recording, or in any information stor-
age or retrieval system, without written permission from the publishers.

For permission to photocopy or use material electronically from this work, please access www.copy-
right.com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222
Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that pro-
vides licenses and registration for a variety of users. For organizations that have been granted a photo-
copy license by the CCC, a separate system of payment has been arranged.

Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are
used only for identification and explanation without intent to infringe.
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com
and the CRC Press Web site at
http://www.crcpress.com
 To my wife Sheila, and
our children Leila and Sara
Contents

Preface xv

1 Image Super-Resolution: Historical Overview and Future

Challenges 1
Jianchao Yang and Thomas Huang
1.1 Introduction to Super-Resolution . . . . . . . . . . . . . . . . 1
1.2 Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 Techniques for Super-Resolution . . . . . . . . . . . . . . . . 5
1.3.1 Image Observation Model . . . . . . . . . . . . . . . . 5
1.3.2 Super-Resolution in the Frequency Domain . . . . . . 7
1.3.3 Interpolation-Restoration: Non-Iterative Approaches . 8
1.3.4 Statistical Approaches . . . . . . . . . . . . . . . . . . 9
1.3.4.1 Maximum Likelihood . . . . . . . . . . . . . 11
1.3.4.2 Maximum a Posteriori . . . . . . . . . . . . . 12
1.3.4.3 Joint MAP Restoration . . . . . . . . . . . . 13
1.3.4.4 Bayesian Treatments . . . . . . . . . . . . . 14
1.3.5 Example-Based Approaches . . . . . . . . . . . . . . . 15
1.3.6 Set Theoretic Restoration . . . . . . . . . . . . . . . . 18
1.4 Challenge Issues for Super-Resolution . . . . . . . . . . . . . 20
1.4.1 Image Registration . . . . . . . . . . . . . . . . . . . . 20
1.4.2 Computation Efficiency . . . . . . . . . . . . . . . . . 21
1.4.3 Robustness Aspects . . . . . . . . . . . . . . . . . . . 22
1.4.4 Performance Limits . . . . . . . . . . . . . . . . . . . 23
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

2 Super-Resolution Using Adaptive Wiener Filters 35

Russell C. Hardie
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
2.2 Observation Model . . . . . . . . . . . . . . . . . . . . . . . 38
2.2.1 Image Formation Model . . . . . . . . . . . . . . . . . 38
2.2.2 Image Motion Model . . . . . . . . . . . . . . . . . . . 41
2.2.3 Image Registration . . . . . . . . . . . . . . . . . . . . 42
2.2.4 System Point-Spread Function . . . . . . . . . . . . . 44
2.3 AWF SR Algorithms . . . . . . . . . . . . . . . . . . . . . . 47
2.4 Experimental Results . . . . . . . . . . . . . . . . . . . . . . 51
2.4.1 SR Results for Simulated Data . . . . . . . . . . . . . 51

vii
viii Contents

2.4.2 SR Results for Infrared Video Data . . . . . . . . . . . 53

2.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
2.6 Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . 58
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

3 Locally Adaptive Kernel Regression for Space-Time

Super-Resolution 63
Hiroyuki Takeda and Peyman Milanfar
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.2 Adaptive Kernel Regression . . . . . . . . . . . . . . . . . . . 67
3.2.1 Classic Kernel Regression in 2-D . . . . . . . . . . . . 67
3.2.2 Steering Kernel Regression in 2-D . . . . . . . . . . . 70
3.2.3 Space-Time (3-D) Steering Kernel Regression . . . . . 72
3.2.4 Kernel Regression with Rough Motion Compensation 78
3.2.5 Implementation and Iterative Refinement . . . . . . . 80
3.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
3.3.1 Spatial Upscaling Examples . . . . . . . . . . . . . . . 83
3.3.2 Spatiotemporal Upscaling Examples . . . . . . . . . . 86
3.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
3.5 Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
3.5.1 Steering Kernel Parameters . . . . . . . . . . . . . . . 91
3.5.2 The Choice of the Regression Parameters . . . . . . . 91
3.5.3 Deblurring . . . . . . . . . . . . . . . . . . . . . . . . 92
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

4 Super-Resolution with Probabilistic Motion Estimation 97

Matan Protter and Michael Elad
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
4.2 Classic Super-Resolution: Background . . . . . . . . . . . . . 99
4.3 The Proposed Algorithm . . . . . . . . . . . . . . . . . . . . 101
4.3.1 The New Formulation . . . . . . . . . . . . . . . . . . 101
4.3.2 Separating the Blur Treatment . . . . . . . . . . . . . 103
4.3.3 The Algorithm: A Matrix-Vector Version . . . . . . . 104
4.3.4 The Algorithm: A Pixel-Wise Version . . . . . . . . . 104
4.3.5 Computing the Weights . . . . . . . . . . . . . . . . . 106
4.3.6 Other Resampling Tasks . . . . . . . . . . . . . . . . . 107
4.4 Experimental Validation . . . . . . . . . . . . . . . . . . . . 108
4.4.1 Experimental Results . . . . . . . . . . . . . . . . . . 108
4.4.2 Computational Complexity . . . . . . . . . . . . . . . 116
4.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
 Contents ix

5 Spatially Adaptive Filtering as Regularization in Inverse

Imaging: Compressive Sensing, Super-Resolution, and
Upsampling 123
Aram Danielyan, Alessandro Foi, Vladimir Katkovnik, and Karen
Egiazarian
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
5.2 Iterative Filtering as Regularization . . . . . . . . . . . . . . 125
5.2.1 Spectral Decomposition of the Operator . . . . . . . . 126
5.2.2 Nonlocal Transform Domain Filtering . . . . . . . . . 126
5.3 Compressed Sensing . . . . . . . . . . . . . . . . . . . . . . . 129
5.3.1 Observation Model and Notation . . . . . . . . . . . . 130
5.3.2 Iterative Algorithm with Stochastic Approximation . . 130
5.3.2.1 Comments on the Algorithm . . . . . . . . . 132
5.3.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . 133
5.3.3.1 Radon Inversion from Sparse Projections . . 134
5.3.3.2 Limited-Angle Tomography . . . . . . . . . . 134
5.3.3.3 Reconstruction from Low-Frequency Data . . 134
5.4 Super-Resolution . . . . . . . . . . . . . . . . . . . . . . . . . 137
5.4.1 Spectral Decomposition for the Super-Resolution
Problem . . . . . . . . . . . . . . . . . . . . . . . . . . 139
5.4.2 Observation Model . . . . . . . . . . . . . . . . . . . . 140
5.4.3 Scaling Family of Transforms . . . . . . . . . . . . . . 140
5.4.4 Multistage Iterative Reconstruction . . . . . . . . . . 142
5.4.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . 143
5.4.5.1 Implementation Details . . . . . . . . . . . . 144
5.4.5.2 Super-Resolution . . . . . . . . . . . . . . . . 144
5.4.5.3 Image Upsampling . . . . . . . . . . . . . . . 145
5.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150

6 Registration for Super-Resolution: Theory, Algorithms, and

Applications in Image and Mobile Video Enhancement 155
Patrick Vandewalle, Luciano Sbaiz, and Martin Vetterli
6.1 Camera Model . . . . . . . . . . . . . . . . . . . . . . . . . . 157
6.2 What Is Resolution? . . . . . . . . . . . . . . . . . . . . . . . 161
6.3 Super-Resolution as a Multichannel Sampling Problem . . . 162
6.3.1 Fourier Series . . . . . . . . . . . . . . . . . . . . . . . 163
6.4 Registration of Totally Aliased Signals . . . . . . . . . . . . 166
6.4.1 Variable Projection Method . . . . . . . . . . . . . . . 166
6.4.2 Frequency Analysis Method . . . . . . . . . . . . . . . 167
6.4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . 168
6.5 Registration of Partially Aliased Signals . . . . . . . . . . . . 168
6.5.1 Super-Resolution Using Frequency Domain
Registration . . . . . . . . . . . . . . . . . . . . . . . . 168
x Contents

6.5.1.1 Image Registration . . . . . . . . . . . . . . . 168

6.5.1.2 Image Reconstruction . . . . . . . . . . . . . 170
6.5.1.3 Results . . . . . . . . . . . . . . . . . . . . . 170
6.5.2 Super-Resolution from Low-Quality Videos . . . . . . 176
6.5.2.1 Motion Model . . . . . . . . . . . . . . . . . 176
6.5.2.2 Image Registration . . . . . . . . . . . . . . . 177
6.5.2.3 Image Reconstruction . . . . . . . . . . . . . 179
6.5.2.4 Results on Video Sequences . . . . . . . . . . 181
6.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184

7 Towards Super-Resolution in the Presence of Spatially

Varying Blur 187
Michal Šorel, Filip Šroubek, and Jan Flusser
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
7.1.1 Representation of Spatially Varying PSF . . . . . . . . 189
7.1.2 General Model of Resolution Loss . . . . . . . . . . . 189
7.1.3 Bayesian View of Solution . . . . . . . . . . . . . . . . 191
7.2 Defocus and Optical Aberrations . . . . . . . . . . . . . . . . 194
7.2.1 Geometrical Optics . . . . . . . . . . . . . . . . . . . . 195
7.2.2 Approximation of PSF by 2D Gaussian Function . . . 197
7.2.3 General Form of PSF for Axially-Symmetric Optical
Systems . . . . . . . . . . . . . . . . . . . . . . . . . . 197
7.2.4 Diffraction . . . . . . . . . . . . . . . . . . . . . . . . 198
7.2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . 201
7.3 Camera Motion Blur . . . . . . . . . . . . . . . . . . . . . . 202
7.3.1 Rotation . . . . . . . . . . . . . . . . . . . . . . . . . . 202
7.3.2 No Rotation . . . . . . . . . . . . . . . . . . . . . . . . 203
7.4 Scene Motion . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
7.5 Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
7.5.1 Super-Resolution of a Scene with Local Motion . . . . 205
7.5.2 Smoothly Changing Blur . . . . . . . . . . . . . . . . 207
7.5.3 Depth-Dependent Blur . . . . . . . . . . . . . . . . . . 210
7.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
7.7 Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . 215
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215

8 Toward Robust Reconstruction-Based Super-Resolution 219

Masayuki Tanaka and Masatoshi Okutomi
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
8.2 Overviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
8.2.1 Super-Resolution Reconstruction . . . . . . . . . . . . 221
8.2.2 Robust SR Reconstruction . . . . . . . . . . . . . . . 222
8.2.3 Robust Registration . . . . . . . . . . . . . . . . . . . 224
 Contents xi

8.3 Robust SR Reconstruction with Pixel Selection . . . . . . . . 225

8.3.1 Displacement and Similarity Measure . . . . . . . . . 225
8.3.2 Proposed Pixel Selection Algorithm . . . . . . . . . . 227
8.3.2.1 Pixel Selection Based on Similarity Measure
and Displacement Estimation . . . . . . . . . 227
8.3.3 Luminance Correction . . . . . . . . . . . . . . . . . . 229
8.3.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . 230
8.4 Robust Super-Resolution Using MPEG Motion Vectors . . . 232
8.4.1 Registration Using MPEG Motion Vectors . . . . . . . 232
8.4.2 Experiments of Robust SR Reconstruction . . . . . . . 233
8.5 Robust Registration for Super-Resolution . . . . . . . . . . . 237
8.5.1 Proposed Multiple Motion Estimation . . . . . . . . . 238
8.5.1.1 Motion Estimation and Region Extraction for
a Single Object . . . . . . . . . . . . . . . . . 239
8.5.1.2 Multiple Motion Estimation . . . . . . . . . 240
8.5.2 Super-Resolution for Multiple Motions . . . . . . . . . 241
8.5.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . 241
8.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244

9 Multiframe Super-Resolution from a Bayesian Perspective 247

Lyndsey Pickup, Stephen Roberts, Andrew Zisserman,
and David Capel
9.1 The Generative Model . . . . . . . . . . . . . . . . . . . . . . 248
9.1.1 Considerations in the Forward Model . . . . . . . . . 249
9.1.2 A Probabilistic Setting . . . . . . . . . . . . . . . . . . 251
9.1.2.1 The Maximum Likelihood Solution . . . . . . 251
9.1.2.2 The ML Solution in Practice . . . . . . . . . 252
9.1.2.3 The Maximum a Posteriori Solution . . . . . 254
9.1.3 Selected Priors Used in MAP Super-Resolution . . . . 255
9.2 Where Super-Resolution Algorithms Go Wrong . . . . . . . . 258
9.2.1 Point-Spread Function Example . . . . . . . . . . . . 259
9.2.2 Photometric Registration Example . . . . . . . . . . . 261
9.2.3 Geometric Registration Example . . . . . . . . . . . . 262
9.3 Simultaneous Super-Resolution . . . . . . . . . . . . . . . . . 263
9.3.1 Super-Resolution with Registration . . . . . . . . . . . 264
9.3.2 Learning Prior Strength Parameters from Data . . . . 265
9.3.3 Scaling and Convergence . . . . . . . . . . . . . . . . . 266
9.3.4 Initialization . . . . . . . . . . . . . . . . . . . . . . . 267
9.3.5 Evaluation on Synthetic Data . . . . . . . . . . . . . . 269
9.3.6 Experiments on Real Data . . . . . . . . . . . . . . . . 271
9.4 Bayesian Marginalization . . . . . . . . . . . . . . . . . . . . 273
9.4.1 Marginalizing over Registration Parameters . . . . . . 274
9.4.2 Marginalizing over the High-Resolution Image . . . . . 277
xii Contents

9.4.3 Implementation Notes . . . . . . . . . . . . . . . . . . 278

9.4.4 Experimental Evaluation . . . . . . . . . . . . . . . . 279
9.4.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 282
9.5 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . 282
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283

10 Variational Bayesian Super-Resolution Reconstruction 285

S. Derin Babacan, Rafael Molina, and Aggelos K. Katsaggelos
10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
10.2 Problem Formulation . . . . . . . . . . . . . . . . . . . . . . 288
10.3 Bayesian Framework for Super-Resolution . . . . . . . . . . . 288
10.3.1 Observation Models . . . . . . . . . . . . . . . . . . . 289
10.3.2 Image Models . . . . . . . . . . . . . . . . . . . . . . . 290
10.3.3 Blur Models . . . . . . . . . . . . . . . . . . . . . . . . 291
10.3.4 Motion (Registration) Models . . . . . . . . . . . . . . 292
10.3.5 Hyperpriors on the Hyperparameters . . . . . . . . . . 292
10.4 Bayesian Inference . . . . . . . . . . . . . . . . . . . . . . . . 293
10.5 Variational Bayesian Inference Using TV Image Priors . . . . 296
10.5.1 Estimation of the HR Image Distribution . . . . . . . 297
10.5.2 Estimation of the Hyperparameter Distributions . . . 298
10.6 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . 301
10.7 Estimation of Motion and Blur . . . . . . . . . . . . . . . . . 305
10.8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
10.9 Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . 309
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309

11 Pattern Recognition Techniques for Image Super-Resolution 315

Karl Ni and Truong Q. Nguyen
11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 316
11.2 Nearest Neighbor Super-Resolution . . . . . . . . . . . . . . 318
11.2.1 k-Nearest Neighbor . . . . . . . . . . . . . . . . . . . . 320
11.2.2 k-Nearest Neighbor Regression . . . . . . . . . . . . . 321
11.2.3 Adaptive k-NN for Super-Resolution . . . . . . . . . . 323
11.2.4 Heuristics for Insufficient Training in Adaptive k-NN
Regression . . . . . . . . . . . . . . . . . . . . . . . . . 325
11.3 Markov Random Fields and Approximations . . . . . . . . . 326
11.4 Kernel Machines for Image Super-Resolution . . . . . . . . . 329
11.4.1 Support Vector Regression . . . . . . . . . . . . . . . 330
11.4.2 Inductively Learning the Kernel Matrix for Regression 332
11.4.3 The Quadratically Constrained Quadratic
Programming Problem . . . . . . . . . . . . . . . . . . 335
11.4.4 Applications to Super-Resolution . . . . . . . . . . . . 337
11.5 Multiple Learners and Multiple Regressions . . . . . . . . . . 338
11.5.1 Neural Networks and Super-Resolution . . . . . . . . . 339
 Contents xiii

11.5.2 Unsupervised Clustering . . . . . . . . . . . . . . . . . 340

11.5.3 Supervised Clustering . . . . . . . . . . . . . . . . . . 342
11.5.4 Integrating Regression . . . . . . . . . . . . . . . . . . 343
11.6 Design Considerations and Examples . . . . . . . . . . . . . 346
11.7 Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 348
11.8 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349

12 Super-Resolution Reconstruction of Multichannel Images 355

Osman G. Sezer and Yucel Altunbasak
12.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 356
12.2 Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 358
12.3 Image Acquisition Model . . . . . . . . . . . . . . . . . . . . 360
12.3.1 Motion Compensation . . . . . . . . . . . . . . . . . . 361
12.3.2 Spatial Filtering . . . . . . . . . . . . . . . . . . . . . 362
12.3.3 Spectral Filtering . . . . . . . . . . . . . . . . . . . . . 363
12.3.4 Multichannel Observation Model . . . . . . . . . . . . 365
12.4 Subspace Representation . . . . . . . . . . . . . . . . . . . . 366
12.4.1 Blind Source Separation . . . . . . . . . . . . . . . . . 367
12.4.2 Observation Model with BSS . . . . . . . . . . . . . . 368
12.5 Reconstruction Algorithm . . . . . . . . . . . . . . . . . . . . 369
12.5.1 The Subspace Observation Model . . . . . . . . . . . 370
12.5.2 POCS with Outliers of Residual . . . . . . . . . . . . 371
12.5.3 POCS with Variance of Residual . . . . . . . . . . . . 372
12.6 Experiments and Discussions . . . . . . . . . . . . . . . . . . 373
12.6.1 Spectral Subspace . . . . . . . . . . . . . . . . . . . . 374
12.6.2 Robustness against Noise . . . . . . . . . . . . . . . . 375
12.6.3 Simultaneous Spatial and Spectral Super-Resolution . 376
12.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 380

13 New Applications of Super-Resolution in Medical Imaging 383

M. Dirk Robinson, Stephanie J. Chiu, Cynthia A. Toth, Joseph
A. Izatt, Joseph Y. Lo, and Sina Farsiu
13.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 384
13.2 The Super-Resolution Framework . . . . . . . . . . . . . . . 385
13.2.1 Image Capture Model . . . . . . . . . . . . . . . . . . 385
13.2.2 Super-Resolution Estimation Framework . . . . . . . . 387
13.3 New Medical Imaging Applications . . . . . . . . . . . . . . 388
13.3.1 Super-Resolution in Low Radiation Digital X-Ray
Mammography . . . . . . . . . . . . . . . . . . . . . . 389
13.3.1.1 Multiframe Shift Estimation . . . . . . . . . 391
13.3.1.2 Multiframe ForWaRD Deconvolution and
Denoising . . . . . . . . . . . . . . . . . . . . 393
xiv Contents

13.3.1.3 Experimental X-Ray Results . . . . . . . . . 394

13.3.2 Super-Resolution in Optical Coherence Tomography . 397
13.3.2.1 Proposed Method: Sparse Repeated Imaging 399
13.3.2.2 Multiframe Joint Registration . . . . . . . . 401
13.3.2.3 Experimental Results . . . . . . . . . . . . . 403
13.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405
13.5 Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . 406
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407

14 Practicing Super-Resolution: What Have We Learned? 413

Nikola Bozinovic
14.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414
14.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 414
14.2.1 Video Quality Trends . . . . . . . . . . . . . . . . . . 415
14.2.2 The Need for Postprocessing . . . . . . . . . . . . . . 415
14.2.3 Why is Super-Resolution Not Used More? . . . . . . . 417
14.2.4 Automation versus User Interaction . . . . . . . . . . 418
14.2.5 Modeling Motion for Super-Resolution . . . . . . . . . 418
14.2.6 Performance Issues . . . . . . . . . . . . . . . . . . . . 419
14.2.7 Relationship to Existing Standards . . . . . . . . . . . 419
14.3 MotionDSP: History and Concepts . . . . . . . . . . . . . . . 420
14.3.1 Design Decisions . . . . . . . . . . . . . . . . . . . . . 421
14.4 Markets and Applications . . . . . . . . . . . . . . . . . . . . 422
14.4.1 Forensic and Real-Time Markets: MotionDSP’s Ikena 423
14.4.2 Consumers: MotionDSP’s vReveal . . . . . . . . . . . 425
14.5 Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427
14.5.1 Robust Parametric Motion Estimation . . . . . . . . . 428
14.6 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 431
14.6.1 Mobile and Digital Still Camera Video . . . . . . . . . 432
14.6.2 DV and HD Video . . . . . . . . . . . . . . . . . . . . 433
14.6.3 Handling Complex Motion . . . . . . . . . . . . . . . . 433
14.6.4 Practical Limits of Super-Resolution . . . . . . . . . . 435
14.7 Lessons Learned . . . . . . . . . . . . . . . . . . . . . . . . . 437
14.8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444

Index 449
Preface

“Enhance 34 to 36. Pan right and pull back. Stop. Enhance 34 to 46. Pull
back. Wait a minute, go right, stop. Enhance 57 to 19. Track 45 left. Stop.
Enhance 15 to 23. Give me a hard copy right there.”

– Harrison Ford in Blade Runner, 1982

If you have never seen the movie Blade Runner, you should. Aside from be-
ing one of the greatest science fiction films of all time, it is uniquely relevant
to the subject of this book: almost 30 years ago, the opening scene of this
movie foresaw the idea of super-resolution. In the intervening years, a great
deal has transpired: computing power has increased by orders of magnitude,
digital cameras are everywhere, and of course digital displays have become
magnificently detailed. Along with these advances, the public’s expectations
for high-quality imagery has naturally intensified, often out of proportion with
the state-of-the-art technology. In fact, in the last few years, the visual quality
of captured images and video has not kept pace with these lofty expectations.
By packing increasingly larger number of pixels into ever smaller spaces, and
using less sophisticated optical elements, public, commercial, and official users
alike have seen an overall decline in the visual quality of their recorded con-
tent. So despite what might at first seem like a losing battle against better
and cheaper sensors, super-resolution technology (and image enhancement
more generally) has really become more relevant than ever. Given that almost
all recorded visual content is now enhanced in one form or another by just
about every digital camera sold today, it is not entirely outrageous then to
believe that before long, super-resolution will become the ”killer application”
for imaging.
Ironically, only two years after the release of Blade Runner, the semi-
nal paper by Tsai and Huang kick-started the modern idea of computational
super-resolution. While results in sampling theory dating as far back as the
’50s (Yan) and the ’70s (Papoulis) had hinted at the idea, it was Tsai and
Huang who explicitly showed that, at least in theory, it was possible to im-
prove resolution by registering and fusing multiple images. The rest, as they
say, is history. We are fortunate to be able to write a little bit of that history
in this book. In the last five years or so, the field of super-resolution imaging
has truly flourished both academically and commercially. The growing impor-
tance of super-resolution imaging has manifested itself in an explosive growth
in the number of papers in this area and citations to these papers (a few dozen

xv
xvi Preface

in 1994, to more than 500 in 2004, and to more than 2000 in 2008). What has
been missing, however, is a book to not only gather key recent contributions
in one place, but also to serve as a starting point for those interested in this
field to begin learning about and exploring the state of the art. This is what
this book hopes to accomplish.
As is probably well-known by now, every super-resolution algorithm ever
developed is sabotaged by at least one spoke of our triumvirate “axis of evil”:
the need for (1) (subpixel) accurate motion estimation, (2) (spatially varying)
deblurring, and (3) robustness to modeling and stochastic errors. To be sure,
these are not independent problems and should ideally be treated in unison
(ambitious graduate students take note!). But realistically, each is sufficiently
complex as to merit its own section in the library, or at least a couple of nice
chapters in this book. This books gathers contributions that will present the
reader with a snapshot of where the field stands, a reasonable idea of where the
field is heading—and perhaps where it should be heading. Chapter 1 provides
an introduction to the history of the subject that should be of broad interest.
Indeed, the collection of citations summarized in this chapter is an excellent
wellspring for continued research on super-resolution.
One of the most active areas of work in image and video enhancement
in recent years has been the subject of locally adaptive processing methods,
which are discussed in Chapters 2, 3, 4, and 5. In contrast to globally optimal
methods (treated later in the book), these methods are built on the notion
that processing should be strongly tailored to the local behavior of the given
data. An explicit goal in some cases, and a happy consequence in others,
local processing enables us to largely avoid direct and detailed estimation of
motion. Readers interested in methods for explicit motion estimation will find
an excellent overview of modern techniques in Chapter 6.
While motion estimation is typically the first step in many super-resolution
algorithms, deblurring is typically the last step. Unfortunately, having been
relegated to the last position has meant that this important aspect of enhance-
ment has not received the respect and attention it deserves. Despite heavy
recent activity in both the image processing and machine vision community,
and some notable successes, deblurring even in its simplest (space-invariant,
known point-spread-function) form is still largely an unsolved problem. Inas-
much as we would like to hope, blur almost never manifests itself in a spatially
uniform fashion. In Chapter 7, the reader will find a well-motivated and di-
rect attack at this challenging problem. Despite our best efforts, a sequential
approach to super-resolution consisting of motion estimation, fusion, and de-
blurring will always be subject to the vagaries of the data, the models, and
noise. As such, building robustness into the reconstruction process, as treated
in Chapter 8, is vital if the algorithm is to be practically useful.
As with most inverse problems, super-resolution is highly ill-posed. In the
most general case, the motion between the frames, the blur kernel(s), and the
high-resolution image of interest are three interwoven unknowns that should
ideally be estimated together (rather than sequentially), and whose effect is
 Preface xvii

directly felt in the three points of weakness to which I referred earlier. Prin-
cipled Bayesian statistical approaches addressing these issues are presented
in Chapters 9 and 10, where the ever important prior information is brought
to bear on the super-resolution problem. Of course, prior information can be
brought to the table either in “bulk” form as a statistical distribution, or in
more specific “piecemeal” form as examples. Naively speaking, this distinction
is indeed what leads us to learning-based methods described in Chapter 11.
In the final three chapters of the book, we concentrate on applications.
Among the many areas of science to which super-resolution has been success-
fully applied in recent years, medicine and remote sensing have perhaps seen
the most direct impact. In Chapter 12, a novel application of super-resolution
to massive multispectral remote-sensing data sets is detailed. Medical imaging
applications of super-resolution in Chapter 13 discuss two important problems.
In what should be good news to everyone, high resolution X-ray imaging is
made possible at lower radiation dosages thanks to super-resolution. In an-
other application, detailed imaging of the retina is made possible for diagnostic
purposes. Finally, in Chapter 14, a successful commercial application of super-
resolution (in which I was fortunate to have a hand) is discussed. This chapter
is quite informative not only because of the interesting perspective it provides,
but also because of the valuable practical nuggets it imparts to the reader.
It is an interesting glimpse into what it really takes to make super-resolution
tick.
Perhaps it is worth saying a few words about how this book can be used.
As with any edited volume, it is intended to provide a snapshot of the field,
which is sure to evolve over time. Yet, I have endeavored to organize the
chapters to be used as a teaching tool as well. Indeed, the first four chapters
can easily be incorporated into the latter part of a graduate-level course on
image processing. Other selected chapters of the book can be used to offer
short courses on the subject to a wide audience of engineers and scientists.
The book as a whole can also be used as a text for a semester-long focused
seminar course on the topic, with one or two lectures dedicated to each chapter.
It is hoped that over time the authors may provide supplementary material
for each chapter, including slides, code, or data, which will be archived at the
book Website – so the interested reader is encouraged to revisit the site.
This book is the collective effort of a kind group of friends and colleagues.
I am grateful to each of the authors for giving generously of their time and
contributing to this book. I am also thankful to my students past and present
for their contributions to this topic, and to this book in particular. Specifically,
I acknowledge (soon to be Dr.) Hiroyuki Takeda for his assistance with myriad
LATEX issues.
It is my hope that this book will help to promote this field of endeavor for
many years to come.
Peyman Milanfar
Menlo Park, March 2010
1
Image Super-Resolution: Historical Overview
and Future Challenges

Jianchao Yang
University of Illinois at Urbana-Champaign

Thomas Huang
University of Illinois at Urbana-Champaign

CONTENTS
1.1 Introduction to Super-Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 Techniques for Super-Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3.1 Image Observation Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3.2 Super-Resolution in the Frequency Domain . . . . . . . . . . . . . . . . . . . . . . 7
1.3.3 Interpolation-Restoration: Non-Iterative Approaches . . . . . . . . . . . . 8
1.3.4 Statistical Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.3.4.1 Maximum Likelihood . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.3.4.2 Maximum a Posteriori . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.3.4.3 Joint MAP Restoration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.3.4.4 Bayesian Treatments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.3.5 Example-Based Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.3.6 Set Theoretic Restoration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
1.4 Challenge Issues for Super-Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
1.4.1 Image Registration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
1.4.2 Computation Efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
1.4.3 Robustness Aspects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
1.4.4 Performance Limits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

1.1 Introduction to Super-Resolution

In most digital imaging applications, high-resolution images or videos are
usually desired for later image processing and analysis. The desire for high-
resolution stems from two principal application areas: improvement of pictorial

1
2 Super-Resolution Imaging

FIGURE 1.1: The 1951 USAF resolution test target, a classic test target used
to determine spatial resolution of imaging sensors and imaging systems.

information for human interpretation; and helping representation for auto-

matic machine perception. Image resolution describes the details contained in
an image, the higher the resolution, the more image details. The resolution
of a digital image can be classified in many different ways: pixel resolution,
spatial resolution, spectral resolution, temporal resolution, and radiometric
resolution. In this context, we are mainly interested in spatial resolution.
Spatial resolution: a digital image is made up of small picture elements
called pixels. Spatial resolution refers to the pixel density in an image and
measures in pixels per unit area. Figure 1.1 shows a classic test target to
determine the spatial resolution of an imaging system.
The image spatial resolution is first limited by the imaging sensors
or the imaging acquisition device. A modern image sensor is typically a
charge-coupled device (CCD) or a complementary metal-oxide-semiconductor
(CMOS) active-pixel sensor. These sensors are typically arranged in a two-
dimensional array to capture two-dimensional image signals. The sensor size
or equivalently the number of sensor elements per unit area in the first place
determines the spatial resolution of the image to capture. The higher density
of the sensors, the higher spatial resolution possible of the imaging system. An
imaging system with inadequate detectors will generate low-resolution images
with blocky effects, due to the aliasing from low spatial sampling frequency.
In order to increase the spatial resolution of an imaging system, one straight-
forward way is to increase the sensor density by reducing the sensor size.
However, as the sensor size decreases, the amount of light incident on each
 Image Super-Resolution: Historical Overview and Future Challenges 3

sensor also decreases, causing the so-called shot noise. Also, the hardware cost
of a sensor increases with the increase of sensor density or corresponding im-
age pixel density. Therefore, the hardware limitation on the size of the sensor
restricts the spatial resolution of an image that can be captured.
While the image sensors limit the spatial resolution of the image, the
image details (high-frequency bands) are also limited by the optics, due to lens
blurs (associated with the sensor point spread function (PSF)), lens aberration
effects, aperture diffractions, and optical blurring due to motion. Constructing
imaging chips and optical components to capture very high-resolution images
is prohibitively expensive and not practical in most real applications, e.g.,
widely used surveillance cameras and cell phone built-in cameras. Besides the
cost, the resolution of a surveillance camera is also limited in the camera speed
and hardware storage. In some other scenarios such as satellite imagery, it is
difficult to use high resolution sensors due to physical constraints. Another way
to address this problem is to accept the image degradations and use signal
processing to post-process the captured images, to trade off computational
cost with the hardware cost. These techniques are specifically referred to as
Super-Resolution (SR) reconstruction.
Super-Resolution (SR) are techniques that construct high-resolution (HR)
images from several observed low-resolution (LR) images, thereby increasing
the high-frequency components and removing the degradations caused by the
imaging process of the low-resolution camera. The basic idea behind SR is to
combine the non-redundant information contained in multiple low-resolution
frames to generate a high-resolution image. A closely related technique with
SR is the single-image interpolation approach, which can be also used to in-
crease the image size. However, since there is no additional information pro-
vided, the quality of the single-image interpolation is very much limited due to
the ill-posed nature of the problem, and the lost frequency components cannot
be recovered. In the SR setting, however, multiple low-resolution observations
are available for reconstruction, making the problem better constrained. The
nonredundant information contained in the these LR images is typically intro-
duced by subpixel shifts between them. These subpixel shifts may occur due to
uncontrolled motions between the imaging system and scene, e.g., movements
of objects, or due to controlled motions, e.g., the satellite imaging system
orbits the earth with predefined speed and path.
Each low-resolution frame is a decimated, aliased observation of the true
scene. SR is possible only if there exists subpixel motions between these low-
resolution frames,1 and thus the ill-posed upsampling problem can be better
conditioned. Figure 1.2 shows a simplified diagram describing the basic idea
of SR reconstruction. In the imaging process, the camera captures several
LR frames, which are downsampled from the HR scene with subpixel shifts
between each other. SR construction reverses this process by aligning the LR
1 The mainstream SR techniques rely on motions, although there are some works using

defocus as a cue.
4 Super-Resolution Imaging

FIGURE 1.2: The basic idea for super-resolution reconstruction from multiple
low-resolution frames. Subpixel motion provides the complementary informa-
tion among the low-resolution frames that makes SR reconstruction possible.

observations to subpixel accuracy and combining them into a HR image grid

(interpolation), thereby overcoming the imaging limitation of the camera. SR
(some of which described in this book), arises in many areas such as:

1. Surveillance video [20, 55]: frame freeze and zoom region of interest
(ROI) in video for human perception (e.g., look at the license plate
in the video), resolution enhancement for automatic target recognition
(e.g., try to recognize a criminal’s face).

2. Remote sensing [29]: several images of the same area are provided, and
an improved resolution image can be sought.

3. Medical imaging (CT, MRI, Ultrasound, etc.) [59, 70, 47, 60]: several
images limited in resolution quality can be acquired, and SR technique
can be applied to enhance the resolution.

4. Video standard conversion, e.g., from NTSC video signal to HDTV sig-
nal.

This chapter targets an introduction to the SR research area, by explaining

some basic techniques of SR, an overview of the literature, and discussions
about some challenging issues for future research.
 Image Super-Resolution: Historical Overview and Future Challenges 5

1.2 Notations
Before talking about SR techniques, we introduce the notations we use in this
chapter. Uppercase bold letters X and Y denote the vector form in lexico-
graphical order for HR and LR images respectively. Lowercase bold letters x
and y denote the vector form in lexicographical order for HR and LR image
patches respectively. Underlined uppercase bold letters are used to denote a
vector concatenation of multiple vectors, e.g., Y is a vector concatenation of
Yk (k = 1, 2, ..., K). We use plain uppercase symbols to denote matrices, and
plain lowercase symbols to denote scalars.

1.3 Techniques for Super-Resolution

SR reconstruction has been one of the most active research areas since the
seminal work by Tsai and Huang [99] in 1984. Many techniques have been
proposed over the last two decades [4, 65] representing approaches from fre-
quency domain to spatial domain, and from the signal processing perspective
to the machine learning perspective. Early works on super-resolution mainly
followed the theory of [99] by exploring the shift and aliasing properties of
the Fourier transform. However, these frequency domain approaches are very
restricted in the image observation model they can handle, and real problems
are much more complicated. Researchers nowadays most commonly address
the problem mainly in the spatial domain, for its flexibility to model all kinds
of image degradations. This section talks about these techniques, starting from
the image observation model.

1.3.1 Image Observation Model

The digital imaging system is not perfect due to hardware limitations, acquir-
ing images with various kinds of degradations. For example, the finite aperture
size causes optical blur, modeled by Point Spread Function (PSF). The finite
aperture time results in motion blur, which is very common in videos. The fi-
nite sensor size leads to sensor blur; the image pixel is generated by integration
over the sensor area instead of impulse sampling. The limited sensor density
leads to aliasing effects, limiting the spatial resolution of the achieved image.
These degradations are modeled fully or partially in different SR techniques.
Figure 1.3 shows a typical observation model relating the HR image with
LR video frames, as introduced in the literature [65, 82]. The input of the imag-
ing system is continuous natural scenes, well approximated as band-limited
signals. These signals may be contaminated by atmospheric turbulence be-
6 Super-Resolution Imaging

FIGURE 1.3: The observation model of a real imaging system relating a high
resolution image to the low-resolution observation frames with motion between
the scene and the camera.

fore reaching the imaging system. Sampling the continues signal beyond the
Nyquist rate generates the high resolution digital image (a) we desire. In
our SR setting, usually there exists some kind of motion between the cam-
era and scene to capture. The inputs to the camera are multiple frames of
the scene, connected by possibly local or global shifts, leading to image (b).
Going through the camera, these motion related high-resolution frames will
incur different kinds of blurring effects, such as optical blur and motion blur.
These blurred images (c) are then downsampled at the image sensors (e.g.,
CCD detectors) into pixels, by an integral of the image falling into each sen-
sor area. These downsampled images are further affected by the sensor noise
and color filtering noise. Finally, the frames captured by the low-resolution
imaging system are blurred, decimated, and noisy versions of the underlying
true scene.
Let X denote the HR image desired, i.e., the digital image sampled above
Nyquist sampling rate from the band-limited continuous scene, and Yk be
the k-th LR observation from the camera. X and Yk s are represented in
lexicographical order. Assume the camera captures K LR frames of X, where
the LR observations are related with the HR scene X by
Yk = Dk Hk Fk X + Vk , k = 1, 2, ..., K, (1.1)
where Fk encodes the motion information for the k-th frame, Hk models the
blurring effects, Dk is the down-sampling operator, and Vk is the noise term.
These linear equations can be rearranged into a large linear system
⎡ ⎤ ⎡ ⎤
Y1 D1 H1 F1
⎢ Y2 ⎥ ⎢ D2 H2 F2 ⎥
⎢ ⎥ ⎢ ⎥
⎢ · ⎥=⎢ · ⎥X + V (1.2)
⎢ ⎥ ⎢ ⎥
⎣ · ⎦ ⎣ · ⎦
YK DK HK FK
or equivalently
Y = MX + V (1.3)
 Image Super-Resolution: Historical Overview and Future Challenges 7

The involved matrices Dk , Hk , Fk , or M are very sparse, and this linear system
is typically ill-posed. Furthermore, in real imaging systems, these matrices are
unknown and need to be estimated from the available LR observations, leaving
the problem even more ill-conditioned. Thus, proper prior regularization for
the high resolution image is always desirable and often even crucial. In the
following, we will introduce some basic super-resolution techniques proposed
in the literature and give an overview of the recent developments.

1.3.2 Super-Resolution in the Frequency Domain

The pioneering work for super-resolution traces back to Tsai and Huang [99],
in which the authors related the high resolution image with multiple shifted
low-resolution images by a frequency domain formulation based on the shift
and aliasing properties of the Continuous and Discrete Fourier Transforms.
Let x(t1 , t2 ) denote a continuous high-resolution scene. The global translations
yield K shifted images, xk (t1 , t2 ) = x(t1 + k1 , t2 + k2 ), with k = 1, 2, ..., K,
where k1 and k2 are arbitrary but known shifts. The continuous Fourier
transform (CFT) of the scene is given by X (u1 , u2 ) and those of the translated
scenes by Xk (u1 , u2 ). Then by the shifting properties of the CFT, the CFT of
the shifted images can be written as

Xk (u1 , u2 ) = exp [j2π(k1 u1 + k2 u2 )] X (u1 , u2 ). (1.4)

The shifted images are impulse sampled with the sampling period T1 and T2 to
yield observed low-resolution image yk [n1 , n2 ] = xk (n1 T1 + k1 , n2 T2 + k2 )
with n1 = 0, 1, 2, ..., N1 − 1 and n2 = 0, 1, 2, ..., N2 − 1. Denote the discrete
Fourier transforms (DFTs) of these low-resolution images by Yk [r1 , r2 ]. The
CFTs of the shifted images are related with their DFTs by the aliasing prop-
erty:
∞
 ∞

1 2π r1 2π r2
Yk [r1 , r2 ] = Xk − m1 , − m2 .
T1 T2 m1 =−∞ m2 =−∞
T1 N1 T2 N2
(1.5)

Assuming X (u1 , u2 ) is band-limited, |X (u1 , u2 )| = 0 for |u1 | ≥ (N1 π)/T1 ,

|u2 | ≥ (N2 π)/T2 , combining Eqn. 1.4 and Eqn. 1.5 we relate the DFT coeffi-
cients of Yk [r1 , r2 ] with the samples of the unknown CFT of x(t1 , t2 ) in matrix
form as 2
Y = ΦX , (1.6)
where Y is a K × 1 column vector with the k th element being the DFT coeffi-
cient Yk [r1 , r2 ], X is a N1 N2 × 1 column vector containing the samples of the
unknown CFT coefficients of x(t1 , t2 ), and Φ is a K × N1 N2 matrix relating
2 Strictly, subscripts {r , r } should be used in the following equation. We omit those for
1 2
an uncluttered presentation.
8 Super-Resolution Imaging

Y and X . Eqn. 1.6 defines a set of linear equations from which we intend to
solve X and then use the inverse DFT to obtain the reconstructed image.
The above formulation for SR reconstruction assumes a noise-free and
global translation model with known parameters. The downsampling process
is assumed to be impulse sampling, with no sensor blurring effects modeled.
Along this line of work, many extensions have been proposed to handle more
complicated observation models. Kim et al. [49] extended [99] by taking into
account the observation noise as well as spatial blurring. Their later work in
[5] extend the work further by introducing Tikohonov regularization [95]. In
[89], a local motion model is considered by dividing the images into overlap-
ping blocks and estimating motions for each local block individually. In [98],
the restoration and motion estimation are done simultaneously using an EM
algorithm. However, the frequency domain SR theory of these works did not
go beyond as what was initially proposed. These approaches are computation-
ally efficient, but limited in their abilities to handle more complicated image
degradation models and include various image priors as proper regularization.
Later works on super-resolution reconstruction have been almost exclusively
in the spatial domain.

1.3.3 Interpolation-Restoration: Non-Iterative Approaches

Many spatial domain approaches [4, 82, 65, 2] have been proposed over the
years to overcome the difficulties of the frequency domain methods. As the
HR image and the LR frames are related in a sparse linear system (1.3),
similar to the traditional single image restoration problem [26], many flexible
estimators can be applied to the SR reconstruction. These include Maximum
Likelihood (ML), Maximum a Posteriori (MAP)[84, 35], and Projection Onto
Convex Sets (POCS)[88]. In this section, we start with the simplest and a
noniterative forward model for SR reconstruction in the spatial domain, in
analogy to the frequency domain approach.
Assume Hk is Linearly Spatial Invariant (LSI) and is the same for all K
frames, and we denote it as H. Suppose Fk considers only simple motion
models such as translation and rotation, then H and Fk commute [27, 30] and
we get
Yk = Dk Fk HX + Vk = Dk Fk Z, k = 1, 2, ..., K, (1.7)
which motivates a forward noniterative approach based on interpolation and
restoration. There are three stages for this approach 1) low-resolution image
registration; 2) nonuniform interpolation to get Z, and 3) deblurring and noise
removal to get X. Figure 1.4 shows the procedure of such an approach. The
low-resolution frames are first aligned by some image registration algorithm
[77] to subpixel accuracy. These aligned low-resolution frames are then put
on a high-resolution image grid, where nonuniform interpolation methods are
used to fill in those missing pixels on the HR image grid to get Z. At last, Z
is deblurred by any classical deconvolutional algorithm with noise removal to
 Image Super-Resolution: Historical Overview and Future Challenges 9

achieve X. Keren et al. [48] proposed an early two-step approach to SR recon-

struction based on a global translation and rotation model. Gross et al. [101]
proposed a nonuniform interpolation of a set of spatially shifted low-resolution
images by utilizing the generalized multi-channel sampling theorem by Yen
[109] and later Papulis [64], followed by deblurring. Nguyen and Milanfar [62]
proposed an efficient wavelet-based interpolation SR reconstruction algorithm
by exploiting the interlacing sampling structure in the low-resolution data.
Alam et al. [1] presented an efficient interpolation scheme based on weighted
nearest neighbors, followed by Wiener filtering for deblurring. Focusing on the
special case of SR reconstruction where the observation is composed of pure
translation, space invariant blur, and additive Gaussian noise, Elad and Hel-
Or [27] presented a very computationally efficient algorithm. [52] proposed
a triangulation-based method for interpolating irregularly sampled data. The
triangulation method, however, is not robust to noise commonly present in real
applications. Based on the normalized convolution [50], Pham et al. [71] pro-
posed a robust certainty and a structure-adaptive applicability function to the
polynomial facet model and applied it to fusion of irregularly sampled data.
Recently, Takeda et al. [91] proposed an adaptive steering kernel regression
for interpolation on the high-resolution image grid where the low-resolution
images are registered and mapped on.
These interpolation-restoration forward approaches are intuitive, simple,
and computationally efficient [30], [18], assuming simple observation models.
However, the step-by-step forward approach does not guarantee optimality
of the estimation. The registration error can easily propagate to the later
processing. Also, the interpolation step is suboptimal without considering the
noise and blurring effects. Moreover, without the HR image prior as proper
regularization, the interpolation based approaches need special treatment of
limited observations in order to reduce aliasing.

1.3.4 Statistical Approaches

Unlike the interpolation-restoration approaches, statistical approaches relate
the SR reconstruction steps stochastically toward optimal reconstruction. The
HR image and motions among low-resolution inputs can be both regarded as
stochastic variables. Let M (ν, h) denote the degradation matrix defined by
the motion vector ν and blurring kernel h, the SR reconstruction can be cast
10 Super-Resolution Imaging

FIGURE 1.4: The interpolation SR approach based on alignment and post

processing of deblurring.

into a full Bayesian framework:

X = arg max P r(X|Y )

X

= arg max P r(X, M (ν, h)|Y )dν

X ν,h
P r(Y |X, M (ν, h))P r(X, M (ν, h)) (1.8)
= arg max dν
X ν,h P r(Y )

= arg max P r(Y |X, M (ν, h))P r(X)P r(M (ν, h))dν.
X ν,h

Note that X and M (ν, h) are statistically independent [35]. Here

P r(Y |X, M (ν, h)) is the data likelihood, P r(X) is the prior term on the
desired high-resolution image and P r(M (ν, h)) is a prior term on the motion
estimation. V in Eqn. 1.3 usually stands for additive noise, assumed to be a
zero-mean and white Gaussian random vector. Therefore,

1
P r(Y |X, M (ν, h)) ∝ exp − Y − M (ν, h)X2 . (1.9)
2σ 2

P r(X) is typically defined using the Gibbs distribution in an exponential form

1
P r(X) = exp{−αA(X)}, (1.10)
Z
where A(X) is a non-negative potential function, and Z is just a normalization
factor. The Bayesian formulation in Equation 1.8 is complicated and difficult
to evaluate due to the integration over motion estimates. If M (ν, h) is given
 Image Super-Resolution: Historical Overview and Future Challenges 11

or estimated beforehand (denote as M ), Eqn. 1.8 can be simplified as

X = arg max P r(Y |X, M )P r(X)

X
(1.11)
= arg min{Y − M X2 + λA(X)},
X

where λ absorbs the variance of the noise and α in Eqn. 1.10, balancing the
data consistence and the HR image prior strength. Eqn. 1.11 is the popular
Maximum a Posteriori (MAP) formulation for SR, where M is assumed to be
known. The statistical approaches discussed below vary in the ways of treat-
ing degradation matrix M (ν, h), prior term P r(X), and statistical inference
methods toward Equation 1.8.

1.3.4.1 Maximum Likelihood

If we assume uniform prior over X, Eqn. 1.11 reduces to the simplest max-
imum likelihood (ML) estimator (motion estimation is assumed as a prior).
The ML estimator relies on the observations only, seeking the most likely
solution for the observations to take place by maximizing p(Y |X), giving

X̂ML = arg min Y − M X2 . (1.12)

X

Differentiating Eqn. 1.12 with respect to X and setting the derivative to be

zero gives the classical pseudo-inverse result

X̂ML = (M T M )−1 M T Y . (1.13)

If M T M is singular, the problem is ill-posed and there are infinite many

possible solutions due to the null space of M . This naturally leads to the term
of regularization for the sake of a unique solution from purely the algebraic
point of view, which although can be interpreted in the MAP framework.
For computation, direct inverse of matrix as M T M is usually prohibitive in
practice due to the high-dimensionality problem. For example, if the low-
resolution images are of size 100 × 100 and are to be zoomed to a single high-
resolution frame X of 300 × 300, M is of the size 90000 × 90000, requiring
inverse of a matrix of size 90000 × 90000. Therefore, many iterative methods
for practical ways to solve this large set of sparse linear equations have been
suggested in the literature [111].
Irani and Peleg proposed a simple but very popular method, based on an
error back-projection scheme inspired by computer-aided tomography, in [39,
40, 41]. The algorithm iteratively updates the current estimation by adding
back the warped simulation error convolved with a back-projection function
(BPF): 
X i+1 = X i + c Fk−1 [hbpf ∗ S ↑ (Yk − Yk )], (1.14)
k
where c is a constant, hbpf is the back-projection kernel, S ↑ is the upsam-
pling operator, and Yk is the simulated k-th LR frame from the current HR
12 Super-Resolution Imaging

estimation. In [41], the authors applied this idea to real applications by in-
corporating a multiple motion tracking algorithm to deal with partially oc-
cluded objects, transparent objects and some objects of interest. The back-
projection algorithm is simple and flexible in handing many observations with
different degradation procedures. However, the solution of back-projection is
not unique, depending on the initialization and the choice of back-projection
kernel. As shown in [26] and [10], the back-projection algorithm is none other
than an ML estimator. The choice of BPF implies some underlying assumption
about the noise covariance of the observed low-resolution pixels [10]. Treating
the motion estimates M (ν) as unknown, Tom et al. [98] proposed an ML SR
image estimation algorithm to estimate the subpixel shifts, noise of the image,
and the HR image simultaneously. The proposed ML estimation is treated by
the Expectation-Maximization (EM) algorithm.
As in the image denoising and single image expansion case, direct ML
estimator without regularization in SR where the number of observations is
limited can be severely ill-posed, especially when the zoom factor is large
(e.g. greater than 2). The ML estimator is usually very sensitive to noise,
registration estimation errors, and PSF estimation errors [10], and therefore
proper regularization on the feasible solution space is always desirable. This
leads to the mainstream SR reconstruction approaches based on MAP.

1.3.4.2 Maximum a Posteriori

Many works [46, 84, 15] in SR reconstruction have followed the MAP approach
in Eqn. 1.11, where the techniques vary in the observation model assumptions
and the prior term P r(X) for the desired solution. Different kinds of priors
for natural images have been proposed in the literature, but none of them
stands out as the lead. In the following, we list three commonly used image
priors for the SR reconstruction techniques.

1. Gaussian MRF. The Gaussian Markov Random Field (GMRF) [37, 33]
takes the form
A(X) = X T QX, (1.15)
where Q is a symmetric positive matrix, capturing spatial relations be-
tween adjacent pixels in the image by its off-diagonal elements. Q is
often defined as ΓT Γ, where Γ acts as some first or second derivative
operator on the image X. In such a case, the log likelihood of the prior
is
log p(X) ∝ ΓX2 , (1.16)
which is well-known as the Tikhonov regularization [95, 26, 63], the most
commonly used method for regularization of ill-posed problems. Γ is usu-
ally referred as Tikhonov matrix. Hardie et al. [35] proposed a joint MAP
framework for simultaneous estimation of the high-resolution image and
motion parameters with Gaussian MRF prior for the HR image. Bishop
and Tipping [96] proposed a simple Gaussian process prior where the
 Image Super-Resolution: Historical Overview and Future Challenges 13

covariance matrix Q is constructed by spatial correlations of the image

pixels. The nice analytical property of Gaussian process prior allows a
Bayesian treatment of the SR reconstruction problem, where the un-
known high-resolution image is integrated out for robust estimation of
the observation model parameters (unknown PSFs and registration pa-
rameters). Although the GMRF prior has many analytical advantages, a
common criticism for it associated with super-resolution reconstruction
is that the results tend to be overly smooth, penalizing sharp edges that
we desire to recover.
2. Huber MRF. The problem with GMRF can be ameliorated by mod-
eling the image gradients with a distribution with heavier tails than
Gaussian, leading to the popular Huber MRF (HMRF) where the Gibbs
potentials are determined by the Huber function,

a2 |a| ≤ α
ρ(a) = (1.17)
2α|a| − α2 otherwise,

where a is the first derivative of the image. Such a prior encourages piece-
wise smoothness, and can preserve edges well. Schultz and Stevenson
[83] applied this Huber MRF to the single image expansion problem,
and later to the SR reconstruction problem in [84]. Many later works on
super-resolution employed the Huber MRF as the regularization prior,
such as [11, 12, 15, 13, 73, 74] and [3].
3. Total Variation. The Total Variation (TV) norm as a gradient penalty
function is very popular in the image denoising and deblurring litera-
ture [81, 54, 16]. The TV criterion penalizes the total amount of change
in the image as measured by the 1 norm of the magnitude of the gra-
dient
A(X) = ∇X1 (1.18)
where ∇ is a gradient operator that can be approximated by Laplacian
operators [81]. The 1 norm in the TV criterion favors sparse gradients,
preserving steep local gradients while encouraging local smoothness[13].
Farsiu et al. [30] generalized the notation of TV and proposed the so
called bilateral TV (BTV) for robust regularization.

For more comparisons of these generic image priors on effecting the solution
of super-resolution, one can further refer to [10] and [25].

1.3.4.3 Joint MAP Restoration

Multiple frame SR reconstruction can be divided into two subproblems: LR
registration and HR estimation. Many previous algorithms treat these two
processes as two distinct processes: first do registration and then estimation by
MAP, which is suboptimal as registration and estimation are interdependent.
Motion estimation and HR estimation can benefit each other if interactions
14 Super-Resolution Imaging

between them are allowed. In joint MAP restoration, Eqn. 1.11 is extended to
include the motion and PSF estimates as unknowns for inference:
{X, ν, h} = arg max P r(Y |X, M (ν, h))P r(X)P r(M (ν, h))
X,ν,h

= arg min − log [P r(Y |X, M (ν, h))] − log [P r(X)] (1.19)
X,ν,h

− log [P r(M (ν, h))] .

Tom et al. [98] divided the SR problem into three subproblems, namely
registration, restoration and interpolation. Instead of solving them indepen-
dently they simultaneously estimated registration and restoration by maxi-
mizing likelihood using Expectation-Maximization (EM). Later they included
interpolation into the framework and estimated all of the unknowns using EM
in [97]. [35] applied the MAP framework for simultaneous estimation of the
high-resolution image and translation motion parameters (PSF is taken as a
known prior). The high-resolution image and motion parameters are estimated
using a cyclic coordinate-descent optimization procedure. The algorithm con-
verges slowly but improves the estimation a lot. Segall et al. [86, 85] presented
an approach of joint estimation of dense motion vectors and HR images applied
to compressed video. Woods et al.[105] treated the noise variance, regulariza-
tion and registration parameters all as unknowns and estimated them jointly
in a Bayesian framework based on the available observations. Chung et al. [19]
proposed a joint optimization framework and showed superior performance to
the coordinate descent approach [46]. The motion model they handled is affine
transformations. To handle more complex multiple moving objects problems
in the SR setting, Shen et al. [87] addressed the problem by MAP formulation
combining motion estimation, segmentation and SR reconstruction together.
The optimization is done in a cyclic coordinate descent process similar to [46].

1.3.4.4 Bayesian Treatments

Due to limited low-resolution observations, the SR reconstruction problem is
ill-posed in nature. Joint MAP estimation of motion parameters, PSF, and the
HR image may face the problem of overfitting [96]. While motion and blur-
ring is difficult to model in general, simple models spanned by few parameters
are sufficient for SR applications in many scenarios. Given the low-resolution
observations, however, estimating these parameters by integrating over the
unknown high-resolution image is a useful approach. Bishop and Tipping [96]
proposed such a Bayesian approach for SR where the unknown high-resolution
image is integrated out and the marginal is used to estimate the PSF and
motion parameters. To make the problem analytically tractable, a Gaussian
Markov Random Field (GMRF) prior is used to model the high-resolution
image. Even though an unfavorable GMRF is used for the high-resolution im-
age, the PSF and motion parameters can still be estimated quite accurately.
Then the estimated parameters are fixed, and a MAP estimation of the HR
image is performed. An in-depth analysis similar to this for blind deconvo-
 Image Super-Resolution: Historical Overview and Future Challenges 15

lution is discussed in [53]. Such a Bayesian approach outperforms the joint

MAP approaches in Subsection 1.3.4.3, which will easily get overfitting with
the PSF parameters. However, the integration over the high-resolution image
is computationally heavy and the Gaussian prior over the image leads the final
reconstruction toward excess smoothness. Instead of marginalization over the
unknown high-resolution image, Pickup et al. proposed in their recent works
[73, 74, 72] to integrate over the unknown PSF and motion parameters as
in Eqn. 1.8, which is motivated to overcome the uncertainty of the registra-
tion parameters [79]. The registration parameters are estimated beforehand
and then treated as Gaussian variables with the pre-estimated values as the
means to model their uncertainty. The HR image estimation can be combined
with any favorable image prior for MAP estimation after integrating the ob-
servation model parameters. Sharper results can be obtained with such an
approach compared with [96] as reported in [73, 74, 72].
Such Bayesian treatments, by marginalizing the unknowns, demonstrate
promising power for SR recovery. However, in order for integration to be
tractable, image priors or registration parameters have to take simple para-
metric forms, limiting these models in dealing with more complex cases that
may happen in real videos. Computation could also be a concern for such
algorithms in realistic applications.

1.3.5 Example-Based Approaches

Previous super-resolution approaches rely on aggregating multiple frames that
contain complementary spatial information. Generic image priors are usually
deployed to regularize the solution properly. The regularization becomes es-
pecially crucial when an insufficient number of measurements is supplied, as
in the extreme case, only one single low-resolution frame is observed. In such
cases, generic image priors do not suffice as an effective regularization for
SR [2]. A recently emerging methodology for regularizing the ill-posed super-
resolution reconstruction is to use examples, in order to break the super-
resolution limit caused by inadequate measurements. Different from previous
approaches where the prior is in a parametric form regularizing on the whole
image, the example-based methods develop the prior by sampling from other
images, similar to [24],[38] in a local way.
One family of example-based approaches is to use the examples directly,
with the representative work proposed by Freeman et al. [31]. Such approaches
usually work by maintaining two sets of training patches, {xi }ni=1 sampled
from the high-resolution images, and {yi }ni=1 sampled from the low-resolution
images correspondingly. Each patch pair (xi , yi ) is connected by the obser-
vation model yi = DHxi + v. This high- and low-resolution co-occurrence
model is then applied to the target image for predicting the high-resolution
image in a patch-based fashion, with a Markov Random Field (MRF) model
as shown in Figure 1.5. The observation model parameters have to be known
as a prior, and the training sets are tightly coupled with the image targeted.
16 Super-Resolution Imaging

FIGURE 1.5: The MRF model for single frame super-resolution.

Patch size should also be chosen properly. If the patch size is very small, the
co-occurrence prior is too weak to make the prediction meaningful. On the
other hand, if the patch size is too large, one may need a huge training set to
find proximity patches for the current observations.
A naive way to do super-resolution with such a coupled training sets is, for
each low-resolution patch in the low-resolution image, find its nearest neigh-
bor ỹ in {yi }ni=1 , and then put the corresponding x̃ from {xi }ni=1 to the high-
resolution image grid. Unfortunately, this simple approach will produce dis-
turbing artifacts due to noise and the ill-posed nature of super-resolution [25].
Relaxing the nearest neighbor search to k-nearest neighbors can ensure that
the proximity patch we desire will be included. Freeman et al. [31] proposed
a belief propagation [108] algorithm based on the above MRF model to se-
lect the best high-resolution patch found by k-nearest neighbors that has best
compatibility with adjacent patches. Sun et al.[90] extended this idea using
the sketch prior to enhance only the edges in the image, aiming to speed up
the algorithm. The IBP [39] algorithm is then applied as a post processing
step to ensure data consistence on the whole image. Wang et al. [103] further
followed this line of work and proposed a statistical model that can handle
unknown PSF.
The above methods are based on image patches directly, requiring large
training sets to include any patterns possibly encountered in testing. Chang
et al. [17] proposed another simple but effective method based on neighbor
embedding [93], with the assumption of correspondence between the two man-
ifolds formed by the low- and high-resolution image patches. For each low-
resolution image patch ykt from the test image (superscript “t” distinguishes
the test patch from the training patches), the algorithm finds its k-nearest
neighbors Nt from {yi }ni=1 , and computes the reconstruction weights by neigh-
 Image Super-Resolution: Historical Overview and Future Challenges 17

bor embedding

ŵs = arg min ykt − ws ys 2 ,
ws
ys ∈Nt
 (1.20)
s.t. ws = 1.
ys ∈Nt

The reconstruction weights  are then applied to generate the corresponding

high-resolution patch x̂tk = ys ∈Nt ŵs xs . To handle the compatibility prob-
lem between adjacent patches, simple averaging in the overlapping regions is
performed. The algorithm works nicely even with smaller patch databases than
[108]. However, fixing k for each low-resolution patch may result in overfitting
or underfitting. Yang et al. [107] proposed another patch-based single frame
super-resolution method. The method is derived from the compressive sensing
theory, which ensures that linear relationships among high-resolution signals
can be precisely recovered from their low-dimensional projections [9], [22]. The
algorithm models the training sets as two dictionaries: Dh = [x1 , x2 , ..., xn ]
and Dl = [y1 , y2 , ..., yn ]. Given a test low-resolution image patch ykt , the
approach basically seeks the supports by an 1 minimization [23]

ŵ = arg min w1

w
(1.21)
s.t. yt − Dl w2 ≤ σ 2 ,

which can be rewritten with Lagrange multiplier as an unconstrained opti-

mization problem known as Lasso in the statistics literature [94]. The corre-
sponding high-resolution patch is recovered by xtk = Dh ŵ. Compared to the
neighbor embedding method with fixed k neighbors, Yang’s method adaptively
chooses the fewest necessary supports for reconstruction, avoiding overfitting.
Moreover, the 1 minimization formulation is more robust to noise than the
previous mentioned patch-based methods. In a later version [42], this approach
is further extended by learning a coupled dictionary instead of using the raw
patches, allowing the algorithm to be much more efficient.
One criticism with the aforementioned methods with direct examples is
that operating on local patches cannot guarantee global optimality of the es-
timation. Another kind of example-based approach seeks to perform MAP
estimation with local priors on the image space sampled from examples. The
pioneering work by Baker and Kanade [2] formulated an explicit regulariza-
tion that demands proximity between the spatial derivatives of the unknown
image to those of the found examples. The examples are formed by a pyramid
derivative set of features, instead of raw data directly. Similar method is ap-
plied to text super-resolution in [75]. Datsenko and Elad [21] presented a global
MAP estimation where the example-based regularization is given by a binary
weighted average instead of the nearest neighbor, bypassing outliers due to
noise. This work is further extended and elaborated in [21], where the binary
weighting scheme is relaxed. Another noteworthy approach for example-based
18 Super-Resolution Imaging

approach is by Protter et al. [78], generalized from the nonlocal means denois-
ing algorithm [8]. Instead of sampling examples from other training images,
the algorithm explores self-similarities within the image (or sequence) and ex-
tract the example patches from the target image (or sequence) itself. A recent
work by Glasner et al. further explored self-similarities in images for SR by
combining the classical algorithm based on subpixel displacements and the
example-based method based on patch pairs extracted from the target image.
The use of examples can be much more effective when dealing with nar-
row families of images, such as text and face images. A group of algorithms
have emerged targeting face super-resolution in recent years due to its im-
portance in surveillance scenarios. Face super-resolution is usually referred
to as face hallucination, following the early work by Baker and Kanade [2].
Capel and Zisserman [14] proposed an algorithm where PCA [45] subspace
models are used to learn parts of the faces. Liu et al. [58], [57] proposed a
two-step approach toward super-resolution of faces, where the first step uses
the eigenface [100] to generate a medium resolution face, followed by the non-
parametric patch-based approach [31] in the second step. Such an Eigenface-
based approach has been explored in several later works [32],[104] too. Yang
et al. [106] proposed a similar two-step approach. Instead of using the holistic
PCA subspace, [106] uses local Nonnegative Matrix Factorization (NMF)[51]
to model faces and the patch-based model in the second step is adopted from
[107]. Jia and Gong [43], [44] proposed the tensor approach to deal with more
face variations, such as illuminations and expressions. Although these face hal-
lucination algorithms work surprisingly well, they only apply to frontal faces,
and only few works have been devoted on evaluating face hallucination for
recognition [32], [36].
Example-based regularization is effective in our SR problem when insuffi-
cient observations are available. There are still a number of questions we need
to answer regarding this kind of approaches. First, how to choose the opti-
mal patch size given the target image. Perhaps a multiresolution treatment is
needed. Second, how to choose the database. Different images have different
statistics, and thereby need different databases. An efficient method for dic-
tionary adaptation to the current target image may suggest a way out. Third,
how to use the example-based prior more efficiently. The computation issue
could be a difficulty for practical applications. Readers are suggested to refer
to [25] for more detailed analysis on example-based regularization for inverse
problems.

1.3.6 Set Theoretic Restoration

Besides the optimization approaches derived from stochastic view as discussed
above, another stream of methods is through the well-known Projection onto
Convex Sets (POCS) [110]. The POCS methods approach the SR problem by
formulating multiple constraining convex sets containing the desired image as
a point within the sets. Defining such convex sets is flexible and can incorpo-
 Image Super-Resolution: Historical Overview and Future Challenges 19

rate different kinds of constraints or priors, even nonlinear and nonparametric

constraints. As an example, we introduce several commonly used convex sets
in the POCS methods. The data consistency or reconstruction constraints can
be modeled as K convex sets:
  
Ck = X Dk Hk Fk X − Yk 2 ≤ σ 2 , 1 ≤ k ≤ K . (1.22)

Smoothness constraints can be defined as

  
CΓ = X ΓXp < σ2 . (1.23)

where p = 1, 2, ∞ denotes different norms. Amplitude constraints can also be

modeled:   
CA = X A1 ≤ X[m, n] ≤ A2 . (1.24)
With a group of M convexM sets, the desired solution lies in the intersection
of these sets X ∈ Cs = i=1 Ci . The POCS technique suggests the following
recursive algorithm for finding a point within the intersection set given an
initial guess:
Xk+1 = PM PM−1 · · · P2 P1 Xk , (1.25)
where X0 is an initial guess, and Pi is the projection operator that projects
a point onto a closed, convex set Ci .
Early POCS techniques for SR reconstruction were proposed by Stark and
Oskoui [88]. Extensions were proposed to handle space-varying PSF, motion
blur, sensor blur, and aliasing sampling effects in [68], [67], [69]. Many super-
resolution works only consider nonzero aperture size (the lens blur, PSF),
but not finite aperture time (motion blur) which is quite common in real
low-resolution videos. [69] is the early work to take into account the motion
blur in SR reconstruction of videos based on POCS technique. As the motion
blurring caused by a finite aperture time will in general be space- and per-
haps time-varying, it cannot be factored out of the SR restoration problem
and performed as a separate post-processing step. The POCS technique can
conveniently handle such problems. Extending this method, Eren et al. [28]
proposed a POCS-based approach for robust, object-based SR reconstruction.
The proposed method employs a validity map to disable projections based on
observations with inaccurate motion estimation, and a segmentation map for
object-based processing. Elad and Feuer [26] analyzed and compared the
ML, MAP and POCS methods for super-resolution, and proposed a hybrid
approach. Patti and Altunbasak [66] extended their earlier work in the image
observation model to allow high order interpolation and modified constraint
sets to reduce the edge ringing artifacts.
The advantage of the POCS technique lies in its simplicity to incorporate
any kinds of constraints and priors that may present as impossible for those
stochastic approaches. However, POCS is notorious for its heavy computation
and slow convergence. The solution is not unique, depending on the initial
guess. The POCS methods also assume priors on the motion parameters and
20 Super-Resolution Imaging

system blurs. They cannot estimate those registration parameters and the
high-resolution image as in the stochastic approaches simultaneously. The hy-
brid approach combining a stochastic view and the POCS philosophy suggests
a promising way to pursue.

1.4 Challenge Issues for Super-Resolution

In the previous sections, we have discussed several basic techniques for SR
reconstruction. Although many different approaches have been proposed since
the SR concept was introduced, most approaches work well on toy data rather
than in real problems. In building a practical SR system, many challenging
issues still lay ahead preventing the SR techniques from wide applications. In
the following, we list several challenges that we think are important for the
future development and application of SR techniques.

1.4.1 Image Registration

Image registration is critical for the success of multiframe SR reconstruction,
where complementary spatial samplings of the HR image are fused. The im-
age registration is a fundamental image processing problem that is well known
as ill-posed. The problem is even more difficult in the SR setting, where the
observations are low-resolution images with heavy aliasing artifacts. The per-
formance of the standard image registration algorithms decreases as the res-
olution of the observations goes down, resulting in more registration errors.
Artifacts caused by these registration errors are visually more annoying than
the blurring effect resulting from the interpolation of a single image. Tradi-
tional SR reconstruction usually treats image registration as a distinct process
from the HR image estimation. Therefore, the recovered HR image quality de-
pends largely on the image registration accuracy from the previous step. Many
image registration techniques derived from different principles have been pro-
posed in the literature [7, 114]. However, Robinson and Milanfar [79] showed
that the registration performance is bounded even for the simplest case of
global translation.
LR image registration and the HR image estimation are actually inter-
dependent [80]. On one hand, accurate subpixel motion estimation benefits
HR image estimation. On the other hand, high-quality HR image can facili-
tate accurate motion estimation. Therefore, tailored to the SR reconstruction
problem, the LR image registration can be addressed together with the HR
image reconstruction, leading to the joint ML [97] or MAP [35, 87, 76] frame-
work for simultaneous estimation. These joint estimation algorithms capture
the dependence between LR image registration and HR image estimation, and
performance improvements are observed. However, with limited observations,
 Image Super-Resolution: Historical Overview and Future Challenges 21

the joint estimation for registration parameters and HR image may result in
overfitting. To overcome this overfitting problem, Tipping and Bishop [96]
employed a Bayesian approach for estimating both registration and blur pa-
rameters by marginalizing the unknown high-resolution image. The algorithms
shows noteworthy estimation accuracy both for registration and blur param-
eters, however the computation cost is very high. Pickup et al. [73, 74, 72]
instead cast the Bayesian approach in another way by marginalizing the un-
known registration parameters, to address the uncertainty inherent with the
image registration [79].
The stochastic approaches associating the HR image estimation toward im-
age registration do demonstrate promising results, however such parametric
methods are limited in the motion models they can effectively handle. Usually,
some simple global motion models are assumed. Real videos are complicated
comprising arbitrary local motions, where parametrization of the motion mod-
els may be intractable. Optical flow motion estimation can be applied to such
scenarios. However, the insufficient measurements for local motion estimations
make these algorithms vulnerable to errors, which may cause disasters for SR
reconstruction [112]. Another promising approach toward SR reconstruction
is the nonparametric methods, which try to bypass the explicit motion esti-
mation. Protter et al. [78] extended the non-local means denoising algorithm
to SR reconstruction, where fuzzy motion estimation based on block matching
is used. Later they proposed a probabilistic motion model in [77], which is a
nonparametric model analogy to [72]. Both [78] and [77] can handle complex
motion patterns in real videos. Compared to the classical SR methods based
on optical flow motion estimation, Protter’s methods reduce the errors caused
by misalignment by a weighting strategy over multiple possible candidates.
Takeda et al. [92] on the other hand applied an 3-D steering kernel proposed
in their early work [91] to video, which also avoids explicit motion estimation,
for denoising and SR reconstruction. The 3-D steering kernel captures both
spatial and temporal structure, encoding implicit motion information, and
thus can be applied to both spatial and temporal SR for video with complex
motion activities. While methods without explicit motion estimation indeed
produce promising results toward the practical applicability of SR techniques,
further improvements may include computation efficiency, combining adaptive
interpolation or regression together with deblurring, and generalizing obser-
vation models to 3-D motions in video, e.g., out-of-plane rotation.

1.4.2 Computation Efficiency

Another difficulty limiting practical application of SR reconstruction is its
intensive computation due to a large number of unknowns, which require ex-
pensive matrix manipulations. Real applications always demand efficiency of
the SR reconstruction to be of practical utility, e.g., in the surveillance video
scenarios, it is desired for the SR reconstruction to occur in real time. Ef-
ficiency is also desirable for SR systems with users in the loop for tuning
22 Super-Resolution Imaging

parameters. Many SR algorithms targeting efficiency fall into the previously

discussed interpolation-restoration approach, such as [27], [1], [61], and [34].
In [34], Hardie showed the computation superiority of his algorithm over pre-
vious efficient algorithms proposed in [1] and [61], and claimed that the al-
gorithm can be applied in real time with global translation model. However,
the computation goes up significant when nontranslation model occurs, which
can be ameliorated by massive parallel computing. Others tried to examine
particular modeling scenarios to speed up the optimization problem. Zomet
and Peleg [115] and Farsiu et al. [30] studied the application of Dk , Hk , and
Fk directly as the corresponding image operations of downsampling, blurring
and shifting, bypassing the need to explicitly construct the matrices, bringing
significant speed ups. [6] combined a slightly modified version of [27] and [30]
and implemented a real-time SR system using FPGA, a nice attempt to the
practical use of SR.
However, such algorithms require precise image registration, which is com-
putation intensive in the first place. Moreover, these algorithm can only handle
simple motion models efficiently up to now, far from application in real com-
plex video scenarios. For videos with arbitrary motions, [92] suggests promis-
ing directions for seeking efficient algorithms. It is also interesting to see how
parallel computing, e.g., GPU, and hardware implementations affect the fu-
ture applications of SR techniques.

1.4.3 Robustness Aspects

Traditional SR techniques are vulnerable to the presence of outliers due to
motion errors, inaccurate blur models, noise, moving objects, motion blur,
etc. These inaccurate model errors cannot be treated as Gaussian noise as the
usual assumption with 2 reconstruction residue. Robustness of SR is of in-
terest because the image degradation model parameters cannot be estimated
perfectly, and sensitivity to outliers may result in visually disturbing artifacts,
which are intolerable in many applications, e.g., video standard conversion.
However, not enough work has been devoted to such an important aspect.
Chiang and Boulte [18] used median estimation to combine the upsampled
images to cope with outliers from non-stationary noise. Zomet et al. [116]
cast the problem in a different way, where a robust median-based gradient
is used for the optimization to bypass the influence of outliers. Farsiu et al.
[30],[82] changed the commonly used 2 norm into 1 norm for robust estima-
tion similar to [18] and robust regularization. [113] introduced a simultaneous
super-resolution with Huber norm as the prior for robust regularization. Pham
et al. [71] proposed a robust certainty to each neighboring sample for inter-
polating unknown data, with the same photometric-based weighting scheme
used in bilateral filtering. A similar uncertainty scheme is also used in the
probabilistic motion model [77] for taking care of optical flow motion esti-
mation errors based on block matching. Many of these algorithms showed
improvements for outliers assumed on the toy data, where more experimental
 Image Super-Resolution: Historical Overview and Future Challenges 23

evaluations are needed to see how much the robustness efforts can benefit real
SR performance.

1.4.4 Performance Limits

The SR reconstruction has become a hot research topic since it was intro-
duced, and thousands of SR papers have bloomed into publications. However,
not much work has been devoted to the fundamental understanding of the
performance limits of these SR reconstruction algorithms. Such a performance
limit understanding is important. For example, it will shed light on SR camera
design, helping to analyze factors such as model errors, zooming factors and
number of frames, etc. In general, an ambitious analysis of the performance
limits for all SR techniques could be intractable. First, SR reconstruction is a
complex task that consists of many interdependent components. Second, it is
still unknown what is the most informative prior given the SR task, especially
for the example-based approaches. Last, a good measure instead of simple
MSE is still needed for performance evaluation. It has been recognized that
an estimation with higher MSE does not have to be visually more appealing.
For example, bicubic interpolation usually achieves smaller MSE compared
with those recovered by some example-based approaches [107].
Several works attempting at the performance understanding have been
proposed over the last several years. [2] analyzed the numerical conditions of
the SR linear systems, and concluded that as the zoom factor increases the
general image prior is of less and less help for SR. [56] derived the performance
limits based on matrix perturbation, but with the assumption that image reg-
istration is known as a prior. With simple translation model, Robinson and
Milanfar in [79], use the Cramér-Rao (CR) bounds to analyze the registration
performance limit. They extend this work in [80] to give a thorough analysis
of SR performance with factors such as motion estimation, decimation factor,
number of frames, and prior information. The analysis is based on the MSE
criterion and the motion model is again assumed to be simple global trans-
lational. Eekeren et al. [102] evaluated several SR algorithms on real-world
data exploring several influential factors empirically. Even though these ef-
forts at understanding performance bounds are far from enough about SR,
they indeed suggest ways for people to follow.
While it is hard to draw consistent conclusions for different SR techniques,
in terms of performance evaluation, some benchmark and realistic datasets
are needed for fair comparison and algorithm understanding. Future research
should pursue more theoretical analysis and performance evaluation for di-
recting SR technique developments.
24 Super-Resolution Imaging

Bibliography
[1] M. S. Alam, J. G. Bognar, R. C. Hardie, and B. J. Yasuda. Infrared im-
age registration and high-resolution reconstruction using multiple trans-
lationally shifted aliased video frames. IEEE Transactions on Instru-
mentation and Measurement, 49(5):915–923, 2000.
[2] S. Baker and T. Kanade. Limits on super-resolution and how to break
them. IEEE Transactions on Pattern Analysis and Machine Intelli-
gence, 24(9):1167–1183, 2002.
[3] S. Borman and R. L. Stevenson. Simultaneous multi-frame MAP super-
resolution video enhancement using spatio-temporal priors. In Proceed-
ings of IEEE International Conference on Image Processing, volume 3,
pages 469–473, 1999.
[4] Sean Borman and Robert L. Stevenson. Super-resolution from image
sequences - A review. In Proceedings of the 1998 Midwest Symposium
on Circuits and Systems, pages 374–378, 1998.
[5] N. K. Bose, H. C. Kim, and H. M. Valenzuela. Recursive implementation
of total least squares algorithm for image reconstruction from noisy,
undersampled multiframes. In Proceedings of the IEEE Conference on
Acoustics, Speech and Signal Processing, volume 5, pages 269–272, 1993.
[6] O. Bowen and C. S. Bouganis. Real-time image super resolution using
an FPGA. In International Conference on Field Programmable Logic
and Applications, pages 89–94, 2008.
[7] L. Brown. A survey of image registration techniques. ACM Computing
Surveys, 24(4):325–376, 1992.
[8] A. Buades, B. Coll, and J. M. More. A non-local algorithm for image
denoising. In Proceedings of IEEE Computer Society Conference on
Computer Vision and Pattern Recognition, pages 60–65, 2005.
[9] E. Candes. Compressive sensing. In Proceedings of International
Congress of Mathematicians, volume 3, pages 1433–1452, 2006.
[10] D. Capel. Image Mosaicing and Super-resolution. Springer, 2004.
[11] D. Capel and A. Zisserman. Automated mosaicing with super-resolution
zoom. In Proceedings of IEEE Computer Society Conference on Com-
puter Vision and Pattern Recognition, pages 885–891, 1998.
[12] D. Capel and A. Zisserman. Super-resolution enhancement of text image
sequences. In Proceedings of the International Conference on Pattern
Recognition, volume 1, pages 1600–1605, 2000.
 Image Super-Resolution: Historical Overview and Future Challenges 25

[13] D. Capel and A. Zisserman. Super-resolution enhancement of text image

sequences. In Proceedings of 15th International Conference on Pattern
Recognition, pages 600–605, 2000.
[14] D. Capel and A. Zisserman. Super-resolution from multiple views using
learnt image models. In Proceedings of IEEE Computer Society Con-
ference on Computer Vision and Pattern Recognition, volume 2, pages
627–634, 2001.
[15] D. Capel and A. Zisserman. Computer vision applied to super-
resolution. IEEE Signal Processing Magazine, 20(3):75–86, 2003.
[16] T. F. Chan, S. Osher, and J. Shen. The digital TV filter and nonlinear
denosing. IEEE Transaction on Image Processing, 10(2):231–241, 2001.
[17] H. Chang, D. Y. Yeung, and Y. Xiong. Super-resolution through neigh-
bor embedding. In Proceedings of IEEE Computer Society Conference
on Computer Vision and Pattern Recognition, volume 1, pages 275–282,
2004.
[18] M. C. Chiang and T. E. Boulte. Efficient super-resolution via image
warping. Image and Vision Computing, 18(10):761–771, 2000.
[19] J. Chung, E. Haber, and J. Nagy. Numerical methods for coupled super-
resolution. Inverse Problems, 22(4):1261–1272, 2006.
[20] Marco Crisani, Dong Seon Cheng, Vittorio Murino, and Donato Pan-
nullo. Distilling information with super-resolution for video surveil-
lance. In Proceedings of the ACM 2nd International Workshop on Video
Surveillance and Sensor Networks, pages 2–11, 2004.
[21] D. Datsenko and M. Elad. Example-based single document image super-
resolution: a global MAP approach with outlier rejection. Multidimen-
sional System and Signal Processing, 18(2-3):103–121, 2007.
[22] D. L. Donoho. Compressed sensing. IEEE Transactions on Information
Theory, 52(4):1289–1306, 2006.
[23] D. L. Donoho. For most large underdetermined systems of linear
equations, the minimal 1 -norm near-solution approximates the spars-
est near-solution. Communications on Pure and Applied Mathematics,
59(7):907–934, 2006.
[24] A. A. Efros and T. K. Leung. Texture synthesis by non-parametric sam-
pling. In Proceedings of IEEE International Conference on Computer
Vision, pages 1033–1038, 1999.
[25] M. Elad and D. Datsenko. Example-based regularization deployed to
super-resolution reconstruction of a single image. The Computer Jour-
nal, 52(1):15–30, 2007.
26 Super-Resolution Imaging

[26] M. Elad and A. Feuer. Restoration of single super-resolution image

from several blurred, noisy and down-sampled measured images. IEEE
Transaction on Image Processing, 6(12):1646–1658, 1997.

[27] M. Elad and Y. Hel-Or. A fast super-resolution reconstruction algorithm

for pure translational motion and common space invariant blur. IEEE
Transactions on Image Processing, 10(8):1187–1193, 2001.

[28] P. E. Eren, M. I. Sezan, and A. M. Tekalp. Robust, object-based high

resolution image reconstruction from low-resolution video. IEEE Trans-
actions on Image Processing, 6(10):1446–1451, 1997.

[29] X. Jia F. Li and D. Fraser. Universal HMT based super resolution for
remote sensing images. In IEEE International Conference on Image
Processing, pages 333–336, 2008.

[30] S. Farsiu, D. Robinson, M. Elad, and P. Milanfar. Fast and robust

multi-frame super-resolution. IEEE Transaction on Image Processing,
13(10):1327–1344, 2004.

[31] W. T. Freeman, T. R. Jones, and E. C. Pasztor. Example-based super-

resolution. IEEE Computer Graphics and Applications, 22(2):56–65,
2002.

[32] B. K. Gunturk, A. U. Batur, Y. Altunbasak, M. H. Hayes, and R. M.

Mersereau. Eigenface-domain super-resolution for face recognition.
IEEE Transactions on Image Processing, 12(5):597–606, 2003.
[33] K. M. Hanson and G. W. Wecksung. Bayesian approach to limited-angle
reconstruction in computed tomography. Journal of Optical Society of
America, 73(11):1501–1509, 1983.

[34] R. Hardie. A fast image super-resolution algorithm using an adaptive

Wiener filter. IEEE Transactions on Image Processing, 16(12):2953–
2964, 2007.

[35] R. C. Hardie, K. J. Barnard, and E. E. Armstrong. Join MAP registra-

tion and high resolution image estimation using a sequence of undersam-
pled images. IEEE Transactions on Image Processing, 6(12):1621–1633,
1997.

[36] P. H. Hennings-Yeomans, S. Baker, and B. V. K. V. Kumar. Simul-

taneous super-resolution and feature extraction for recognition of low-
resolution faces. In Proceedings of IEEE Computer Society Conference
on Computer Vision and Pattern Recognition, pages 1–8, 2008.

[37] G. T. Herman, H. Hurwitz, A. Lent, and H-P. Lung. On the Bayesian

approach to image reconstruction. Information and Control, 42(1):60–
71, 1979.
 Image Super-Resolution: Historical Overview and Future Challenges 27

[38] A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. H. Salesin.

Image analogies. In Proceedings of the 28th annual conference Computer
Graphics and Interactive Techniques, pages 327–340, 2001.
[39] M. Irani and S. Peleg. Super resolution from image sequences. In Pro-
ceedings of 10th International Conference on Pattern Recognition, vol-
ume 2, pages 115–120, 1990.
[40] M. Irani and S. Peleg. Improving resolution by image registration.
CVGIP: Graphical Models and Imaging Processing, 53(3):231–239, 1991.
[41] M. Irani and S. Peleg. Motion analysis for image enhancement: reso-
lution, occlusion and tranparency. Journal of Visual Communications
and Image Representation, 4(4):324–335, 1993.
[42] T. S. Huang J. Yang, J. Wright and Y. Ma. Super-resolution via sparse
representation. Submitted to IEEE Transactions on Image Processing,
2009.
[43] K. Jia and S. Gong. Multi-model tensor face for simultaneous super-
resolution and recognition. In Proceedings of IEEE International Con-
ference on Computer Vision, volume 2, pages 1683–1690, 2005.
[44] K. Jia and S. Gong. Generalized face super-resolution. IEEE Transac-
tions on Image Processing, 17(6):873–886, 2008.
[45] I. T. Jolliffe. Principal Component Analysis. Series: Springer Series in
Statistics, 2002.
[46] E. Kaltenbacher and R. C. Hardie. High-resolution infrared image re-
construction using multiple low resolution aliased frames. In Proceedings
of the IEEE National Aerospace Electronics Conference, volume 2, pages
702–709, 1996.
[47] J. A. Kennedy, O. Israel, A. Frenkel, R. Bar-Shalom, and A. Haim.
Super-resolution in PET imaging. IEEE Transactions on Medical Imag-
ing, 25(2):137–147, 2006.
[48] D. Keren, S. Peleg, and R. Brada. Image sequence enhancement us-
ing subpixel displacements. In Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, pages 742–746, 1988.
[49] S. P. Kim, N. K. Bose, and H. M. Valenzuela. Recursive reconstruction
of high resolution image from noisy undersampled multiframes. IEEE
Transactions on Acoustics, Speech and Signal Processing, 38(6):1013–
1027, 1990.
[50] H. Knutsson and C.-F. Westin. Normalized and differential convolution.
In Proceedings of IEEE Computer Society Conference on Computer Vi-
sion and Pattern Recognition, pages 515–523, 1993.
28 Super-Resolution Imaging

[51] D. D. Lee and H. S. Seung. Learning the parts of objects by non-negative

matrix factorizaiton. Nature, 401(6755):788–791, 1999.
[52] S. Lerttrattanapanich and N. K. Bost. High resolution image formation
from low resolution frames using delaunay triangulation. IEEE Trans-
action on Image Processing, 11(12):1427–1441, 2002.

[53] A. Levin, Y. Weiss, F. Durand, and W. Freeman. Understanding and

evaluating blind deconvolution algorithms. In Proceedings of IEEE Com-
puter Society Conference on Computer Vision and Pattern Recognition,
pages 1964–1971, 2009.

[54] Y. Li and F. Santosa. A computational algorithm for minimizing total

variation in image restoration. IEEE Transactions on Image Processing,
5(6):987–995, 1996.

[55] Frank Lin, Clinton B. Fookes, Vinod Chandran, and Sridha Sridharan.
Investigation into optical flow super-resolution for surveillance applica-
tions. In The Austrilian Pattern Recognition Society Worshop on Digital
Image Computing, 2005.

[56] Z. Lin and H.-Y. Shum. Fundamental limits on reconstruction-based

superresolution algorithms under local translation. IEEE Transactions
on Pattern Analysis and Machine Intelligence, 26(1):83–97, 2004.

[57] C. Liu, H. Y. Shum, and W. T. Freeman. Face hallucination: theory

and practice. International Journal of Computer Vision, 75(1):115–134,
2007.

[58] C. Liu, H. Y. Shum, and C. S. Zhang. Two-step approach to hallucinat-

ing faces: global parametric model and local nonparametric model. In
Proceedings of IEEE Computer Society Conference on Computer Vision
and Pattern Recognition, volume 1, pages 192–198, 2001.

[59] J. Maintz and M. Viergever. A survey of medical image registration.

Medical Image Analysis, 2(1):1–36, 1998.

[60] K. Malczewski and R. Stasinski. Toeplitz-based iterative image fusion

scheme for MRI. In IEEE International Conference on Image Process-
ing, pages 341–344, 2008.

[61] B. Narayanan, R. C. Hardie, K. E. Barner, and M. Shao. A computa-

tionally efficient super-resolution algorithm for video processing using
partition filters. IEEE Transactions on Circuits and Systems for Video
Technology, 17(5):621–634, 2007.

[62] N. Nguyen and P. Milanfar. An efficient wavelet-based algorithm for

image super-resolution. In Proceedings of International Conference on
Image Processing, volume 2, pages 351–354, 2000.
 Image Super-Resolution: Historical Overview and Future Challenges 29

[63] N. Nguyen, P. Milanfar, and G. H. Golub. A computationally efficient

image superresolution algorithm. IEEE Transactions on Image Process-
ing, 10(5):573–583, 2001.

[64] A. Papulis. Generalized sampling expansion. IEEE Transactions on

Circuits and Systems, 24(11):652–654, 1977.

[65] Sung C. Park, Min K. Park, and Moon G. Kang. Super-resolution image
reconstruction: a technical overview. IEEE Signal Processing Magazine,
20(3):21–36, 2003.

[66] A. J. Patti and Y. Altunbasak. Aritifact reduction for set theoretic

super resolution image reconstruction with edge adaptive constraints
and higher-order interpolants. IEEE Transaction on Image Processing,
10(1):179–186, 2001.

[67] A. J. Patti, M. Sezan, and A. M. Tekalp. Robust methods for high qual-
ity stills from interlaced video in the presence of dominant motion. IEEE
Transactions on Circuits and Systems for Video Technology, 7(2):328–
342, 1997.

[68] A. J. Patti, M. I. Sezan, and A. M. Tekalp. High-resolution image

reconstruction from a low-resolution image sequence in the presence
of time-varing motion blur. In Proceedings of the IEEE International
Conference on Image Processing, volume 1, pages 343–347, 1994.

[69] A. J. Patti, M. I. Sezan, and A. M. Tekalp. Superresolution video re-

construction with arbitrary sampling lattices and nonzero aperture time.
IEEE Transactions on Image Processing, 6(8):1064–1076, 1997.

[70] S. Peleg and Y. Yeshurun. Superresolution in MRI: application to hu-

man white matter fiber tract visualization by diffusion tensor imaging.
Magnetic Resonance in Medicine, 45(1):29–35, 2001.

[71] T. Q. Pham, L. J. Vliet, and K. Schutte. Robust fusion of irregularly

sampled data using adaptive normalized convolution. EURASIP Journal
on Applied Signal Processing, 2006.
[72] L. C. Pickup, D. P. Capel, S. J. Robert, and A. Zisserman. Over-
coming registration uncertainty in image super-resolution: maximize or
marginalize? EURASIP Journal on Advances in Signal Processing, 2007.

[73] L. C. Pickup, D. P. Capel, S. J. Roberts, and A. Zisserman. Bayesian

image super-resolution, continued. In Proceedings of Advances in Neural
Information and Proceedings Systems, pages 1089–1096, 2006.

[74] L. C. Pickup, D. P. Capel, S. J. Roberts, and A. Zisserman. Bayesian

methods for image super-resolution. The Computer Journal, 52(1):101–
113, 2009.
30 Super-Resolution Imaging

[75] L. C. Pickup, S. J. Robert, and A. Zisserman. A sampled texture prior

for image super-resolution. In Proceedings of Advances in Neural Infor-
mation and Processing System, pages 1587–1594, 2003.

[76] L. C. Pickup, S. J. Roberts, and A. Zisserman. Optimizing and learning

for super-resolution. In British Machine Vision Conference, volume 2,
pages 439–448, 2006.

[77] M. Protter and M. Elad. Super resolution with probabilistic motion

estimation. IEEE Transactions on Image Processing, 18(8):1899–1904,
2009.

[78] M. Protter, M. Elad, H. Takeda, and P. Milanfar. Generalizing the

nonlocal-means to super-resolution reconstruction. IEEE Transactions
on Image Processing, 18(1):36–51, 2009.

[79] D. Robinson and P. Milanfar. Fundamental performance limits in image

registration. IEEE Transactions on Image Processing, 13(9):1185–1199,
2004.

[80] D. Robinson and P. Milanfar. Statistical performance analysis of super-

resolution. IEEE Transactions on Image Processing, 15(6):1413–1428,
2006.

[81] L. Rudin, S. Osher, and E. Fatemi. Nonlinear total variation based noise
removal algorithms. Physica D: Nonlinear Phenomena, 60(1-4):259–268,
1992.
[82] M. Elad S. Farsiu, D. Robinson and P. Milanfar. Advances and chal-
lenges in super-resolution. International Journal of Imaing Systems and
Technology, 14(2):47–57, 2004.

[83] R. R. Schultz and R. L. Stevenson. A Bayesian approach to image

expansion for improved definition. IEEE Transactions on Image Pro-
cessing, 3(3):233–242, 1994.

[84] R. R. Schultz and R. L. Stevenson. Extraction of high-resolution

frames from video sequences. IEEE Transactions on Image Processing,
5(6):996–1011, 1996.

[85] C. A. Segall, A. K. Katsaggelos, R. Molina, and J. Mateos. Bayesian

resolution enhancement of compressed video. IEEE Transactions on
Image Processing, 13(7):898–910, 2004.
[86] C. A. Segall, R. Molina, and A. K. Katsaggelos. High resolution images
from low-resolution compressed video. IEEE Signal Processing Maga-
zine, 20(3):37–38, 2003.
 Image Super-Resolution: Historical Overview and Future Challenges 31

[87] H. Shen, L. Zhang, B. Huang, and P. Li. A MAP approach for joint mo-
tion estimation, segmentation and super-resolution. IEEE Transactions
on Image Processing, 16(2):479–490, 2007.

[88] H. Stark and P. Oskoui. High-resolution image recovery from image-

plane arrays, using convex projections. Journal of Optical Society of
America A, 6(11):1715–1726, 1989.

[89] W. Su and S. P. Kim. High-resolution restoration of dynamic image

sequences. International Journal of Imaging Systems and Technology,
5(4):330–339, 1994.

[90] J. Sun, N. N. Zheng, H. Tao, and H. Shum. Image hallucination with

primal sketch priors. In Proceedings of IEEE Computer Society Con-
ference on Computer Vision and Pattern Recognition, volume 2, pages
729–736, 2003.
[91] H. Takeda, S. Farsiu, and P. Milanfar. Kernel regression for image
processing and reconstruction. IEEE Transactions on Image Processing,
16(2):349–366, 2007.

[92] H. Takeda, P. Milanfar, M. Protter, and M. Elad. Super-resolution with-

out explicit subpixel motion estimation. IEEE Transaction on Image
Processing, 18(9):1958–1975, 2009.

[93] J. B. Tenenbaum, V. Silva, and J. C. Langford. A global geo-

metric framework for nonlinear dimensionality reduction. Science,
290(5500):2319–2323, 2000.

[94] R. Tibshirani. Regression shrinkge and selection via the Lasso. Jour-
nal of Royal Statistical Society: Series B (Statistical Methodology),
59(1):267–288, 1996.

[95] A. N. Tikhonov and V. A. Arsenin. Solution of ill-posed problems. Win-

ston & Sons, Washington, 1997.

[96] Michael E. Tipping and Christopher M. Bishop. Bayesian image super-

resolution. In Proceedings of Advances in Neural Information Proceeding
Systems, pages 1279–1286, 2003.

[97] B. C. Tom and A. K. Katsaggelos. Reconstuction of a high-resolution

image by simultaneous registration, restoration and interpolation of low-
resolution images. In Proceedings of the IEEE International Conference
on Image Processing, volume 2, page 2539, 1995.
[98] B. C. Tom, A. K. Katsaggelos, and N. P. Galatsanos. Reconstruction of a
high resolution image from registration and restoration of low resolution
images. In Proceedings of IEEE International Conference on Image
Processing, pages 553–557, 1994.
32 Super-Resolution Imaging

[99] R. Y. Tsai and T. S. Huang. Multipleframe image restoration and regis-

tration. In Advances in Computer Vision and Image Processing, pages
317–339. Greenwich, CT: JAI Press Inc., 1984.
[100] M. Turk and A. Pentland. Face recognition using eigenfaces. In Pro-
ceedings of IEEE Computer Society Conference on Computer Vision and
Pattern Recognition, pages 586–591, 1991.
[101] H. Ur and D. Gross. Improved resolution from subpixel shifted pictures.
CVGIP: Graphical Models and Image Processing, 54(2):181–186, 1992.
[102] A. W. M. van Eekeren, K. Schutte, O. R. Oudegeest, and L. J. van Vilet.
Performance evaluation of super-resolution reconstruction methods on
real-world data. EURASIP Journal on Advances in Signal Processing,
2007.
[103] Q. Wang, X. Tang, and H. Shum. Patch based blind image super-
resolution. In Proceedings of IEEE International Conference on Com-
puter Vision, volume 1, pages 709–716, 2005.
[104] X. Wang and X. Tang. Hallucinating face by eigentransformation. IEEE
Transactions on Systems, Man, and Cybernetics, 35(3):425–434, 2003.
[105] N. A. Woods, N. P. Galatsanos, and A. K. Katsaggelos. Stochastic
methods for joint registration, restoration and interpolation of mul-
tiple undersampled images. IEEE Transactions on Image Processing,
15(1):210–213, 2006.
[106] Jianchao Yang, Hao Tang, Yi Ma, and Thomas Huang. Face hallucina-
tion via sparse coding. In Proceedings of IEEE International Conference
on Image Processing, pages 1264–1267, 2008.
[107] Jianchao Yang, John Wright, Thomas Huang, and Yi Ma. Image super-
resolution as sparse representation of raw image patches. In Proceedings
of IEEE Computer Society Conference on Computer Vision and Pattern
Recognition, pages 1–8, 2008.
[108] J. S. Yedidia, W. T. Freeman, and Y. Weiss. Generalized belief prop-
agation. In Proceedings of Advances in Neural Information Processing
Systems, pages 689–695, 2001.
[109] L. J. Yen. On non-uniform sampling of bandwidth limited signals. IRE
Transactions on Circuits Theory, 3(4):251–257, 1956.
[110] D. C. Youla and H. Webb. Image registration by the method of convex
projections: Part 1-thoery. IEEE Transactions on Medical Imaging,
1(2):81–94, 1982.
[111] D. M. Young. Iterative solution of large linear systems. New York:
Academic, 1971.
 Image Super-Resolution: Historical Overview and Future Challenges 33

[112] W. Zhao and H. S. Sawhney. Is super-resolution with optical flow feasi-

ble? In Proceedings of the 7th European Conference on Computer Vision,
pages 599–613, 2002.

[113] M. V. W. Zibetti and J. Mayer. Outlier robust and edge-preserving

simultaneous super-resolution. In Proceedings of IEEE International
Conference on Image Processing, pages 1741–1744, 2006.

[114] B. Zitová and J. Flusser. Image registration methods: a survey. Image

and Vision Computing, 21(11):977–1000, 2003.

[115] A. Zomet and S. Peleg. Efficient super-resolution and applications to

mosaics. In Proceedings of International Conference on Pattern Recog-
nition, pages 579–583, 2000.

[116] A. Zomet, A. Rav-Acha, and S. Peleg. Robust super-resolution. In

Proceedings of the IEEE Workshop on Applications of Computer Vision,
pages 645–650, 2001.
Image Super-Resolution: Historical Overview and Future Challenges
M. S. Alam , J. G. Bognar , R. C. Hardie , and B. J. Yasuda . Infrared image registration and high-resolution reconstruction using multiple
translationally shifted aliased video frames. IEEE Transactions on Instrumentation and Measurement, 49(5): 915923, 2000.
S. Baker and T. Kanade . Limits on super-resolution and how to break them. IEEE Transactions on Pattern Analysis and Machine
Intelligence, 24(9): 11671183, 2002.
S. Borman and R. L. Stevenson . Simultaneous multi-frame MAP super-resolution video enhancement using spatio-temporal priors. In
Proceedings of IEEE International Conference on Image Processing, volume 3, pages 469473, 1999.
Sean Borman and Robert L. Stevenson . Super-resolution from image sequences - A review. In Proceedings of the 1998 Midwest
Symposium on Circuits and Systems, pages 374378, 1998.
N. K. Bose , H. C. Kim , and H. M. Valenzuela . Recursive implementation of total least squares algorithm for image reconstruction from
noisy, undersampled multiframes. In Proceedings of the IEEE Conference on Acoustics, Speech and Signal Processing, volume 5, pages
269272, 1993.
O. Bowen and C. S. Bouganis . Real-time image super resolution using an FPGA. In International Conference on Field Programmable
Logic and Applications, pages 8994, 2008.
L. Brown. A survey of image registration techniques. ACM Computing Surveys, 24(4): 325376, 1992.
A. Buades , B. Coll , and J. M. More . A non-local algorithm for image denoising. In Proceedings of IEEE Computer Society Conference on
Computer Vision and Pattern Recognition, pages 6065, 2005.
E. Candes . Compressive sensing. In Proceedings of International Congress of Mathematicians, volume 3, pages 14331452, 2006.
D. Capel . Image Mosaicing and Super-resolution. Springer, 2004.
D. Capel and A. Zisserman . Automated mosaicing with super-resolution zoom. In Proceedings of IEEE Computer Society Conference on
Computer Vision and Pattern Recognition, pages 885891, 1998.
D. Capel and A. Zisserman . Super-resolution enhancement of text image sequences. In Proceedings of the International Conference on
Pattern Recognition, volume 1, pages 16001605, 2000.
25 D. Capel and A. Zisserman . Super-resolution enhancement of text image sequences. In Proceedings of 15th International Conference
on Pattern Recognition, pages 600605, 2000.
D. Capel and A. Zisserman . Super-resolution from multiple views using learnt image models. In Proceedings of IEEE Computer Society
Conference on Computer Vision and Pattern Recognition, volume 2, pages 627634, 2001.
D. Capel and A. Zisserman . Computer vision applied to super-resolution. IEEE Signal Processing Magazine, 20(3): 7586, 2003.
T. F. Chan , S. Osher , and J. Shen . The digital TV filter and nonlinear denosing. IEEE Transaction on Image Processing, 10(2): 231241,
2001.
H. Chang , D. Y. Yeung , and Y. Xiong . Super-resolution through neighbor embedding. In Proceedings of IEEE Computer Society
Conference on Computer Vision and Pattern Recognition, volume 1, pages 275282, 2004.
M. C. Chiang and T. E. Boulte . Efficient super-resolution via image warping. Image and Vision Computing, 18(10): 761771, 2000.
J. Chung , E. Haber , and J. Nagy . Numerical methods for coupled super-resolution. Inverse Problems, 22(4): 12611272, 2006.
Marco Crisani , Dong Seon Cheng , Vittorio Murino , and Donato Pannullo . Distilling information with super-resolution for video surveil-
lance. In Proceedings of the ACM 2nd International Workshop on Video Surveillance and Sensor Networks, pages 211, 2004.
D. Datsenko and M. Elad . Example-based single document image super-resolution: a global MAP approach with outlier rejection.
Multidimensional System and Signal Processing, 18(23):103121, 2007.
D. L. Donoho . Compressed sensing. IEEE Transactions on Information Theory, 52(4): 12891306, 2006.
D. L. Donoho . For most large underdetermined systems of linear equations, the minimal 1-norm near-solution approximates the sparsest
near-solution. Communications on Pure and Applied Mathematics, 59(7): 907934, 2006.
A. A. Efros and T. K. Leung . Texture synthesis by non-parametric sampling. In Proceedings of IEEE International Conference on
Computer Vision, pages 10331038, 1999.
M. Elad and D. Datsenko . Example-based regularization deployed to super-resolution reconstruction of a single image. The Computer
Journal, 52(1): 1530, 2007.
26 M. Elad and A. Feuer . Restoration of single super-resolution image from several blurred, noisy and down-sampled measured images.
IEEE Transaction on Image Processing, 6(12): 16461658, 1997.
M. Elad and Y. Hel-Or . A fast super-resolution reconstruction algorithm for pure translational motion and common space invariant blur.
IEEE Transactions on Image Processing, 10(8): 11871193, 2001.
P. E. Eren , M. I. Sezan , and A. M. Tekalp . Robust, object-based high resolution image reconstruction from low-resolution video. IEEE
Transactions on Image Processing, 6(10): 14461451, 1997.
X. Jia F. Li and D. Fraser . Universal HMT based super resolution for remote sensing images. In IEEE International Conference on Image
Processing, pages 333336, 2008.
S. Farsiu , D. Robinson , M. Elad , and P. Milanfar . Fast and robust multi-frame super-resolution. IEEE Transaction on Image Processing,
13(10): 13271344, 2004.
W. T. Freeman , T. R. Jones , and E. C. Pasztor . Example-based super-resolution. IEEE Computer Graphics and Applications, 22(2):
5665, 2002.
B. K. Gunturk , A. U. Batur , Y. Altunbasak , M. H. Hayes , and R. M. Mersereau . Eigenface-domain super-resolution for face recognition.
IEEE Transactions on Image Processing, 12(5): 597606, 2003.
K. M. Hanson and G. W. Wecksung . Bayesian approach to limited-angle reconstruction in computed tomography. Journal of Optical
Society of America, 73(11): 15011509, 1983.
R. Hardie . A fast image super-resolution algorithm using an adaptive Wiener filter. IEEE Transactions on Image Processing, 16(12):
29532964, 2007.
R. C. Hardie , K. J. Barnard , and E. E. Armstrong . Join MAP registration and high resolution image estimation using a sequence of
undersampled images. IEEE Transactions on Image Processing, 6(12): 16211633, 1997.
P. H. Hennings-Yeomans , S. Baker , and B. V. K. V. Kumar . Simultaneous super-resolution and feature extraction for recognition of low-
resolution faces. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 18, 2008.
G. T. Herman , H. Hurwitz , A. Lent , and H.P. Lung . On the Bayesian approach to image reconstruction. Information and Control, 42(1):
6071, 1979.
27 A. Hertzmann , C. E. Jacobs , N. Oliver , B. Curless , and D. H. Salesin . Image analogies. In Proceedings of the 28th annual
conference Computer Graphics and Interactive Techniques, pages 327340, 2001.
M. Irani and S. Peleg . Super resolution from image sequences. In Proceedings of 10th International Conference on Pattern Recognition,
volume 2, pages 115120, 1990.
M. Irani and S. Peleg . Improving resolution by image registration. CVGIP: Graphical Models and Imaging Processing, 53(3): 231239,
1991.
M. Irani and S. Peleg . Motion analysis for image enhancement: resolution, occlusion and tranparency. Journal of Visual Communications
and Image Representation, 4(4): 324335, 1993.
T. S. Huang J. Yang , J. Wright and Y. Ma . Super-resolution via sparse representation. Submitted to IEEE Transactions on Image
Processing, 2009.
K. Jia and S. Gong . Multi-model tensor face for simultaneous super-resolution and recognition. In Proceedings of IEEE International
Conference on Computer Vision, volume 2, pages 16831690, 2005.
K. Jia and S. Gong . Generalized face super-resolution. IEEE Transactions on Image Processing, 17(6): 873886, 2008.
I. T. Jolliffe . Principal Component Analysis. Series: Springer Series in Statistics, 2002.
E. Kaltenbacher and R. C. Hardie . High-resolution infrared image reconstruction using multiple low resolution aliased frames. In
Proceedings of the IEEE National Aerospace Electronics Conference, volume 2, pages 702709, 1996.
J. A. Kennedy , O. Israel , A. Frenkel , R. Bar-Shalom , and A. Haim . Super-resolution in PET imaging. IEEE Transactions on Medical
Imaging, 25(2): 137147, 2006.
D. Keren , S. Peleg , and R. Brada . Image sequence enhancement using subpixel displacements. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, pages 742746, 1988.
S. P. Kim , N. K. Bose , and H. M. Valenzuela . Recursive reconstruction of high resolution image from noisy undersampled multiframes.
IEEE Transactions on Acoustics, Speech and Signal Processing, 38(6): 10131027, 1990.
H. Knutsson and C.F. Westin . Normalized and differential convolution. In Proceedings of IEEE Computer Society Conference on
Computer Vision and Pattern Recognition, pages 515523, 1993.
28 D. D. Lee and H. S. Seung . Learning the parts of objects by non-negative matrix factorizaiton. Nature, 401(6755): 788791, 1999.
S. Lerttrattanapanich and N. K. Bost . High resolution image formation from low resolution frames using delaunay triangulation. IEEE
Transaction on Image Processing, 11(12): 14271441, 2002.
A. Levin , Y. Weiss , F. Durand , and W. Freeman . Understanding and evaluating blind deconvolution algorithms. In Proceedings of IEEE
Computer Society Conference on Computer Vision and Pattern Recognition, pages 19641971, 2009.
Y. Li and F. Santosa . A computational algorithm for minimizing total variation in image restoration. IEEE Transactions on Image
Processing, 5(6): 987995, 1996.
Frank Lin, Clinton B. Fookes , Vinod Chandran , and Sridha Sridharan . Investigation into optical flow super-resolution for surveillance
applications. In The Austrilian Pattern Recognition Society Worshop on Digital Image Computing, 2005.
Z. Lin and H.Y. Shum . Fundamental limits on reconstruction-based superresolution algorithms under local translation. IEEE Transactions
on Pattern Analysis and Machine Intelligence, 26(1): 8397, 2004.
C. Liu , H. Y. Shum , and W. T. Freeman . Face hallucination: theory and practice. International Journal of Computer Vision, 75(1):
115134, 2007.
C. Liu , H. Y. Shum , and C. S. Zhang . Two-step approach to hallucinating faces: global parametric model and local nonparametric model.
In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 1, pages 192198, 2001.
J. Maintz and M. Viergever . A survey of medical image registration. Medical Image Analysis, 2(1): 136, 1998.
K. Malczewski and R. Stasinski . Toeplitz-based iterative image fusion scheme for MRI. In IEEE International Conference on Image
Processing, pages 341344, 2008.
B. Narayanan , R. C. Hardie , K. E. Barner , and M. Shao . A computationally efficient super-resolution algorithm for video processing
using partition filters. IEEE Transactions on Circuits and Systems for Video Technology, 17(5): 621634, 2007.
N. Nguyen and P. Milanfar . An efficient wavelet-based algorithm for image super-resolution. In Proceedings of International Conference
on Image Processing, volume 2, pages 351354, 2000.
29 N. Nguyen , P. Milanfar , and G. H. Golub . A computationally efficient image superresolution algorithm. IEEE Transactions on Image
Processing, 10(5): 573583, 2001.
A. Papulis . Generalized sampling expansion. IEEE Transactions on Circuits and Systems, 24(11): 652654, 1977.
Sung C. Park , Min K. Park , and Moon G. Kang . Super-resolution image reconstruction: a technical overview. IEEE Signal Processing
Magazine, 20(3): 2136, 2003.
A. J. Patti and Y. Altunbasak . Aritifact reduction for set theoretic super resolution image reconstruction with edge adaptive constraints and
higher-order interpolants. IEEE Transaction on Image Processing, 10(1): 179186, 2001.
A. J. Patti , M. Sezan , and A. M. Tekalp . Robust methods for high quality stills from interlaced video in the presence of dominant motion.
IEEE Transactions on Circuits and Systems for Video Technology, 7(2): 328342, 1997.
A. J. Patti , M. I. Sezan , and A. M. Tekalp . High-resolution image reconstruction from a low-resolution image sequence in the presence of
time-varing motion blur. In Proceedings of the IEEE International Conference on Image Processing, volume 1, pages 343347, 1994.
A. J. Patti , M. I. Sezan , and A. M. Tekalp . Superresolution video reconstruction with arbitrary sampling lattices and nonzero aperture
time. IEEE Transactions on Image Processing, 6(8): 10641076, 1997.
S. Peleg and Y. Yeshurun . Superresolution in MRI: application to human white matter fiber tract visualization by diffusion tensor imaging.
Magnetic Resonance in Medicine, 45(1): 2935, 2001.
T. Q. Pham , L. J. Vliet , and K. Schutte . Robust fusion of irregularly sampled data using adaptive normalized convolution. EURASIP
Journal on Applied Signal Processing, 2006.
L. C. Pickup , D. P. Capel , S. J. Robert , and A. Zisserman . Over-coming registration uncertainty in image super-resolution: maximize or
marginalize? EURASIP Journal on Advances in Signal Processing, 2007.
L. C. Pickup , D. P. Capel , S. J. Roberts , and A. Zisserman . Bayesian image super-resolution, continued. In Proceedings of Advances in
Neural Information and Proceedings Systems, pages 10891096, 2006.
L. C. Pickup , D. P. Capel , S. J. Roberts , and A. Zisserman . Bayesian methods for image super-resolution. The Computer Journal, 52(1):
101113, 2009.
30 L. C. Pickup , S. J. Robert , and A. Zisserman . A sampled texture prior for image super-resolution. In Proceedings of Advances in
Neural Information and Processing System, pages 15871594, 2003.
L. C. Pickup , S. J. Roberts , and A. Zisserman . Optimizing and learning for super-resolution. In British Machine Vision Conference,
volume 2, pages 439448, 2006.
M. Protter and M. Elad . Super resolution with probabilistic motion estimation. IEEE Transactions on Image Processing, 18(8): 18991904,
2009.
M. Protter , M. Elad , H. Takeda , and P. Milanfar . Generalizing the nonlocal-means to super-resolution reconstruction. IEEE Transactions
on Image Processing, 18(1): 3651, 2009.
D. Robinson and P. Milanfar . Fundamental performance limits in image registration. IEEE Transactions on Image Processing, 13(9):
11851199, 2004.
D. Robinson and P. Milanfar . Statistical performance analysis of super-resolution. IEEE Transactions on Image Processing, 15(6):
14131428, 2006.
L. Rudin , S. Osher , and E. Fatemi . Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena, 60(1-
4):259268, 1992.
M. Elad S. Farsiu , D. Robinson and P. Milanfar . Advances and challenges in super-resolution. International Journal of Imaing Systems
and Technology, 14(2): 4757, 2004.
R. R. Schultz and R. L. Stevenson . A Bayesian approach to image expansion for improved definition. IEEE Transactions on Image
Processing, 3(3): 233242, 1994.
R. R. Schultz and R. L. Stevenson . Extraction of high-resolution frames from video sequences. IEEE Transactions on Image Processing,
5(6): 9961011, 1996.
C. A. Segall , A. K. Katsaggelos , R. Molina , and J. Mateos . Bayesian resolution enhancement of compressed video. IEEE Transactions
on Image Processing, 13(7): 898910, 2004.
C. A. Segall , R. Molina , and A. K. Katsaggelos . High resolution images from low-resolution compressed video. IEEE Signal Processing
Magazine, 20(3): 3738, 2003.
31 H. Shen , L. Zhang , B. Huang , and P. Li . A MAP approach for joint motion estimation, segmentation and super-resolution. IEEE
Transactions on Image Processing, 16(2): 479490, 2007.
H. Stark and P. Oskoui . High-resolution image recovery from image-plane arrays, using convex projections. Journal of Optical Society of
America A, 6(11): 17151726, 1989.
W. Su and S. P. Kim . High-resolution restoration of dynamic image sequences. International Journal of Imaging Systems and Technology,
5(4): 330339, 1994.
J. Sun , N. N. Zheng , H. Tao , and H. Shum . Image hallucination with primal sketch priors. In Proceedings of IEEE Computer Society
Conference on Computer Vision and Pattern Recognition, volume 2, pages 729736, 2003.
H. Takeda , S. Farsiu , and P. Milanfar . Kernel regression for image processing and reconstruction. IEEE Transactions on Image
Processing, 16(2): 349366, 2007.
H. Takeda , P. Milanfar , M. Protter , and M. Elad . Super-resolution without explicit subpixel motion estimation. IEEE Transaction on
Image Processing, 18(9): 19581975, 2009.
J. B. Tenenbaum , V. Silva , and J. C. Langford . A global geometric framework for nonlinear dimensionality reduction. Science, 290(5500):
23192323, 2000.
R. Tibshirani . Regression shrinkge and selection via the Lasso. Journal of Royal Statistical Society: Series B (Statistical Methodology),
59(1): 267288, 1996.
A. N. Tikhonov and V. A. Arsenin . Solution of ill-posed problems. Winston & Sons, Washington, 1997.
Michael E. Tipping and Christopher M. Bishop . Bayesian image super-resolution. In Proceedings of Advances in Neural Information
Proceeding Systems, pages 12791286, 2003.
B. C. Tom and A. K. Katsaggelos . Reconstuction of a high-resolution image by simultaneous registration, restoration and interpolation of
low-resolution images. In Proceedings of the IEEE International Conference on Image Processing, volume 2, page 2539, 1995.
B. C. Tom , A. K. Katsaggelos , and N. P. Galatsanos . Reconstruction of a high resolution image from registration and restoration of low
resolution images. In Proceedings of IEEE International Conference on Image Processing, pages 553557, 1994.
32 R. Y. Tsai and T. S. Huang . Multipleframe image restoration and registration. In Advances in Computer Vision and Image Processing,
pages 317339. Greenwich, CT: JAI Press Inc., 1984.
M. Turk and A. Pentland . Face recognition using eigenfaces. In Proceedings of IEEE Computer Society Conference on Computer Vision
and Pattern Recognition, pages 586591, 1991.
H. Ur and D. Gross . Improved resolution from subpixel shifted pictures. CVGIP: Graphical Models and Image Processing, 54(2): 181186,
1992.
A. W. M. van Eekeren , K. Schutte , O. R. Oudegeest , and L. J. van Vilet . Performance evaluation of super-resolution reconstruction
methods on real-world data. EURASIP Journal on Advances in Signal Processing, 2007.
Q. Wang , X. Tang , and H. Shum . Patch based blind image super-resolution. In Proceedings of IEEE International Conference on
Computer Vision, volume 1, pages 709716, 2005.
X. Wang and X. Tang . Hallucinating face by eigentransformation. IEEE Transactions on Systems, Man, and Cybernetics, 35(3): 425434,
2003.
N. A. Woods , N. P. Galatsanos , and A. K. Katsaggelos . Stochastic methods for joint registration, restoration and interpolation of multiple
undersampled images. IEEE Transactions on Image Processing, 15(1): 210213, 2006.
Jianchao Yang , Hao Tang , Yi Ma , and Thomas Huang . Face hallucination via sparse coding. In Proceedings of IEEE International
Conference on Image Processing, pages 12641267, 2008.
Jianchao Yang , John Wright , Thomas Huang , and Yi Ma . Image super-resolution as sparse representation of raw image patches. In
Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 18, 2008.
J. S. Yedidia , W. T. Freeman , and Y. Weiss . Generalized belief propagation. In Proceedings of Advances in Neural Information
Processing Systems, pages 689695, 2001.
L. J. Yen . On non-uniform sampling of bandwidth limited signals. IRE Transactions on Circuits Theory, 3(4): 251257, 1956.
D. C. Youla and H. Webb . Image registration by the method of convex projections: Part 1-thoery. IEEE Transactions on Medical Imaging,
1(2): 8194, 1982.
D. M. Young . Iterative solution of large linear systems. New York: Academic, 1971.
33 W. Zhao and H. S. Sawhney . Is super-resolution with optical flow feasible? In Proceedings of the 7th European Conference on
Computer Vision, pages 599613, 2002.
M. V. W. Zibetti and J. Mayer . Outlier robust and edge-preserving simultaneous super-resolution. In Proceedings of IEEE International
Conference on Image Processing, pages 17411744, 2006.
B. Zitov and J. Flusser . Image registration methods: a survey. Image and Vision Computing, 21(11): 9771000, 2003.
A. Zomet and S. Peleg . Efficient super-resolution and applications to mosaics. In Proceedings of International Conference on Pattern
Recognition, pages 579583, 2000.
A. Zomet , A. Rav-Acha , and S. Peleg . Robust super-resolution. In Proceedings of the IEEE Workshop on Applications of Computer
Vision, pages 645650, 2001.34
Super-Resolution Using Adaptive Wiener Filters
K. Aizawa , T. Komatsu , and T. Saito . Acquisition of very high resolution images using stereo cameras. In Proc. SPIE Visual
Communication and Image Processing, volume 1, pages 318328, Boston, MA, Nov 1991.
M. S. Alam , J. G. Bognar , R. C. Hardie , and B. J. Yasuda . Infrared image registration using multiple translationally shifted aliased video
frames. IEEE Instrum. Meas. Mag., 49(5), Oct 2000.
K. E. Barner , A. M. Sarhan , and R. C. Hardie . Partition-based weighted sum filters for image restoration. IEEE Trans. Image Process.,
8(5): 740745, May 1999.
B. Bascle , A. Blake , and A. Zisserman . Motion deblurring and super resolution from an image sequence. EECV, 2:573581, Apr 1996.
James R. Bergen , P. An , Th J. Hanna , and Rajesh Hingorani . Hierarchical model-based motion estimation. In Proceedings of the
Second European Conference on Computer Vision, pages 237252. Springer-Verlag, 1992.
S. Cain , R. C. Hardie , and E. E. Armstrong . Restoration of aliased video sequences via a maximum-likelihood approach. In Infrared
Information Symposium (IRIS) on Passive Sensors, Monterey, CA, volume 1, pages 377390, March 1996.
59 R. Chan , T. Chan , M. Ng , W. Tang , and C. Wong . Preconditioned iterative methods for high-resolution image reconstruction from
multisensors. Adv. Signal Process. Algorithms, Architectures, Implementations, 3461:348357, 1998.
P. Cheeseman , B. Kanefsky , R. Kraft , J. Stutz , and R. Hanson . Super-resolved surface reconstruction from multiple images. NASA
Tech. Rep. FIA-94-12, Dec 1994.
T. Connolly and R. Lane . Gradient methods for superresolution. In Proc. Int. Conf. Image Processing, volume 1, pages 917920, 1997.
M. Elad and Y. Hel-Or . A fast super-resolution reconstruction algorithm for pure translational motion and common space invariant blur.
IEEE Trans. Image Process., 10(8): 11871193, Aug 2001.
S. Farsiu , D. Robinson , M. Elad , and P. Milanfar . Fast and robust multiframe super-resolution. IEEE Trans. Image Process., 13(10):
13271344, Oct 2004.
J. C. Gillette , T. M. Stadtmiller , and R. C. Hardie . Reduction of aliasing in staring infrared imagers utilizing subpixel techniques. Opt.
Eng., 34(11): 31303137, Nov 1995.
J.W. Goodman . Introduction to Fourier Optics. McGraw-Hill, 1968.
R. C. Hardie . A fast super-resolution algorithm using an adaptive wiener filter. IEEE Trans. Image Process., 16:29532964, Dec 2007.
R. C. Hardie and K. J. Barnard . Fast adaptive wiener filter based super-resolution applied to affine motion. Submitted to IEEE
Transactions on Image Processing, September 2009.
R. C. Hardie , K. J. Barnard , and E. E. Armstrong . Joint MAP registration and high-resolution image estimation using a sequence of
undersampled images. IEEE Trans. Image Process., 6(12): 16211633, Dec 1997.
R. C. Hardie , K. J. Barnard , J. G. Bognar , E. E. Armstrong , and E. A. Watson . High-resolution image reconstruction from a sequence of
rotated and translated frames and its application to an infrared imaging system. Opt. Eng., 37(1): 247260, Jan 1998.
M. Irani and S. Peleg . Improving resolution by image registration. CHIP: Graph. Models Image Process., 53(3): 231239, May 1991.
S. Lertrattanapanich and N. K. Bose . High resolution image formation from low resolution frames using delaunay triangulation. IEEE
Trans. Image Process., 11(12): 14271441, Dec 2002.
60 Y. Lin , R. C. Hardie , Q. Sheng , M. Shao , and K. E. Barner . Improved optimization of soft-partition-weighted sum filters and their
application to image restoration. Applied Optics, 45(12): 26972706, April 2006.
S. Mann and R. Picard . Virtual bellows: construction of high quality stills from video. In Proc. IEEE Int. Conf. Image Processing, Austin,
TX, Nov 1994.
B. Narayanan , R. C. Hardie , K. E. Barner , and M. Shao . A computationally efficient super-resolution algorithm for video processing
using partition filters. IEEE Trans. Circuits Syst. Video Technol., 17(5): 621634, May 2007.
N. Nguyen and P. Milanfar . A wavelet-based interpolation restoration method for superresolution. Circuits, Syst., Signal Process., 19(4):
321338, Aug 2000.
N. Nguyen , P. Milanfar , and G. Golub . A computationally efficient super-resolution image reconstruction algorithm. IEEE Trans. Image
Process., 10(4): 573583, Apr 2001.
S. C. Park , M. K. Park , and M. G. Kang . Super-resolution image reconstruction: A technical overview. IEEE Signal Process. Mag., pages
2136, May 2003.
A. Patti , M. Sezan , and A. Teklap . Superresolution video reconstruction with arbitrary sampling lattices and nonzero aperture time. IEEE
Trans. Image Process., 6(8): 10641076, Aug 1997.
S. Periaswamy and H. Farid . Elastic registration in the presence of intensity variations. IEEE Transactions on Medical Imaging, 22(7):
865874, 2003.
T. Q. Pham , L. J. van Vliet , and K. Schutte . Robust fusion of irregularly sampled data using adaptive normalized convolution. EURASIP
Journal on Applied Signal Processing, 2006(Article ID 83268):112, 2006.
L. C. Pickup , D. P. Capel , S. J. Roberts , and A. Zisserman . Overcoming registration uncertainty in image super-resolution: Maximize or
marginalize? EURASIP Journal on Advances in Signal Processing, 2007:Article ID 23565, 14 pages, 2007.
K. Sauer and J. Allebach . Iterative reconstruction of band-limited images from non-uniformly spaced samples. IEEE Trans. Circuits Syst.,
CAS-34:14971505, 1987.
R. R. Schultz and R. L. Stevenson . Extraction of high-resolution frames from video sequences. IEEE Trans. Image Process., 5:9961011,
June 1996.
61 R. R. Shannon . Aberrations and their effect on images. In Proceedings SPIE, Geometric Optics, Critical Review of Technology, volume
531, pages 2737.
M. Shao , K. E. Barner , and R. C. Hardie . Partition-based interpolation for image demosaicking and super-resolution reconstruction. Opt.
Eng., 44:107003110700314, Oct 2005.
H. Shekarforoush and R. Chellappa . Data-driven multi-channel super-resolution with application to video sequences. J. Opt. Soc. Amer.
A, 16(3): 481492, Mar 1999.
J. Shi , S. E. Reichenbach , and J. D. Howe . Small-kernel superresolution methods for microscanning imaging systems. Applied Optics,
45(6): 12031214, Feb 2006.
A. Tekalp , M. Ozkan , and M. Sezan . High resolution image reconstruction from lower-resolution image sequences and space-varying
image restoration. In Proc. ICASSP 92, volume 3, pages 169172, San Fransisco, CA, Mar 1992.
C. W. Therrian . Discrete Random Signals and Statistical Signal Processing. Prentice Hall, 1992.
E. Trucco and A. Verri . Introductory Techniques for 3-D Computer Vision. Prentice Hall, 1998.
T. R. Tuinstra and R. C. Hardie . High resolution image reconstruction from digital video by exploitation on non-global motion. Opt. Eng.,
38(5), May 1999.62
Locally Adaptive Kernel Regression for Space-Time Super-Resolution
M. J. Black and P. Anandan . The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Computer Vision
and Image Understanding, 63(1): 75104, January 1996.
A. Buades , B. Coll , and J. M. Morel . A review of image denosing algorithms, with a new one. Multiscale Modeling and Simulation,
Society for Industrial and Applied Mathematics (SIAM) Interdisciplinary Journal, 4(2): 490530, 2005.
93 S. Chaudhuri and S. Chatterjee . Performance analysis of total least squares methods in three-dimensional motion estimation. IEEE
Transactions on Robotics and Automation, 7(5): 707714, October 1991.
K. Dabov , A. Foi , V. Katkovnic , and K. Egiazarian . Image denoising by sparse 3D transform-domain collaborative filtering. IEEE
Transactions of Image Processing, 16(8): 20802095, August 2007.
A. Danielyan , A. Foi , V. Katkovnik , and K. Egiazarian . Image and video super-resolution via spatially adaptive block-matching filtering.
Proceedings of International Workshop on Local and Non-Local Approximation in Image Processing (LNLA), August 2008. Lausanne,
Switzerland.
M. Elad and Y. Hel-Or . A fast super-resolution reconstruction algorithm for pure translational motion and common space-invariant blur.
IEEE Transactions on Image Processing, 10(8): 11871193, August 2001.
Gunnar Farnebck . Polynomial Expansion for Orientation and Motion Estimation. PhD thesis, Linkping University, Sweden, SE-581 83
Linkping, Sweden, 2002. Dissertation No 790, ISBN 91-7373-475-6.
S. Farsiu , D. Robinson , M. Elad , and P. Milanfar . Fast and robust multi-frame super-resolution. IEEE Transactions on Image
Processing, 13(10): 13271344, October 2004.
W. T. Freeman , T. R. Jones , and E. C. Pasztor . Example-based super resolution. IEEE Computer Graphics, 22(2): 5665, March/April
2002.
H. Fu and J. Barlow . A regularized structured total least squares algorithm for high-resolution image reconstruction. Linear Algebra and its
Applications, 391:7598, November 2004.
J. J. Gibson . The Perception of the Visual World. Houghton Mifflin, Boston, 1950.
B. K. Gunturk , Y. Altunbasak , and R. M. Mersereau . Multiframe resolution enhancement methods for compressed video. IEEE Signal
Processing Letter, 9:170174, June 2002.
R. M. Haralick . Edge and region analysis for digital image data. Computer Graphic and Image Processing (CGIP), 12(1): 6073, January
1980.
W. Hardle and P. Vieu . Kernel regression smoothing of time series. Journal of Time Series Analysis, 13:209232, 1992.
B. K. Horn . Robot Vision. MIT Press, Cambridge, 1986.
94 M. Irani and S. Peleg . Super resolution from image sequence. Proceedings of 10th International Conference on Pattern Recognition
(ICPR), 2:115120, 1990.
S. Kanumuri , O. G. Culeryuz , and M. R. Civanlar . Fast super-resolution reconstructions of mobile video using warped transforms and
adaptive thresholding. Proceedings of SPIE, 6696:66960T, 2007.
H. Knutsson . Representing local structure using tensors. Proceedings of the 6th Scandinavian Conference on Image Analysis, pages
244251, 1989.
H. Knutsson and C. F. Westin . Normalized and differential convolutionmethods for interpolation and filtering of incomplete and uncertain
data. Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Regocnition (CVPR), pages 515523, June
1993.
D. N. Lee and H. Kalmus . The optic flow field: the foundation of vision. Philosophical Transactions of the Royal Society of London Series
B-Biological Sciences, 290(1038):169179, 1980.
B. Lucas and T. Kanade . An iterative image registration technique with an application to sterio vision. Proceedings of DARPA Image
Understanding Workshop, pages 121130, 1981.
F. Luisier , T. Blu , and M. Unser . A new sure approach to image denoising: inter-scale orthonormal wavelet thresholding. IEEE
Transactions on Image Processing, 16(3): 593606, March 2007.
E. A. Nadaraya . On estimating regression. Theory of Probability and its Applications, pages 141142, September 1964.
M. K. Ng , J. Koo , and N. K. Bose . Constrained total least squares computations for high resolution image reconstruction with
multisensors. International Journal of Imaging Systems and Technology, 12:3542, 2002.
N. Nguyen , P. Milanfar , and G. H. Golub . Efficient generalized cross-validation with applications to parametric image restoration and
resolution enhancement. IEEE Transactions on Image Processing, 10(9): 12991308, September 2001.
K. S. Ni , S. Kumar , N. Vasconcelos , and T. Q. Nguyen . Single image superresolution based on support vector regression. Proceedings
of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2, May 2006.
S.C. Park , M.K. Park , and M.G. Kang . Super-resolution image reconstruction: A technical overview. IEEE Signal Processing Magazine,
20(3): 2136, May 2003.
95 M. Protter , M. Elad , H. Takeda , and P. Milanfar . Generalizing the nonlocal-means to super-resolution reconstruction. IEEE
Transactions on Image Processing, 16(2): 3651, January 2009.
D. Robinson and P. Milanfar . Statistical performance analysis of super-resolution. IEEE Transactions on Image Processing, 15(6):
14131428, June 2006.
D. Ruppert and M. P. Wand . Multivariate locally weighted least squares regression. The annals of statistics, 22(3): 13461370, September
1994.
H. Takeda , S. Farsiu , and P. Milanfar . Kernel regression for image processing and reconstruction. IEEE Transactions on Image
Processing, 16(2): 349366, February 2007.
H. Takeda , S. Farsiu , and P. Milanfar . Deblurring using regularized locally-adaptive kernel regression. IEEE Transactions on Image
Processing, 17(4): 550563, April 2008.
H. Takeda , P. van Beek , and P. Milanfar . Spatiotemporal video upscaling using motion-assisted steering kernel (MASK) regression
(Book chapter). High-Quality Visual Experience: Creation, Processing and Interactivity of High-Resolution and High-Dimensional Video
Signals, Springer-Verlag, 2010.
J. van de Weijer and R. van den Boomgaard . Least squares and robust estimation of local image structure. Scale Space. International
Conference, 2695(4): 237254, 2003.
P. Vandewalle , L. Sbaiz , M. Vetterli , and S. Susstrunk . Super-resolution from highly undersampled images. Proceedings of International
Conference on Image Processing (ICIP), pages 889892, September 2005. Genova, Italy.
M. P. Wand and M. C. Jones . Kernel Smoothing. Monographs on Statistics and Applied Probability. Chapman and Hall, London; New
York, 1995.
K. Q. Weinberger and G. Tesauro . Metric learning for kernel regression. Proceedings of the Eleventh International Workshop on Artificial
Intelligence and Statistics, pages 608615, 2007. (AISTATS-07), Puerto Rico.
N. A. Woods , N. P. Galatsanos , and A. K. Katsaggelos . Stochastic methods for joint registration, restoration, and interpolation of multiple
under-sampled images. IEEE Transactions on Image Processing, 15(1): 201213, January 2006.
96 J. Wright and R. Pless . Analysis of persistent motion patterns using the 3d structure tensor. Proceedings of the IEEE Workshop on
Motion and Video Computing, 2005.
S. Zhu and K. Ma . A new diamond search algorithm for fast block-matching motion estimation. IEEE Transactions on Image Processing,
9(2): 287290, February 2000.
X. Zhu and P. Milanfar . Automatic parameter selection for denoising algorithms using a no-reference measure of image content. Accepted
for publication in IEEE Transactions on Image Processing.
A. Zomet , A. Rav-Acha , and S. Peleg . Robust super-resolution. Proceedings of the International Conference on Computer Vision and
Pattern Recognition (CVPR), 2001. Hawaii.

Super-Resolution with Probabilistic Motion Estimation

[1] S. Baker and T. Kanade . Limits on super-resolution and how to break them. IEEE Transactions on Pattern Analysis and Machine
Intelligence, 24(1): 11671183, September 2002.
[2] A. Buades , B. Coll , and J.M. Morel . Denoising image sequences does not require motion estimation. In Proc. IEEE Conference on
Advanced Video and Signal Based Surveillance September (AVSS), pages 7074, 2005.
[3] G.M. Callic , S. Lpez , O. Sosa , J.F. Lopez , and R. Sarmiento . Analysis of fast block matching motion estimation algorithms for video
super-resolution systems. IEEE Transactions. on Consumer Electronics, 54(3): 14301438, August 2008.
[4] M. Elad and Y. Hel-Or . A fast super-resolution reconstruction algorithm for pure translational motion and common space-invariant blur.
IEEE Transactions on Image Processing, 10(8): 11871193, August 2001.
[5] S. Farsiu , D. Robinson , M. Elad , and P. Milanfar . Robust shift and add approach to superresolution. In SPIE Conference on
Applications of Digital Signal and Image Processing, pages 121130, August 2003.
[6] S. Farsiu , D. Robinson , M. Elad , and P. Milanfar . Advances and challenges in superresolution. International Journal of Imaging
Systems and Technology, 14(2): 4757, August 2004.
120 [7] S. Farsiu , D. Robinson , M. Elad , and P. Milanfar . Fast and robust multiframe superresolution. IEEE Transactions on Image
Processing, 13(10): 13271344, October 2004.
[8] R.C. Hardie , K.J. Barnard , and E.E. Armstrong . Joint map registration and high-resolution image estimation using a sequence of
undersampled images. IEEE Transactions on Image Processing, 6(12): 16211633, December 1997.
[9] M. Irani and S. Peleg . Improving resolution by image registration. CVGIP: Graphical Models and Image Processing, 53(3): 231239,
May 1991.
[10] M. Mahamoudi and G. Sapiro . Fast image and video denoising via nonlocal means of similar neighbourhoods. IEEE Signal
Processing Letters, 12(12): 839842, December 2005.
[11] L.C. Pickup , D.P. Capel , S.J. Roberts , and A. Zisserman . Overcoming registration uncertainty in image super-resolution: Maximize
or marginalize? EURASIP Journal on Advances In Signal Processing, 2007:Article ID 23565, 14 pages, 2007.
[12] M. Protter and M. Elad . Super-resolution with probabilistic motion estimation. IEEE Transactions on Image Processing, 18(8):
18991904, August 2009.
[13] M. Protter , M. Elad , H. Takeda , and P. Milanfar . Generalizing the nonlocal-means to super-resolution reconstruction. IEEE
Transactions on Image Processing, 18(1): 3651, January 2009.
[14] L.I. Rudin , S.J. Osher , and E. Fatemi . Nonlinear total variation based noise removal algorithms. Physica D, 60:259268, 1992.
[15] R.R. Schultz and R.L. Stevenson . Extraction of high-resolution frames from video sequences. IEEE Transactions on Image
Processing, 5(6): 9961011, June 1996.
[16] H. Shen , L. Zhang , B. Huang , and P. Li . A map approach for joint motion estimation, segmentation, and super resolution. IEEE
Transactions on Image Processing, 16(2): 479490, February 2007.
[17] H. Takeda , P. Milanfar , M. Protter , and M. Elad . Super-resolution without explicit subpixel motion estimation. IEEE Transactions on
Image Processing, 18(9): 19581975, September 2009.
[18] K. Turkowski . Filters for common resampling tasks. In Graphical Gems, pages 147165. Academic Press Professional Inc., San Diego,
CA, USA, 1990.
121 [19] P. van de Walle , S. Susstrunk , and M. Vetterli . A frequency domain approach to registration of aliased images with application
to super-resolution. EURASIP Journal On Applied Signal Processing, 2006:114, March 2006.
[20] P.A. Viola and M.J. Jones . Rapid object detection using a boosted cascade of simple features. In IEEE Computer Society Conference
on Computer Vision and Pattern Recognition (CVPR), volume 1, pages I: 511518, December 2001.
[21] G. Wolberg . Digital Image Warping. IEEE Computer Society Press, Los Alamitos, CA, 1990.122

Spatially Adaptive Filtering as Regularization in Inverse Imaging: Compressive Sensing, Super-

Resolution, and Upsampling
[1] R.G. Baraniuk . Compressive sensing [lecture notes]. Signal Processing Magazine, IEEE, 24(4): 118121, July 2007.
[2] A. Buades , B. Coll , and J. M. Morel . A review of image denoising algorithms, with a new one. Multiscale Modeling and Simulation,
4(2): 490 530, 2005.
151 [3] A. Buades , B. Coll , J.M. Morel , and C. Sbert . Self-similarity driven color demosaicking. Image Processing, IEEE Transactions on,
18(6): 11921202, June 2009.
[4] A. Buades , B. Coll , and J.M. Morel . Nonlocal image and movie denoising. Int. J. Computer Vision, 76(2): 123139, February 2008.
[5] E. J. Candes and J. Romberg . Practical signal recovery from random projections. In Wavelet Applications in Signal and Image
Processing XI, Proc. SPIE Conf., volume 5914, 2005.
[6] E. J. Candes and J. Romberg . Sparsity and incoherence in compressive sampling. Inverse Problems, 23:969985, 2007.
[7] E.J. Candes , J. Romberg , and T. Tao . Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency
information. Information Theory, IEEE Transactions on, 52(2): 489509, Feb. 2006.
[8] J.B. Conway . A course in functional analysis. Springer, 1990.
[9] K. Dabov , A. Foi , and K. Egiazarian . Video denoising by sparse 3D transform-domain collaborative filtering. In Proc. European Signal
Process. Conf., Poznan, Poland, Sep. 2007.
[10] K. Dabov , A. Foi , V. Katkovnik , and K. Egiazarian . Image denoising by sparse 3D transform-domain collaborative filtering. IEEE
Trans. Image Process., 16(8): 20802095, Aug. 2007.
[11] K. Dabov , A. Foi , V. Katkovnik , and K. Egiazarian . Joint image sharpening and denoising by 3d transform-domain collaborative
filtering. In Proc. 2007 Int. TICSP Workshop Spectral Meth. Multirate Signal Process., Moscow, Russia, Sep. 2007.
[12] A. Danielyan , A. Foi , V. Katkovnik , and K. Egiazarian . Image and video super-resolution via spatially adaptive block-matching
filtering. In Proc. Int. Workshop Local Non-Local Approx. Image Process., Lausanne, Switzerland, Aug. 2008.
[13] A. Danielyan , A. Foi , V. Katkovnik , and K. Egiazarian . Image upsampling via spatially adaptive block-matching filtering. In Proc.
European Signal Process. Conf., Lausanne, Switzerland, Aug. 2008.
[14] D. L. Donoho and M. Elad . Maximal sparsity representation via l1 minimization. Proc. Nat. Aca. Sci., 100:21972202, 2003.
[15] D.L. Donoho . Compressed sensing. Information Theory, IEEE Transactions on, 52(4): 12891306, Apr. 2006.
[16] M. Ebrahimi and E. R. Vrscay . Multi-frame super-resolution with no explicit motion estimation. In Proc. Computer Vision and Pattern
Recognition, pages 455459, Las Vegas, Nevada, USA, July 2008.
152 [17] K. Egiazarian , A. Foi , and V. Katkovnik . Compressed sensing image reconstruction via recursive spatially adaptive filtering. In
Proc. IEEE Int. Conf. on Image Process., ICIP 2007, pages 549552, Sept. 2007.
[18] S. Farsiu , D. Robinson , M. Elad , and P. Milanfar . Advances and challenges in super-resolution. International Journal of Imaging
Systems and Technology, 14:4757, 2004.
[19] R. Gribonval and M. Nielsen . Sparse representation in unions of bases. Information Theory, IEEE Transactions on, 49(12): 33203325,
2003.
[20] V. Katkovnik , A. Foi , K. Egiazarian , and J. Astola . From local kernel to nonlocal multiple-model image denoising. International
Journal of Computer Vision, 86(1): 132, Jan. 2010.
[21] V. Katkovnik and O. Yu . Kulchitskii. Parametric statistical estimates for multiextremum optimization problems. Engineering
Cybernetics, (6), 1975.
[22] C. Kervrann and J. Boulanger . Local adaptivity to variable smoothness for exemplar-based image denoising and representation.
Technical Report 5624, INRIA, July 2005.
[23] C. Kervrann and J. Boulanger . Unsupervised patch-based image regularization and representation. In Proc. Eur. Conf. Computer
Vision (ECCV06), volume 4, pages 555567, 2006.
[24] C. Kervrann and J. Boulanger . Local adaptivity to variable smoothness for exemplar-based image regularization and representation.
Int. J. Comput. Vision, 79(1): 4569, 2008.
[25] H. Kushner and G. G. Yin . Stochastic approximation and recursive algorithms and applications, 2nd Edn. Springer, 2003.
[26] S. Lansel . DenoiseLab. http://www.stanford.edu/slansel/DenoiseLab.
[27] T. Olson . Stabilized inversion for limited angle tomography. Engineering in Medicine and Biology Magazine, IEEE, 14(5): 612620,
Sep/Oct 1995.
[28] M. Protter and M. Elad . Super resolution with probabilistic motion estimation. Image Processing, IEEE Transactions on, 18(8):
18991904, Aug. 2009.
[29] M. Protter , M. Elad , H. Takeda , and P. Milanfar . Generalizing the nonlocal-means to super-resolution reconstruction. Image
Processing, IEEE Transactions on, 18(1): 3651, Jan. 2009.
[30] H. Takeda , P. Milanfar , M. Protter , and M. Elad . Super-resolution without explicit subpixel motion estimation. Image Processing,
IEEE Transactions on, 18(9): 19581975, Sept. 2009.
153 [31] J.A. Tropp . Just relax: convex programming methods for identifying sparse signals in noise. Information Theory, IEEE
Transactions on, 52(3): 10301051, March 2006.
[32] J.A. Tropp and A.C. Gilbert . Signal recovery from randommeasurements via orthogonal matching pursuit. Information Theory, IEEE
Transactions on, 53(12): 46554666, Dec. 2007.
[33] Y. Tsaig and D. L. Donoho . Extensions of compressed sensing. Signal Process., 86(3): 549571, March 2006.
[34] M.J. Wainwright . Sharp thresholds for high-dimensional and noisy sparsity recovery using ell 1-constrained quadratic programming
(lasso). Information Theory, IEEE Transactions on, 55(5): 21832202, May 2009.154

Registration for Super-Resolution: Theory, Algorithms, and Applications in Image and Mobile Video
Enhancement
[1] C. B. Barber , D. P. Dobkin , and H. Huhdanpaa . The Quickhull algorithm for convex hulls. ACM Transactions on Mathematical
Software (TOMS), 22(4): 469483, December 1996.
[2] S. Borman and R. Stevenson . Spatial resolution enhancement of low-resolution image sequences - a comprehensive review with
directions for future research. Technical report, University of Notre Dame, 1998.
[3]The Computer Journal, Special Issue on Super-Resolution, 2009.
[4]EURASIP Journal on Applied Signal Processing, Special Issue on Super-Resolution, 2006.
[5] S. Farsiu , M. Elad , and P. Milanfar . Multiframe demosaicing and super-resolution of color images. IEEE Transactions on Image
Processing, 15(1): 141159, January 2006.
[6]D. A. Forsyth and J. Ponce . Computer Vision: A Modern Approach. Prentice Hall, August 2002.
[7] G. Golub and V. Pereyra . Separable nonlinear least squares: the variable projection method and its applications. Inverse Problems,
19(2): R1R26, 2003.
[8] E. Hecht . Optics. Pearson - Addison Wesley, 2002.
[9] P. J. Huber . Robust Statistics. John Wiley and Sons, 1981.
[10]IEEE Signal Processing Magazine, Special Issue on Super-Resolution, May 2003.
[11]International Organization for Standardization. ISO 12233:2000 - Photography - Electronic still picture cameras - Resolution
measurements, 2000.
[12]Mathworks (The). MATLAB function reference: griddata, 2009.
[13] A. Papoulis . Generalized sampling expansion. IEEE Transactions on Circuits and Systems, 24(11): 652654, November 1977.
185 [14] S. C. Park , M. K. Park , and M. G. Kang . Super-resolution image reconstruction: a technical overview. IEEE Signal Processing
Magazine, 20(3): 2136, May 2003.
[15] D. Robinson , S. Farsiu , and P. Milanfar . Optimal registration of aliased images using variable projection with applications to super-
resolution. The Computer Journal, 52(1): 3142, 2009.
[16] P. J. Rousseeuw and A. M. Leroy . Robust Regression and Outlier Detection. Wiley, 1986, 2003.
[17] H. Sawhney and S. Ayer . Compact representations of videos through dominant and multiple motion estimation. IEEE Transactions on
Pattern Analysis and Machine Intelligence, 18:814830, 1996.
[18] R. Y. Tsai and T. S. Huang . Multiframe image restoration and registration. In T. S. Huang , editor, Advances in computer vision and
image processing, volume 1, pages 317339. JAI Press, 1984.
[19] P. Vandewalle . Super-Resolution from Unregistered Aliased Images. PhD thesis, Ecole Polytechnique Fdrale de Lausanne,
Switzerland, July 2006. No. 3591, [Reproducible] http://rr.epfl.ch/6.
[20] P. Vandewalle , L. Sbaiz , J. Vandewalle , and M. Vetterli . Super-Resolution from Unregistered and Totally Aliased Signals using
Subspace Methods. IEEE Transactions on Signal Processing, 55(7, Part 2):36873703, 2007. [Reproducible] http://rr.epfl.ch/4.
[21] P. Vandewalle , S. Ssstrunk , and M. Vetterli . A Frequency Domain Approach to Registration of Aliased Images with Application to
Super-Resolution. EURASIP Journal on Applied Signal Processing, Special Issue on Super-Resolution Imaging, 2006, 2006. Article ID
71459, 14 pages, [Reproducible] http://rr.epfl.ch/3.186

Towards Super-Resolution in the Presence of Spatially Varying Blur

[1] Amit Agrawal , Yi Xu , and Ramesh Raskar . Invertible motion blur in video. ACM Trans. Graph., 28(3):18, 2009.
[2] M. Ahmed and A. Farag . Non-metric calibration of camera lens distortion. In Image Processing, 2001. Proceedings. 2001 International
Conference on, volume 2, pages 157160 vol.2, Oct 2001.
[3] Mark R. Banham and Aggelos K. Katsaggelos . Digital image restoration. IEEE Signal Process. Mag., 14(2):2441, March 1997.
[4] Leah Bar , Nir A. Sochen , and Nahum Kiryati . Restoration of images with piecewise space-variant blur. In SSVM, pages 533544,
2007.
[5] Moshe Ben-Ezra , Zhouchen Lin , and Bennett Wilburn . Penrose pixels super-resolution in the detector layout domain. In Proc. IEEE
Int. Conf. Computer Vision, pages 18, 2007.
[6] M. Born and E. Wolf . Principles of Optics. Pergamon Press, Oxford, 1980.
[7] Dmitry Datsenko and Michael Elad . Example-based single document image super-resolution: a global map approach with outlier
rejection. Multidimensional Syst. Signal Process., 18(2-3):103121, 2007.
[8] Paolo Favaro , Martin Burger , and Stefano Soatto . Scene and motion reconstruction from defocus and motion-blurred images via
anisothropic diffusion. In Tom Pajdla and Ji Matas , editors, ECCV 2004, LNCS 3021, Springer Verlag, Berlin Heidelberg, pages 257269,
2004.
[9] Paolo Favaro and Stefano Soatto . A variational approach to scene reconstruction and image segmentation from motion-blur cues. In
Proc. IEEE Conf. Computer Vision and Pattern Recognition, volume 1, pages 631637, 2004.
[10] Rob Fergus , Barun Singh , Aaron Hertzmann , Sam T. Roweis , and William T. Freeman . Removing camera shake from a single
photograph. ACM Trans. Graph., 25(3):787794, 2006.
[11] William T. Freeman , Thouis R. Jones , and Egon C Pasztor . Example-based super-resolution. IEEE Comput. Graph. Appl.,
22(2):5665, 2002.
216[12] D. J. Heeger and A. D. Jepson . Subspace methods for recovering rigid motion. International Journal of Computer Vision,
7(2):95117, 1992.
[13] Kwang In Kim and Younghee Kwon . Example-based learning for single-image super-resolution. In Proceedings of the 30th DAGM
symposium on Pattern Recognition, pages 456465, Berlin, Heidelberg, 2008. Springer-Verlag.
[14] A. Levin . Blind motion deblurring using image statistics. In NIPS, pages 841848, 2006.
[15] Anat Levin , Robert Fergus , Frdo Durand , and William T. Freeman . Image and depth from a conventional camera with a coded
aperture. ACM Trans. Graph., 26(3):70, 2007.
[16] Anat Levin , Peter Sand , Taeg Sang Cho , Frdo Durand , and William T. Freeman . Motion-invariant photography. In SIGGRAPH 08:
ACM SIGGRAPH 2008 papers, pages 19, New York, NY, USA, 2008. ACM.
[17] S. H. Lim and D. A. Silverstein . Method for deblurring an image. US Patent Application, Pub. No. US2006/0187308 A1, Aug 24 2006.
[18] James G. Nagy and Dianne P. OLeary . Restoring images degraded by spatially variant blur. SIAM J. Sci. Comput., 19(4):10631082,
1998.
[19] T. Q. Pham . Spatiotonal adaptivity in super-resolution of under-sampled image sequences. PhD thesis, TU Delft, 2006.
[20] A. N. Rajagopalan and S. Chaudhuri . An MRF model-based approach to simultaneous recovery of depth and restoration from
defocused images. IEEE Trans. Pattern Anal. Mach. Intell., 21(7), July 1999.
[21] L. I. Rudin , S. Osher , and E. Fatemi . Nonlinear total variation based noise removal algorithms. Physica D, 60:259268, 1992.
[22] Huanfeng Shen , Liangpei Zhang , Bo Huang , and Pingxiang Li . A MAP approach for joint motion estimation, segmentation, and
super resolution. IEEE Trans. Image Process., 16(2):479490, February 2007.
[23] F. roubek , G. Cristobal , and J. Flusser . A Unified Approach to Super-resolution and Multichannel Blind Deconvolution. IEEE
Transactions on Image Processing, 16:23222332, September 2007.
[24] Filip roubek and Jan Flusser . Multichannel blind iterative image restoration. IEEE Trans. Image Process., 12(9):10941106, September
2003.
[25] Filip roubek and Jan Flusser . Multichannel blind deconvolution of spatially misaligned images. IEEE Trans. Image Process.,
14(7):874883, July 2005.
217[26] Muralidhara Subbarao and G. Surya . Depth from defocus: a spatial domain approach. International Journal of Computer Vision,
3(13):271294, 1994.
[27] M. Tico , M. Trimeche , and M. Vehvilainen . Motion blur identification based on differently exposed images. In Proc. IEEE Int. Conf.
Image Processing, pages 20212024, 2006.
[28] D. Tschumperl and R. Deriche . Diffusion PDEs on vector-valued images. IEEE Signal Process. Mag., 19(5):1625, September 2002.
[29] David Tschumperl and Rachid Deriche . Vector-valued image regularization with PDEs: A common framework for different
applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(4):506517, 2005.
[30] Damon L. Tull and Aggelos K. Katsaggelos . Regularized blur-assisted displacement field estimation. In Proc. Int. Conf. Image
Processing, volume 3, pages 8588, September 1996.
[31] Michal orel . Multichannel Blind Restoration of Images with Space-Variant Degradations. PhD thesis, Charles University in Prague,
2007.
[32] Michal orel and Jan Flusser . Space-variant restoration of images degraded by camera motion blur. IEEE Trans. Image Process.,
17(2):105116, February 2008.
[33] Michal orel and Filip roubek . Space-variant deblurring using one blurred and one underexposed image. In Proc. Int. Conf. Image
Processing, 2009.
[34] Filip roubek , Jan Flusser , and Michal orel . Superresolution and blind deconvolution of video. In Proc. Int. Conf. on Pattern
Recognition, pages 14, 2008.
[35] Masahiro Watanabe and Shree K. Nayar . Rational filters for passive depth from defocus. International Journal of Computer Vision,
3(27):203225, 1998.
[36] Wonpil Yu . An embedded camera lens distortion correction method for mobile computing applications. Consumer Electronics, IEEE
Transactions on, 49(4):894901, Nov. 2003.
[37] Lu Yuan , Jian Sun , Long Quan , and Heung-Yeung Shum . Image de-blurring with blurred/noisy image pairs. In SIGGRAPH 07: ACM
SIGGRAPH 2007 papers, page 1, New York, NY, USA, 2007. ACM.
[38] Barbara Zitov and Jan Flusser . Image registration methods: a survey. Image and Vision Computing, 11(21):9771000, 2003.218

[1]Generic coding of moving pictures and associated audio: Video. ISO/IEC, pages 138182, 1996.
[2]Coding of audio-visual objects: Visual. ISO/IEC, pages 144962, 1998.
245 [3] S. Baker and I. Matthews . Lucas-Kanade 20 years on: a unifying frame-work. International Journal of Computer Vision, 56(3):
221255, 2004.
[4] M.J. Black and P. Anandan . The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Computer Vision
and Image Understanding, 63(1): 75104, 1996.
[5] D. Capel . Image mosaicing and super-resolution. Springer-Verlag New York Inc, 2004.
[6] S. Chang , M. Shimizu , and M. Okutomi . Multi-frame super-resolution with multiple motion regions. Korean Japan Joint Workshop on
Pattern Recognition (KJPR), 107(281): 5762, 2007.
[7] O. Choi , H. Kim , and I.S. Kweon . Simultaneous plane extraction and 2D homography estimation using local feature transformations.
Asian Conference on Computer Vision (ACCV), 4844:269278, 207.
[8] O. Chum and J. Matas . Matching with PROSAC-progressive sample consensus. IEEE Computer Society Conference on Computer
Vision and Pattern Recognition (CVPR), 1:220226, 2005.
[9] F. Durand and J. Dorsey . Fast bilateral filtering for the display of high-dynamic-range images. Proc. the Annual Conference on
Computer Graphics and Interactive Techniques, pages 257266, 2002.
[10] N.A. El-Yamany and P.E. Papamichalis . Robust color image super-resolution: an adaptive M-estimation framework. Journal on Image
and Video Processing, 8(2), 2008.
[11] S. Farsiu , M. Elad , P. Milanfar , et al. Multiframe demosaicing and super-resolution of color images. IEEE Transactions on Image
Processing, 15(1): 141159, 2006.
[12] S. Farsiu , D. Robinson , M. Elad , and P. Milanfar . Advances and challenges in super-resolution. International Journal of Imaging
Systems and Technology, 14(2): 4757, 2004.
[13] S. Farsiu , M.D. Robinson , M. Elad , and P. Milanfar . Fast and robust multiframe super resolution. IEEE Transactions on Image
Processing, 13(10): 13271344, 2004.
[14] M.A. Fischler and R.C. Bolles . Random sample consensus: a paradigm for model fitting with applications to image analysis and
automated cartography. Communications of the ACM, 24(6): 381395, 1981.
[15] T. Gotoh and M. Okutomi . Direct super-resolution and registration using raw CFA images. In IEEE Computer Society Conference on
Computer Vision and Pattern Recognition (CVPR), volume 2. IEEE Computer Society; 1999, 2004.
246 [16] H. He and L.P. Kondi . An image super-resolution algorithm for different error levels per frame. IEEE Transactions on image
processing, 15(3): 592603, 2006.
[17] Z.A. Ivanovski , L. Panovski , and L.J. Karam . Robust super-resolution based on pixel-level selectivity. In Proceedings of SPIE,
volume 6077, 2006.
[18] E.S. Lee and M.G. Kang . Regularized adaptive high-resolution image reconstruction considering inaccurate subpixel registration.
IEEE Transactions on Image Processing, 12(7): 826837, 2003.
[19] D.G. Lowe . Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2): 91110, 2004.
[20] S.C. Park , M.K. Park , and M.G. Kang . Super-resolution image reconstruction: a technical overview. IEEE Signal Processing
Magazine, 20(3): 2136, 2003.
[21] V. Patanavijit and S. Jitapunkul . A Lorentzian stochastic estimation for a robust iterative multiframe superresolution reconstruction
with Lorentzian-Tikhonov regularization. EURASIP Journal on Advances in Signal Processing, 2007(2): 2121, 2007.
[22] R.R. Schultz and R.L. Stevenson . Extraction of high-resolution frames from video sequences. IEEE Transactions on Image
Processing, 5(6): 9961011, 1996.
[23] M. Shimizu and M. Okutomi . Subpixel estimation error cancellation on area-based matching. International Journal of Computer
Vision, 63(3): 207224, 2005.
[24] C. Sun . Fast algorithms for stereo matching and motion estimation. Proc. of Australia-Japan Advanced Workshop on Computer
Vision, pages 3848, 2003.
[25] M. Tanaka and M. Okutomi . A fast MAP-based super-resolution algorithm for general motion. Electronic Imaging Computational
Imaging IV, 6065:112, 2006.
[26] W.Y. Zhao and H.S. Sawhney . Is super-resolution with optical flow feasible? Lecture Notes in Computer Science, pages 599613,
2002.
[27] B. Zitov and J. Flusser . Image registration methods: a survey. Image and Vision Computing, 21(11): 9771000, 2003.
[28] A. Zomet , A. Rav-Acha , and S. Peleg . Robust super-resolution. In IEEE Computer Society Conference on Computer Vision and
Pattern Recognition (CVPR), volume 1, 2001.
Multiframe Super-Resolution from a Bayesian Perspective
[1] S. Baker and T. Kanade . Limits on super-resolution and how to break them. 24(9): 11671183, 2002.
[2] C. Bishop . Neural Networks for Pattern Recognition. Oxford University Press, 1995.
[3] M. J. Black and P. Anandan . A framework for the robust estimation of optical flow. pages 231236, 1993.
[4] D. P. Capel . Image Mosaicing and Super-resolution (Distinguished Dissertations). Springer, ISBN: 1852337710, 2004.
[5] S. Farsiu , M. Elad , and P. Milanfar . A practical approach to super-resolution. In Proc. of the SPIE: Visual Communications and Image
Processing, San-Jose, 2006.
[6] S. Farsiu , M. D. Robinson , M. Elad , and P. Milanfar . Fast and robust multiframe super resolution. 13(10): 13271344, October 2004.
[7] R. C. Hardie , K. J. Barnard , and E. E. Armstrong . Joint MAP registration and high-resolution image estimation using a sequence of
undersampled images. 6(12): 16211633, 1997.
[8] R. I. Hartley and A. Zisserman . Multiple View Geometry in Computer Vision. Cambridge University Press, ISBN: 0521540518, 2nd
Edn., 2004.
[9] P. J. Huber . Robust Statistics. John Wiley and Sons, 1981.
[10] M. Irani and S. Peleg . Improving resolution by image registration. Graphical Models and Image Processing, 53:231239, 1991.
[11] D. Kundur and D. Hatzinakos . Blind image deconvolution. IEEE Signal Processing Magazine, 13(3): 4346, May 1996.
[12] I. Nabney . Netlab Algorithms for Pattern Recognition. Springer, 2002.
[13] N. Nguyen , P. Milanfar , and G. Golub . Efficient generalized cross-validation with applications to parametric image restoration and
resolution enhancement. 10(9): 12991308, September 2001.
[14] L. C. Pickup . Machine Learning in Multi-frame Image Super-resolution. PhD thesis, University of Oxford, February 2008.
[15] M. E. Tipping and C. M. Bishop . Bayesian image super-resolution. In S. Thrun , S. Becker , and K. Obermayer , editors, Advances in
Neural Information Processing Systems, volume 15, pages 12791286, Cambridge, MA, 2003. MIT Press.
284 [16] W. Triggs , P. McLauchlan , R. Hartley , and A. Fitzgibbon . Bundle adjustment: A modern synthesis. In W. Triggs , A. Zisserman ,
and R. Szeliski , editors, Vision Algorithms: Theory and Practice, LNCS, pages 298375. Springer Verlag, 2000.

Variational Bayesian Super-Resolution Reconstruction

[1] MDSP super-resolution and demosaicing datasets . http://www.soe.ucsc.edu/milanfar/software/sr-datasets.html, 2007.
[2] S. D. Babacan , R. Molina , and A. K. Katsaggelos . Generalized Gaussian Markov field image restoration using variational distribution
approximation. In IEEE International Conf. on Acoustics, Speech, and Signal Processing (ICASSP08), Las Vegas, Nevada, February
2008.
[3] S. D. Babacan , R. Molina , and A. K. Katsaggelos . Total variation super resolution using a variational approach. In IEEE International
Conf. on Image Processing 2008, October 2008.
[4] S. D. Babacan , R. Molina , and A.K. Katsaggelos . Variational Bayesian super resolution. Submitted to IEEE Trans. on Image
Processing, 2009.
[5] S.D. Babacan , R. Molina , and A.K. Katsaggelos . Variational Bayesian blind deconvolution using a total variation prior. IEEE
Transactions on Image Processing, 18:1226, January 2009.
310 [6] S.D. Babacan , J. Wang , R. Molina , and A.K. Katsaggelos . Bayesian blind deconvolution from differently exposed image pairs. In
IEEE International Conference on Image Processing, Cairo, Egypt, July 2009.
[7] M.J. Beal . Variational Algorithms for Approximate Bayesian Inference. PhD thesis, The Gatsby Computational Neuroscience Unit,
University College London, 2003.
[8] J. O. Berger . Statistical Decision Theory and Bayesian Analysis, chapter 3 and 4. New York, Springer Verlag, 1985.
[9] C.M. Bishop . Pattern Recognition and Machine Learning. Springer, 2006.
[10] T. E. Bishop , S. D. Babacan , B. Amizic , A. K. Katsaggelos , T. Chan , and R. Molina . Blind image deconvolution: problem
formulation and existing approaches. In P. Campisi and K. Egiazarian , editors, Blind image deconvolution: theory and applications,
chapter 1. Boca Raton, FL: CRC Press, 2007.
[11] T. F. Chan , M. K. Ng , A. C. Yau , and A. M. Yip . Superresolution image reconstruction using fast inpainting algorithms. Applied and
Computational Harmonic Analysis, 23(1):324, 2007. Special Issue on Mathematical Imaging.
[12] T. F. Chan , N. Ng , A. Yau , and A. Yip . Superresolution image reconstruction using fast inpainting algorithms. Applied and
Computational Harmonic Analysis, 23(1):324, July 2007.
[13] T. F. Chan and C.K. Wong . Total variation blind deconvolution. IEEE Transactions on Image Processing, 7(3):370375, Mar. 1998.
[14] A. D. Dempster , N. M. Laird , and D. B. Rubin . Maximum likelihood from incomplete data via the E-M algorithm. Journal of the Royal
Statistical Society, Series B, 39:137, 1977.
[15] M. Elad and A. Feuer . Restoration of a single superresolution image from several blurred, noisy, and undersampled measured
images. IEEE Transactions on Image Processing, 6(12):16461658, Dec 1997.
[16] S. Farsiu . MDSP resolution enhancement software. University of California at Santa Cruz, 2004.
[17] S. Farsiu , M. D. Robinson , M. Elad , and P. Milanfar . Fast and robust multiframe super resolution. IEEE Transactions on Image
Processing, 13(10):13271344, Oct. 2004.
[18] R. Fergus , B. Singh , A. Hertzmann , S. T. Roweis , and W.T. Freeman . Removing camera shake from a single photograph. ACM
Transactions on Graphics, SIGGRAPH 2006 Conference Proceedings, Boston, MA, 25:787794, 2006.
311 [19] M.A.T. Figueiredo , J.M. Bioucas-Dias , and R.D. Nowak . Majorizationminimization algorithms for wavelet-based image
restoration. IEEE Transactions on Image Processing, 16(12):29802991, Dec. 2007.
[20] R.C. Hardie , K.J. Barnard , and E.E. Armstrong . Joint MAP registration and high-resolution image estimation using a sequence of
undersampled images. IEEE Transactions on Image Processing, 6(12):16211633, 1997.
[21] H. He and L. P. Kondi . An image super-resolution algorithm for different error levels per frame. IEEE Transactions on Image
Processing, 15(3):592603, March 2006.
[22] Y. He , K. H. Yap , L. Chen , and L. P. Chau . A nonlinear least square technique for simultaneous image registration and super-
resolution. IEEE Transactions on Image Processing, (11):28302841, 2007.
[23] M. Irani and S. Peleg . Motion analysis for image enhancement: Resolution, occlusion, and transparency. Journal of Visual
Communication and Image Representation, 4(4):324335, 1993.
[24] A. Kanemura , S.I. Maeda , and S. Ishii . Superresolution with compound Markov random fields via the variational EM algorithm.
Neural Networks, 22(7):10251034, 2009.
[25] M.G. Kang and S. Chaudhuri (Eds.). Super-resolution image reconstruction. IEEE Signal Processing Magazine, 20(3), 2003.
[26] A. K. Katsaggelos , R. Molina , and J. Mateos . Super Resolution of Images and Video. Morgan and Claypool, 2007.
[27] H. Kim , J.H. Jang , and K.S. Hong . Edge-enhancing super-resolution using anisotropic diffusion. In Proceedings of the IEEE
Conference on Image Processing, volume 3, pages 130133, 2001.
[28] E. S. Lee and M. G. Kang . Regularized adaptive high-resolution image reconstruction considering inaccurate subpixel registration.
IEEE Transactions on Image Processing, 12(7):826837, July 2003.
[29] B.D. Lucas and T. Kanade . An iterative image registration technique with an application to stereo vision. In Proceedings of Imaging
Understanding Workshop, pages 121130, 1981.
[30] J. Miskin . Ensemble Learning for Independent Component Analysis. PhD thesis, Astrophysics Group, University of Cambridge, 2000.
[31] R. Molina , A. K. Katsaggelos , J. Abad , and J. Mateos . A Bayesian approach to blind deconvolution based on Dirichlet distributions.
In 1997 International Conference on Acoustics, Speech and Signal Processing (ICASSP97), volume 4, pages 28092812, Munich
(Germany), 1997.
312 [32] R. Molina , A. K. Katsaggelos , and J. Mateos . Bayesian and regularization methods for hyperparameter estimation in image
restoration. IEEE Transactions on Image Processing, 8(2):231246, 1999.
[33] R. Molina , J. Mateos , and A.K. Katsaggelos . Blind deconvolution using a variational approach to parameter, image, and blur
estimation. IEEE Transactions on Image Processing, November 2006.
[34] M. Ng , T. Chan , M.G. Kang , and P. Milanfar . Super-resolution imaging: Analysis, algorithms, and applications. EURASIP Journal on
Applied Signal Processing, 2006:Article ID 90531, 2 pages, 2006.
[35] M. Ng , J. Koo , and N. Bose . Constrained total least-squares computations for high-resolution image reconstruction with
multisensors. International Journal of Imaging Systems and Technology, 12(1):3542, 2002.
[36] M. K. Ng , H. Shen , E. Y. Lam , and L. Zhang . A total variation regularization based super-resolution reconstruction algorithm for
digital video. EURASIP Journal on Advances in Signal Processing, (74585), 2007.
[37] J. Palmer , D. Wipf , K. Kreutz-Delgado , and B. Rao . Variational EM algorithms for non-Gaussian latent variable models. In Y. Weiss
, B. Schlkopf , and J. Platt , editors, Advances in Neural Information Processing Systems 18, pages 10591066. MIT Press, Cambridge,
MA, 2006.
[38] L. C. Pickup , D. P. Capel , S. J. Roberts , and A. Zisserman . Bayesian methods for image super-resolution. The Computer Journal,
52:101113, 2009.
[39] M. Protter , M. Elad , H. Takeda , and P. Milanfar . Generalizing the nonlocal-means to super-resolution reconstruction. IEEE
Transactions on Image Processing, 18(1):3651, Jan. 2009.
[40] D. Rajan and S. Chaudhuri . Generation of super-resolution images from blurred observations using an MRF model. Journal of
Mathematical Imaging and Vision, 16:515, 2002.
[41] L. I. Rudin , S. Osher , and E. Fatemi . Nonlinear total variation based noise removal algorithms. Physica D, pages 259268, 1992.
[42] C. A. Segall , R. Molina , A. K. Katsaggelos , and J. Mateos . Bayesian resolution enhancement of compressed video. IEEE
Transactions on Image Processing, 13(7):898911, 2004.
[43] C.A. Segall , R. Molina , and A.K. Katsaggelos . High-resolution images from low-resolution compressed video. IEEE Signal
Processing Magazine, 20:3748, 2003.
313 [44] F. roubek and J. Flusser . Multichannel blind deconvolution of spatially misaligned images. IEEE Transactions on Image
Processing, 7:4553, July 2005.
[45] H. Stark and P. Oskoui . High resolution image recovery from image-plane arrays, using convex projections. Journal of the Optical
Society of America A, 6:17151726, 1989.
[46] H. Takeda , P. Milanfar , M. Protter , and M. Elad . Super-resolution without explicit subpixel motion estimation. IEEE Transactions on
Image Processing, 18(9):19581975, Sept. 2009.
[47] A.M. Tekalp , M.K. Ozkan , and M.I. Sezan . High-resolution image reconstruction from lower-resolution image sequences and space
varying image restoration. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, volume 3,
pages 169172, 1992.
[48] M. E. Tipping and C. M. Bishop . Bayesian image super-resolution. In Advances in Neural Information Processing Systems 15 (NIPS).
MIT Press, 2003.
[49] P. Vandewalle , L. Sbaiz , J. Vandewalle , and M. Vetterli . Super-resolution from unregistered and totally aliased signals using
subspace methods. IEEE Transactions on Signal Processing, 55(7, Part 2):36873703, 2007.
[50] P. Vandewalle , S. Ssstrunk , and M. Vetterli . A frequency domain approach to registration of aliased images with application to
super-resolution. EURASIP Journal on Applied Signal Processing (special issue on Super-resolution), 2006:Article ID 71459, 14 pages,
2006.
[51] P. Vandewalle , P. Zbinden , and C. Perez . Superresolution v2.0. http://lcavwww.epfl.ch/software/superresolution/index.html, 2006.
[52] N.A. Woods , N.P. Galatsanos , and A.K. Katsaggelos . Stochastic methods for joint registration, restoration, and interpolation of
multiple undersampled images. IEEE Transactions on Image Processing, 15:201213, 2006.
[53] A. Zomet , A. Rav-Acha , and S. Peleg . Robust super-resolution. In IEEE Computer Society Conference on Computer Vision and
Pattern Recognition (CVPR 2001), pages 645650, 2001.

Pattern Recognition Techniques for Image Super-Resolution

[1] F. Ahmed , S. Gustafson , and M. Karim . High-fidelity image interpolation using radial basis function neural networks. Proceedings of
IEEE National Aerospace and Electronics Conference, 2:588592, 1995.
350 [2] A. Atiya . Estimating the posterior probabilities using the k-nearest neighbor rule. Neural Comp., 17(3):731740, 2005.
[3] C. Atkins and C. Bouman . Classification based methods in optimal image interpolation. PhD thesis, Purdue University, 1998.
[4] C. Atkins , C. Bouman , and J. Allebach . Tree-based resolution synthesis. In Proceedings of the Image Processing, Image Quality,
Image Capture Systems Conference, pages 405410, 1999.
[5] R. Avnimelech and N. Intrator . Boosted mixture of experts: An ensemble learning scheme. Neural Computation, 11:483497, 1999.
[6] F. R. Bach and M. I. Jordan . Predictive low-rank decomposition for kernel methods. In ICML 05: Proceedings of the 22nd international
conference on Machine learning, pages 3340, 2005.
[7] D. M. Bates and D. G. Watts . Nonlinear Regression Analysis and Its Applications. John Wiley and Sons, Inc., New York, 1988.
[8] S. Boyd and L. Vandenberghe . Convex Optimization. Cambridge University Press, New York, NY, 2004.
[9] D.C. Burr , M.C. Morrone , and D. Spinelli . Evidence for edge and bar detectors in human vision. Vision Research, 4:419431, 1989.
[10] E. Byvatov and G. Schneider . Support vector machine applications in bioinformatics. Applied Bioinformatics, 2(2):6777, 2003.
[11] F. M. Candocia and J. C. Principe . Super-resolution of images based on local correlations. IEEE Transactions on Neural Networks,
10(2):372, March 1999.
[12] H. Chang , D. Yeung , and Y. Xiong . Super-resolution through neighbor embedding. IEEE Conference on Computer Vision and
Pattern Recognition, 1:275282, 2004.
[13] S. Chen , K. Jeong , and W. Hardle . Recurrent support vector regression for a nonlinear ARMA model with applications to forecasting
financial returns. SFB 649 Discussion Papers SFB649DP2008-051, Sonderforschungsbereich 649, Humboldt University, Berlin, Germany,
July 2008.
[14] T. Cover . Estimation by the nearest neighbor rule. IEEE Transactions on Information Theory, IT-14(1), January 1968.
[15] N. Cristianini and J. Shawe-Taylor . An Introduction to Support VectorMachines and Other Kernel-Based Learning Methods.
Cambridge University Press, 1st edition, March 2000.
351 [16] C. A. Dvila and B. R. Hunt . Training of a neural network for image super-resolution based on a nonlinear interpolative vector
quantizer. Applied Optics, 39:34733485, July 2000.
[17] A. P. Dempster , N. M. Laird , and D. B. Rubin . Maximum likelihood from incomplete data via the EM algorithm. J. Royal Statistics
Society, 39:138, 1977.
[18] L. Devroye , L. Gyorfi , A. Krzyzak , and G. Lugosi . On the strong universal consistency of nearest neighbor regression function
estimates. The Annals of Statistics, 22(3):13711385, 1994.
[19] L. Devroye and T. Wagner . Distribution-free consistency results in non-parametric discrimination and regression function estimation.
The Annals of Statistics, 8(2):231239, 1980.
[20] H. Drucker , C. Burges , L. Kaufman , A. J. Smola , and V. Vapnik . Support vector regression machines. In NIPS, pages 155161,
1996.
[21] R. Duda , H. Hart , and D. Stork . Pattern Classification. Wiley Interscience, 2nd edition, 2000.
[22] N. Emelyanov and R. Gilbert . Astrometric results of observations of mutual occultations and eclipses of the Galilean satellites of
Jupiter in 2003. Astronomy and Astrophysics, 453:11411149, 2006.
[23] L. Fausett , editor. Fundamentals of neural networks: Architectures, algorithms, and applications. Prentice-Hall, Inc., Upper Saddle
River, NJ, USA, 1994.
[24] M. A. Fischler and R. Bolles . Random sample consensus: A paradigm for model fitting with applications to image analysis and
automated cartography. Communications of the ACM, 24(6):381395, 1981.
[25] W. Freeman , T. Jones , and E. Pasztor . Example-based super-resolution. IEEE Computer Graphics and Applications, 22(2):5665,
2002.
[26] W.T. Freeman , E.C. Pasztor , and O.T. Carmichael . Learning low-level vision. International Journal of Computer Vision, 40(1):2547,
2000.
[27] Y. Freund and R. Schapire . A Short Introduction to Boosting. Journal of Japanese Society for Artificial Intelligence, 14(5):771780,
1999.
[28] S. Geman and D. Geman . Stochastic relaxation, Gibbs distribution, and the bayesian restoration of images. IEEE Trans. Pattern
Analysis and Machine Intelligence, 6(4):721741, November 1984.
[29] P. M. Goebel and A. N. Belbachir . Single image superresolution interpolation by wavelet support vector regression. Wavelets and
Applications Semester and Conference, 2006.
352 [30] J. Hartigan . Clustering Algorithms. John Wiley and Sons, Inc., 1975.
[31] S. Haykin . Neural Networks: A Comprehensive Foundation. Macmillan, New York, 1994.
[32] R. Hecht-Nielsen . Theory of the backpropagation neural network. Neural networks for perception (Vol. 2): Computation, learning,
architectures, pages 6593, 1992.
[33] S. Hong , R. Park , S. Yang , and J. Kim . Image interpolation using interpolative classified vector quantization. Image Vision Comput.,
26(2):228239, 2008.
[34] K. I. Kim , M. Franz , and B. Scholkopf . Kernel hebbian algorithm for single-frame super-resolution. Statistical Learning in Computer
Vision, pages 135149, 2004.
[35] B. Kullis , M. Sustik , and I. Dhillon . Learning low-rank kernel matrices. International Conference on Machine Learning, 2006.
[36] J. Kwok . Support vector mixture for classification and regression problems. In Proceedings of the Fourteenth International Conference
on Pattern Recognition, volume 1, pages 255258, Brisbane, Qld., Australia, 1998.
[37] G. Lanckriet , N. Cristianini , P. Bartlett , L. El Ghaoui , and M. Jordan . Learning the kernel matrix with semidefinite programming.
Journal of Machine Learning Research, 5:2772, 2004.
[38] A. B. Lee , K. Pedersen , and D. Mumford . The nonlinear statistics of high-contrast patches in natural images. Int. J. Comput. Vision,
54(1-3):83103, 2003.
[39] P. Lewis and J. Stevens . Nonlinear modeling of time series using multivariate adaptive regression splines (Mars). Journal of the
American Statistical Association, 86(416):864877, 1991.
[40] D. Li . Support vector regression based image denoising. Image Vision Comput., 27(6):623627, 2009.
[41]Clodoaldo A.M. Lima , Andre L.V. Coelho , and Fernando J. Von Zuben . Hybridizing mixtures of experts with support vector machines:
Investigation into nonlinear dynamic systems identification. Information Sciences, 177:20492074, 2007.
[42] H. Mansouri , R. Gilbert , T. Trafalis , L. Leslie , and M. Richman . Ocean surface wind vector forecasting using support vector
regression. In C. H. Dagli , A. L. Buczak , D. L. Enke , M. J. Embrechts , and O. Ersoy , editors, Smart Systems Engineering:
Computational Intelligence in Architecturing Complex Engineering Systems, volume 17 of Intelligent Engineering Systems through Artificial
Neural Networks, pages 333338, New York, NY, USA, 2007.
353 [43]C. A. Micchelli and M. Pontil . On learning vector-valued functions. Neural Computation, 17, 2005.
[44] C. Miravet and F. Rodriguez . A hybrid mlp-pnn architecture for fast image superresolution. Lecture Notes Computer Science,
2714:401, 2003.
[45] K. Ni and T. Nguyen . Image superresolution using support vector regression. IEEE Transactions on Image Processing,
16(6):15961610, 2007.
[46] E. Osuna , R. Freund , and F. Girosi . Training support vector machines: An application to face detection. pages 130136, 1997.
[47] N. Plaziac . Image interpolation using neural networks. IEEE Transactions on Image Processing, 8:16471651, November 1999.
[48] S. Qiu and T. Lane . Multiple kernel learning for support vector regression. Technical report, University of New Mexico, 2005.
[49] D. Rajan and S. Chaudhuri . An MRF-based approach to generation of super-resolution images from blurred observations. J. Math.
Imaging Vis., 16(1):515, 2002.
[50] A. Roorda . Human visual system - image formation. The Encyclopedia of Imaging Science and Technology, 1:539557, 2002.
[51] B. Scholkopf . Statistical learning and kernel methods. Technical report, Microsoft Research Limited, 2000.
[52] G. Seber and C. Wild . Nonlinear Regression. John Wiley and Sons, Inc., New York, 2003.
[53] G. Shakhnarovich , T. Darrell , and P. Indyk . Nearest-Neighbor Methods in Learning and Vision: Theory and Practice. The MIT Press,
2005.
[54] K. Su , Q. Tian , Q. Que , N. Sebe , and J. Ma . Neighborhood issue in single-frame image superresolution. pages 11221125,
Amsterdam, The Netherlands, 2005. IEEE Computer Society.
[55] S. Szedmak , J. Shawe-Taylor , and E. Parado-Hernandez . Learning via linear operators: Maximum margin regression. Technical
report, PASCAL, Southampton, UK, 2005.
[56] L. Tuncel . On the slater condition for the SDP relaxations of nonconvex sets. Operations Research Letters, 29:181186, 2001.
[57] V. Vapnik . The Nature of Statistical Learning Theory. Springer-Verlag, New York, 1995.
354 [58] V. Vapnik , S. Golowich , and A. Smola . Support vector method for function approximation, regression estimation and signal
processing. In NIPS, pages 281287, 1996.
[59] E. Vazquez and E. Walter . Multi-output support vector regression. 13th IFAC Symposium on System Identification, pages 18201825,
2003.
[60] K. Weinberger and L. Saul . Unsupervised learning of image manifolds by semidefinite programming. Int. J. Comput. Vision,
70(1):7790, 2006.
[61] K. Weinberger and L. Saul . Fast solvers and efficient implementations for distance metric learning. In ICML, pages 11601167, 2008.
[62] K. Weinberger , F. Sha , and L. Saul . Learning a kernel matrix for nonlinear dimensionality reduction. In ICML04: Proceedings of the
Twenty-First International Conference on Machine Learning. ACM, 2004.
[63] C. Yao , P. Yu , and R. Hung . Extractive support vector algorithm on support vector machines for image restoration. Fundam. Inf.,
90(1-2):171190, 2009.
[64] Z. Zhang , D. Yeung , and J. Kwok . Bayesian inference for transductive learning of kernel matrix using the Tanner-Wong data
augmentation algorithm. Proceedings of the 21st International Conference on Machine Learning, 2004.

Super-Resolution Reconstruction of Multichannel Images

[1] T. Akgun , Y. Altunbasak , and R. M. Mersereau . Super-resolution reconstruction of hyperspectral images. IEEE Transactions on
Image Processing, 14:18601875, 2005.
[2] Hardie R. C. and M. T. Eismann . MAP estimation for hyperspectral image resolution enhancement with an auxiliary sensor. IEEE
Trans. Image Proc., 13(13):11741184, 2004.
[3] Chein-I Chang . Hyperspectral Data Exploitation : Theory and Applications. John Wiley & Sons, 2007.
[4] AVIRIS Standard Data . Available: http://aviris.jpl.nasa.gov/. Nasa Jet Propulsion Laboratory, Caltech.
[5] P. E. Eren , Sezan M. I. , and A. M. Tekalp . Robust, object-based highresolution image reconstruction from low-resolution video. IEEE
Transactions on Image Processing, 6:14461452, October 1997.
[6] Y. Gu , Zhang Y. , and Zhang J. Integration of Spatial-Spectral Information for Resolution Enhancement in Hyperspectral Images. IEEE
Trans. Geosci. Remote Sens., 46(46):13471358, 2008.
[7] B. Gunturk , A. U. Batur , Y. Altunbasak , M. H. Hayes III , and R. M. Mersereau . Eigenface-domain super-resolution for face
recognition. IEEE Transactions on Image Processing, 12(12):597606, 2003.
[8] R. I. Hartley and A. Zisserman . Multiple View Geometry in Computer Vision. Cambridge University Press, ISBN: 0521540518, second
edition, 2004.
[9] A. Hyvrinen . Fast and robust fixed-point algorithms for independent component analysis. IEEE Transactions on Neural Networks, 10.
[10] D. Landgrebe . Hyperspectral image data analysis. IEEE Signal Processing Magazine, 19:1728, January 2002.
[11] ASTER Spectral Library . Available: http://speclib.jpl.nasa.gov/. Nasa Jet Propulsion Laboratory, Caltech.
[12] T.M. Lillesand , R.W. Kiefer , and J.W. Chipman . Remote Sensing and Image Interpretation. John Wiley & Sons, 6th edition, 2008.
[13] J. P. Nadal and N. Parga . Non-linear neurons in the low noise limit: A factorial code maximizes information transfer. Network,
5:565581, 1994.
381 [14] J. M. P. Nascimento and J. M. B. Dias . Does independent component analysis play a role in unmixing hyperspectral data? IEEE
Transactions on Geoscience and Remote Sensing, 43:175187, 2005.
[15] A. J. Patti and Y. Altunbasak . Artifact reduction for POCS-based super-resolution with edge adaptive regularization and higher-order
interpolants. IEEE Transactions on Image Processing, 10:176186, January 2001.
[16] A. J. Patti , Sezan M. I. , and A. M. Tekalp . Super-resolution video reconstruction with arbitrary sampling lattice and nonzero aperture
time. IEEE Transactions on Image Processing, 6:10641076, August 1997.
[17] T. Ranchin and L. Wald . Fusion of high spatial and spectral images: The ARSIS concept and its implementation. Photogramm Eng.
Remote Sens., 66:4956, 2000.
[18] G.D. Robinson , H.N. Gross , and J.R. Schott . Evaluation of two applications of spectral mixing models to image fusion. Proc. SPIE,
71:272281, 2000.
[19] M. I. Sezan . An overview of convex projections theory and its application to image recovery problem. Ultramicroscopy, 40:5567, 1992.
[20] O. G. Sezer , Y. Altunbasak , and A. Ercil . Face recognition with independent component based super-resolution. In Proceedings of
SPIE Visual Communications and Image Processing Conference, volume 6077, San Jose, CA, 2006.
[21] G. Shaw and D. Manolakis . Signal processing for hyperspectral image exploitation. IEEE Signal Processing Magazine, 19:1216,
January 2002.
[22] H. Stark and Yang Y. Vector Space Projections: A Numerical Approach to Signal and Image Processing. Neural Nets, and Optics,
Wiley series in Telecommunications and Signal Processing, 1998.
[23] H. J. Trussell and M. R. Civanlar . The feasible solution in signal processing. IEEE Trans. Acous, Speech, and Signal Process., ASSP-
32(32):201212, 1984.382

New Applications of Super-Resolution in Medical Imaging

[1] Y. Altunbasak , A. Patti , and R. Mersereau . Super-resolution still and video reconstruction from MPEG-coded video. IEEE Trans.
Circuits and Syst. Video Technol., 12(4):217226, Apr. 2002.
[2] L.D. Alvarez , J. Mateos , R. Molina , and A.K. Katsaggelos . High resolution images from compressed low resolution video: Motion
estimation and observable pixels. International Journal of Imaging Systems and Technology, 14(2):5866, October 2004.
[3] S. Borman and R.L. Stevenson . Super-resolution from image sequences - a review. In Proc. of the 1998 Midwest Symposium on
Circuits and Systems, volume 5, Apr. 1998.
[4] S.G. Chang , B. Yu , and M. Vetterli . Adaptive wavelet thresholding for image denoising and compression. IEEE Transactions on
Image Processing, 9(9):15321546, 2000.
[5] S.H. Chavala , S. Farsiu , R. Maldonado , D.K. Wallace , S.F. Freedman , and C.A. Toth . Insights into advanced retinopathy of
prematurity using handheld spectral domain optical coherence tomography imaging. in press, Ophthalmology, 2009.
[6] G.T. Chong , S. Farsiu , S.F. Freedman , N. Sarin , A.F. Koreishi , J.A. Izatt , and C.A. Toth . Abnormal foveal morphology in ocular
albinism imaged with spectral-domain optical coherence tomography. Archives of Ophthalmology, 127(1):3744, 2009.
[7] M. Do and M. Vetterli . The finite ridgelet transform for image representation. IEEE Trans. Image Process., 12(1):1628, January 2003.
[8] C. Dunsby and P.M.W. French . Techniques for depth-resolved imaging through turbid media including coherence-gated imaging.
Journal of Physics D: Applied Physics, 36:207227, July 2003.
[9] N.A. El-Yamany and P.E. Papamichalis . Robust color image superresolution: An adaptive m-estimation framework. EURASIP Journal
on Image and Video Processing, 8:112, 2008.
[10] M. Elad and A. Feuer . Restoration of a single superresolution image from several blurred, noisy, and undersampled measured
images. IEEE Transactions on Image Processing, 6(12):16461658, Dec 1997.
[11] M. Elad and Y. Hel-Or . A fast super-resolution reconstruction algorithm for pure translational motion and common space invariant
blur. IEEE Transactions on Image Processing, 10(8):11861193, August 2001.
408 [12] S. Farsiu , B.A. Bower , J.A. Izatt , and C.A. Toth . Image fusion based resolution enhancement of retinal spectral domain optical
coherence tomography images. Invest. Ophthalmol. Vis. Sci., 49(5):Eabstract1845, 2008.
[13] S. Farsiu , J. Christofferson , B. Eriksson , P. Milanfar , B. Friedlander , A. Shakouri , and R. Nowak . Statistical detection and imaging
of objects hidden in turbid media using ballistic photons. Applied Optics, 46(23):58055822, 2007.
[14] S. Farsiu , M. Elad , and P. Milanfar . Constrained, globally optimal, multiframe motion estimation. In Proc. of the 2005 IEEE Workshop
on Statistical Signal Processing, pages 13961401, July 2005.
[15] S. Farsiu , D. Robinson , M. Elad , and P. Milanfar . Advances and challenges in super-resolution. International Journal of Imaging
Systems and Technology, 14:4757, 2004.
[16] S. Farsiu , M.D. Robinson , M. Elad , and P. Milanfar . Fast and robust multiframe super resolution. IEEE Transactions on Image
Processing, 13(10):13271344, October 2004.
[17] A.F. Fercher , W. Drexler , C.K. Hitzenberger , and T. Lasser . Optical coherence tomography - principles and applications. Reports on
Progress in Physics, 66(2):239303, 2003.
[18] G. Golub and V. Pereyra . Separable nonlinear least squares: the variable projection method and its applications. Institute of Physics,
Inverse Problems, 19:R1R26, 2002.
[19] V.M. Govindu . Lie-algebraic averaging for globally consistent motion estimation. In Proc. of the Int. Conf. on Computer Vision and
Pattern Recognition (CVPR), volume 1, pages 684691, July 2004.
[20] H. Greenspan . Super-resolution in medical imaging. The Computer Journal, 52: 4363, 2009.
[21] H. Greenspan , G. Oz , N. Kiryati , and S. Peled . MRI inter-slice reconstruction using super-resolution. Magnetic Resonance Imaging,
20(5):437446, 2002.
[22] D. Hammer , R.D. Ferguson , N. Iftimia , T. Ustun , G. Wollstein , H. Ishikawa , M. Gabriele , W. Dilworth , L. Kagemann , and J.
Schuman . Advanced scanning methods with tracking optical coherence tomography. Optics Express, 13(20):79377947, 2005.
[23] R. Hardie . A fast image super-resolution algorithm using an adaptive wiener filter. IEEE Transactions on Image Processing,
16(12):29532964, December 2007.
409 [24] R. Hardie , K. Barnard , and E. Armstrong . Joint MAP registration and high-resolution image estimation using a sequence of
undersampled images. IEEE Transactions on Image Processing, 6(12):16211633, 1997.
[25] M.R. Hee , J.A. Izatt , E.A. Swanson , D. Huang , J.S. Schuman , C.P. Lin , C.A. Puliafito , and J.G. Fujimoto . Optical coherence
tomography of the human retina. Archives of Ophthalmology, 113(3):325332, 1995.
[26] D.B. Hess , S.G. Asrani , M.G. Bhide , L.B. Enyedi , S.S. Stinnett , and S.F. Freedman . Macular and retinal nerve fiber layer analysis
of normal and glaucomatous eyes in children using optical coherence tomography. American Journal of Ophthalmology, 139(3):509517,
2005.
[27] D. Huang , E.A. Swanson , C.P. Lin , J.S. Schuman , W.G. Stinson , W. Chang , M.R. Hee , T. Flotte , K. Gregory , C.A. Puliafito , et
al. Optical coherence tomography. Science, 254(5035):11781181, 1991.
[28] S. Jiao , R. Knighton , X. Huang , G. Gregori , and C. Puliafito . Simultaneous acquisition of sectional and fundus ophthalmic images
with spectral-domain optical coherence tomography. Optics Express, 13(2):444452, 2005.
[29] J.A. Kennedy , O. Israel , A. Frenkel , R. Bar-Shalom , and H. Azhari . Super-resolution in PET imaging. IEEE Transactions on Medical
Imaging, 25(2):137147, 2006.
[30] J.A. Kennedy , O. Israel , A. Frenkel , R. Bar-Shalom , and H. Azhari . Improved image fusion in PET/CT using hybrid image
reconstruction and super-resolution. Int. J. Biomed. Imaging, 46846, 2007.
[31] A. Khare and U.S. Tiwary . A new method for deblurring and denoising of medical images using complex wavelet transform. Proc.
IEEE Conference Engineering in Medicine and Biology, pages 18971900, Sept. 2005.
[32] S. Lertrattanapanich and N.K. Bose . High resolution image formation from low resolution frames using Delaunay triangulation. IEEE
Trans. Image Processing, 11(12):14271441, Dec. 2002.
[33] N. Nassif , B. Cense , B.H. Park , S.H. Yun , T.C. Chen , B.E. Bouma , G.J. Tearney , and J.F. de Boer . In vivo human retinal imaging
by ultrahighspeed spectral domain optical coherence tomography. Optics Letters, 29(5):480482, 2004.
[34] R. Neelamani , H. Choi , and R. Baraniuk . Forward: Fourier-wavelet regularized deconvolution for ill-conditioned systems. IEEE
Transactions on Image Processing, 52(2):418433, February 2004.
[35] S.C. Park , M.K. Park , and M.G. Kang . Super-resolution image reconstruction: A technical overview. Signal Processing Magazine,
20(3):2136, 2003.
413 [36] V. Patanavijit and S. Jitapunkul . A Lorentzian stochastic estimation for a robust iterative multiframe super-resolution
reconstruction with Lorentzian-Tikhonov regularization. EURASIP Journal on Image and Video Processing, 2007(2), 2007.
[37] R.R. Peeters , P. Kornprobst , M. Nikolova , S. Sunaert , T. Vieville , G. Malandain , R. Deriche , O. Faugeras , M. Ng , and P. Van
Hecke . The use of super-resolution techniques to reduce slice thickness in functional MRI. International Journal of Imaging Systems and
Technology, 14(3):131138, 2004.
[38] S. Peled and Y. Yeshurun . Superresolution in MRI: Application to human white matter fiber tract visualization by diffusion tensor
imaging. Magnetic Resonance in Medicine, 45(1):2935, 2001.
[39] L.C. Pickup , D.P. Capel , S.J. Roberts , and A. Zisserman . Bayesian methods for image super-resolution. The Computer Journal,
(1):101113, 2009.
[40] C.G. Pieroni , A.J. Witkin , T.H. Ko , J.G. Fujimoto , A. Chan , J.S. Schuman , H. Ishikawa , E. Reichel , and J.S. Duker . Ultrahigh
resolution optical coherence tomography in non-exudative age related macular degeneration. British Medical Journal, 90(2):191197, 2006.
[41] D. Robinson , S. Farsiu , J.Y. Lo , P. Milanfar , and C.A. Toth . Efficient multiframe registration of aliased x-ray images. Proceedings of
the 41th Asilomar Conference on Signals, Systems, and Computers, pages 215219, November 2007.
[42] D. Robinson , S. Farsiu , and P. Milanfar . Optimal registration of aliased images using variable projection with applications to super-
resolution. The Computer Journal, 52(1):3142, January 2009.
[43] D. Robinson and P. Milanfar . Statistical performance analysis of superresolution. IEEE Transactions on Image Processing,
15(6):14131428, June 2006.
[44] M.D. Robinson , C.A. Toth , J.Y. Lo , and S. Farsiu . Efficient fourier-wavelet super-resolution with applications in low-dosage digital x-
ray imaging. Submitted to IEEE Transactions on Image Processing, January 2009.
[45] R.R. Schultz and R.L. Stevenson . Extraction of high-resolution frames from video sequences. IEEE Transactions on Image
Processing, 5(6):9961011, June 1996.
[46] S.G. Schuman , A.F. Koreishi , S. Farsiu , S. Jung , J.A. Izatt , and C.A. Toth . Photoreceptor layer thinning over drusen in eyes with
age-related macular degeneration imaged in vivo with spectral-domain optical coherence tomography. Ophthalmology, 116(3):488496,
2009.
411 [47] A.W. Scott , S. Farsiu , L.B. Enyedi , D.K. Wallace , and C.A. Toth . Imaging the infant retina with a hand-held spectral-domain
optical coherence tomography device. American Journal of Ophthalmology, 147(2):364373, 2009.
[48] E. Shechtman , Y. Caspi , and M. Irani . Space-time super-resolution. IEEE Transactions on Pattern Analysis and Machine
Intelligence, 27(4):531545, 2005.
[49] V.J. Srinivasan , M. Wojtkowski , A.J. Witkin , J.S. Duker , T.H. Ko , M. Carvalho , J.S. Schuman , A. Kowalczyk , and J.G. Fujimoto .
High-definition and 3-dimensional imaging of macular pathologies with high-speed ultrahigh-resolution optical coherence tomography.
Ophthalmology, 113(11):20542054, 2006.
[50] J. Starck , E. Candes , and D. Donoho . The curvelet transform for image denoising. IEEE Trans. Image Process., 11(6):670684, June
2002.
[51] M. Stopa , B.A. Bower , E. Davies , J.A. Izatt , and C.A. Toth . Correlation of pathologic features in spectral domain optical coherence
tomography with conventional retinal studies. Retina, 28(2):298308, 2008.
[52] H. Takeda , S. Farsiu , and P. Milanfar . Kernel regression for image processing and reconstruction. IEEE Trans. Image Process.,
16(2):349366, Feb. 2007.
[53] H. Takeda , S. Farsiu , and P. Milanfar . Deblurring using regularized locally adaptive kernel regression. IEEE Transactions on Image
Processing, 17(4):550563, April 2008.
[54] P. Thevenaz , U.E. Ruttimann , and M. Unser . A pyramid approach to subpixel registration based on intensity. IEEE Transactions on
Image Processing, 7(1):2741, 1998.
[55] C.A. Toth , R. Birngruber , S.A. Boppart , M.R. Hee , J.G. Fujimoto , C.D. DiCarlo , E.A. Swanson , C.P. Cain , D.G. Narayan , G.D.
Noojin , et al. Argon laser retinal lesions evaluated in vivo by optical coherence tomography. American Journal of Ophthalmology,
123(2):188198, 1997.
[56] C.A. Toth , S. Farsiu , A.A. Khanifar , and G.T. Chong . Optical coherence tomography in age-related macular degeneration. In Gabriel
Coscas , editor, Application of Spectral Domain OCT in AMD, pages 1534. Springer Medizin Verlag Heidelberg, 2009.
[57] M. Ushizaki , T. Okatani , and K. Deguchi . Video synchronization based on co-occurrence of appearance changes in video
sequences. International Conference on Pattern Recognition, 3:7174, 2006.
412 [58] P. Vandewalle , L. Sbaiz , J. Vandewalle , and M. Vetterli . Super-resolution from unregistered and totally aliased signals using
subspace methods. IEEE Transactions on Signal Processing, 55(7):36873703, July 2007.
[59] P. Vandewalle , S. Susstrunk , and M. Vetterli . A frequency domain approach to registration of aliased images with application to
superresolution. EURASIP Journal on Applied Signal Processing, page Article ID 71459, 2006.
[60] M. Wojtkowski , R. Leitgeb , A. Kowalczyk , T. Bajraszewski , and A.F. Fercher . In vivo human retinal imaging by fourier domain
optical coherence tomography. Journal of Biomedical Optics, 7(3):457463, 2002.
[61] M. Wojtkowski , V. Srinivasan , T. Ko , J. Fujimoto , A. Kowalczyk , and J. Duker . Ultrahigh-resolution, high-speed, fourier domain
optical coherence tomography and methods for dispersion compensation. Optics Express, 12(11):24042422, 2004.
[62] N. Woods , N. Galatsanos , and A. Katsaggelos . Stochastic methods for joint registration, restoration, and interpolation of multiple
undersampled images. IEEE Transactions on Image Processing, 15(1):201213, January 2006.
[63] Y. Yasuno , J. Sugisaka , Y. Sando , Y. Nakamura , S. Makita , M. Itoh , and T. Yatagai . Non-iterative numerical method for laterally
superre-solving Fourier domain optical coherence tomography. Optics Express, 14(3):10061020, 2006.
[64] S. Yun , G. Tearney , J. de , Boer, N. Iftimia, and B. Bouma . High-speed optical frequency-domain imaging. Optics Express,
11(22):29532963, 2003.
[65] R. Zawadzki , S. Jones , S. Olivier , M. Zhao , B. Bower , J. Izatt , S. Choi , S. Laut , and J. Werner . Adaptive-optics optical coherence
tomography for high-resolution and high-speed 3d retinal in vivo imaging. Optics Express, 13(21):85328546, 2005.
[66] R.J. Zawadzki , A.R. Fuller , S.S. Choi , D.F. Wiley , B.Hamann, and J.S. Werner . Improved representation of retinal data acquired
with volumetric Fd-OCT: co-registration, visualization, and reconstruction of a large field of view. volume 6844, page 68440C. SPIE
Ophthalmic Technologies XVIII, 2008.

Practicing Super-Resolution: What Have We Learned?

[1] S. Baker and T. Kanade . Hallucinating faces. In Fourth International Conference on Automatic Face and Gesture Recognition, pages
8388, 2000.
[2] B. Bascle , A. Blake , and A. Zisserman . Motion deblurring and superresolution from an image sequence. In ECCV 96: Proceedings of
the 4th European Conference on Computer Vision-Volume II, pages II:571582, 1996.
[3] BBC News. 1,000 cameras solve one crime. http://news.bbc.co.uk (Web site), August 2009.
http://news.bbc.co.Uk/2/hi/uk_news/england/london/8219022.stm.
[4] M. Ben-Ezra , A. Zomet , and S. K. Nayar . Video super-resolution using controlled subpixel detector shifts. IEEE Transactions on
Pattern Analysis and Machine Intelligence, 27(6):977987, 2005.
[5] N. Bozinovic . Advanced motion modeling for 3D video coding. PhD thesis, Boston University, ECE Deptartment., December 2006.
[6] T. Brand and M. P. Queluz . No-reference image quality assessment based on DCT domain statistics. Signal Processing,
88(4):822833, 2008.
[7] D. Bulwa . San Francisco security cameras choppy video. http://www.sfgate.com, January 2008. http://www.sfgate.com/cgi-
bin/article.cgi?f=/c/a/2008/01/28/MN37TKH6.DTL.
[8] B. Drawbaugh . Panasonic does it again, introduces 152-inch 3D 4K HDTV. http://www.engadget.com (Web site), 2009.
http://www.engadget.com/2010/01/06/panasonic-does-it-again-introduces-152-inch-3d-4k-hdtv/.
445 [9] C. Drew . Military is awash in data from drones. The New York Times, page A1, January 10, 2010.
http://www.nytimes.com/2010/01/11/business/lldrone.html.
[10] A. Eisenberg . Those big bright eyes may soon be brighter. The New York Times, page BU3, June 20, 2009.
http://www.nytimes.com/2009/06/21/business/21novel.html.
[11] Law Enforcement and Emergency Services Video Association. www.leva.org (Web site), 2010. http://www.leva.org.
[12] S. Farsiu , M. Elad , and P. Milanfar . Video-to-video dynamic superresolution for grayscale and color sequences. EURASIP Journal of
Applied Signal Processing, Special Issue on Superresolution Imaging, 2006:115, 2006.
[13] S. Farsiu , D. Robinson , M. Elad , and P. Milanfar . Advances and challenges in super-resolution. International Journal of Imaging
Systems and Technology, 14(2):4757, 2004.
[14] S. Farsiu , D. Robinson , M. Elad , and P. Milanfar . Fast and robust multi-frame super-resolution. IEEE Transactions on Image
Processing, 13(10):13271344, Oct. 2004.
[15] B. Girod , A.M. Aaron , S. Rane , and D. Rebollo Monedero . Distributed video coding. IEEE Transactions on Image Processing,
93(1):7183, January 2005.
[16] B. K. Gunturk , A. U. Batur , Y. Altunbasak , M. H. III Hayes , and R. M. Mersereau . Eigenface-domain super-resolution for face
recognition. IEEE Transactions on Image Processing, 12(5):597606, 2003.
[17] S.T. Hsiang and J.W. Woods . Invertible three-dimensional analysis/synthesis system for video coding with half-pixel-accurate motion
compensation. In SPIE Visual Communications and Image Processing, volume 3653, pages 537546, January 1999.
[18] S.T. Hsiang , J.W. Woods , and J.R. Ohm . Invertible temporal sub-band/wavelet filter banks with half-pixel-accurate motion
compensation. IEEE Transactions on Image Processing, 13:10181028, August 2004.
[19] iSuppli Corporation . Camera penetration in mobile phones, 2004-2010. http://www.isuppli.com (Web site).
[20] J. Kincaid . Youtube mobile uploads up 400 percent since iPhone 3GS launch. http://www.techcrunch.com (Web site), June 2009.
http://www.techcrunch.com/2009/06/25/youtube-mobile-uploads-up-400-since-iphone-3gs-launch/.
446 [21] Z.C. Lin and H.Y. Shum . Fundamental limits of reconstruction-based superresolution algorithms under local translation. IEEE
Transactions on Pattern Analysis and Machine Intelligence, 26(1):8397, January 2004.
[22] NVIDIA Corporation . NVIDIA CUDA. http://www.nvidia.com (Web site), 2009. http://www.nvidia.com/cuda.
[23] NVIDIA Corporation . NVIDIA Tesla personal supercomputer. www.nvidia.com (Web site), 2009.
http://www.nvidia.com/object/personal_supercomputing.html.
[24] NVIDIA Corporation . Universities teaching CUDA. www.nvidia.com (Web site), 2009.
http://www.nvidia.com/object/cuda_university_courses.html.
[25] P.J. Rousseeuw and A.M. Leroy . Robust Regression and Outlier Detection. John Wiley & Sons, Inc. New York, NY, USA, 1987.
[26] M. Protter and M. Elad . Super resolution with probabilistic motion estimation. IEEE Transactions on Image Processing,
18(8):18991904, August 2009.
[27] M. Protter , M. Elad , H. Takeda , and P. Milanfar . Generalizing the nonlocal-means to super-resolution reconstruction. IEEE
Transactions on Image Processing, 18(1):3651, Jan 2009.
[28] R. Puri , A. Majumdar , and K. Ramchandran . Prism: A video coding paradigm with motion estimation at the decoder. IEEE
Transactions on Image Processing, 16(10):24362448, October 2007.
[29] RealNetworks, Inc. Video downloader. http://www.real.com (Web site), 2010. http://www.real.com/realplayer/video-downloader.
[30] B. D. Rizzo . NVIDIA announces strategic partnership with MotionDSP. http://www.nvidia.com (Web site), 2008.
http://www.nvidia.com/object/io_1222345067642.html?_templateId=320.
[31] D. Robinson and P. Milanfar . Statistical performance analysis of superresolution. IEEE Transactions on Image Processing,
15(10):14131428, June 2006.
[32] D. Robinson and D. Stork . Joint digital-optical design of imaging systems for gray-scale objects. In SPIE European Optical Design
Conference, volume 7100, pages 710011710011.9, 2008.
[33] Sarnoff Corporation . Acadia video processors. http://www.sarnoff.com (Web site), 2010. http://www.sarnoff.com/products/acadia-
video-processors.
447 [34] H. S. Sawhney and S. Ayer . Compact representations of videos through dominant and multiple motion estimation. IEEE
Transactions on Pattern Analysis and Machine Intelligence, 18:814830, 1996.
[35] H. Schwarz and D. Marpe and. T. Wiegand . MCTF and scalability extension of H.264/AVC. In Procedings Picture Coding
Symposium, December 2004.
[36] S. Seguin . NVIDIA launches Tesla personal supercomputer. http://www.tomshardware.com (Web site), 2008.
http://www.tomshardware.com/news/Nvidia-Tesla-Supercomputer,6616.html.
[37] M.G. Siegler . Every minute, just about a days worth of video is now uploaded to YouTube. http://www.techcrunch.com (Web site),
2009. http://www.techcrunch.com/2009/05/20/every-minute-just-about-a-days-worth-of-video-is-uploaded-to-youtube/.
[38] Webware staff . Webware 100 winner: FixMyMovie. http://news.cnet.com (Web site), 2008. http://news.cnet.com/8301-13546_109-
9914732-29.html.
[39] D. Stork and D. Robinson . Theoretical foundations for joint digital-optical analysis of electro-optical imaging systems. Applied Optics,
47:B64B75, April 2008.
[40] H. Takeda , P. Milanfar , M. Protter , and M. Elad . Superresolution without explicit subpixel motion estimation. IEEE Transactions on
Image Processing, 18(9): 19581975, Sep. 2009.
[41] R. Y. Tsai and T. S. Huang . Multiframe image restoration and registration. Advances in Computer Vision and Image Processing,
1:317339, 1984.
[42] Y. Wang , Y-Q. Zhang, and J. Ostermann . Video Processing and Communications. Prentice Hall PTR, Upper Saddle River, NJ, USA,
2001.
[43] Z. Wang and A. C. Bovik . Modern Image Quality Assessment. Morgan and Claypool Publishers, 2006.
[44] Z. Wang , H.R. Sheikh , and A.C. Bovik . No-reference perceptual quality assessment of JPEG compressed images. In IEEE
International Conference on Image Processing, pages I: 477480, 2002.