Image Deblurring
Image Deblurring
Samuel Jaffe
Date: 12/29/13
_____________________
Professor Homer Walker, Major Advisor
Abstract
This project was carried out in order to improve our understanding of the mechanisms
behind image deblurring. In order to gain a more thorough understanding of the basic processes
that are employed in image deblurring, a number of references and textbooks were used to build
up a background of understanding. Experiments were performed to explore the effectiveness of
the basic methods. Through this investigation, we were able to develop an understanding of the
sensitivity towards error that is experienced by some of the nave deblurring solutions that were
investigated. Additionally, we gained a better appreciation of the ability of spectral filtering
techniques to mitigate the influence of noise on deblurring techniques.
ii
Table of Contents
Abstract
...................................................................................................................................................
ii
1
Introduction
.......................................................................................................................................
1
2
Background
........................................................................................................................................
3
2.1
General
Method
of
Image
Deblurring
................................................................................................
3
2.2
Point
Spread
Function
and
Boundary
Conditions
.........................................................................
5
2.3
Spectral
Analysis
......................................................................................................................................
9
3
Experimental
Results
....................................................................................................................
11
3.1
Blurring
and
Deblurring
without
Noise
.........................................................................................
12
3.2
Sensitivity
of
the
Eigenvalue
Matrix
...............................................................................................
15
3.3
Spectral
Filtering
Techniques
...........................................................................................................
19
3.4
Extension
to
General
Blurry
Images
...............................................................................................
21
4
Weaknesses
and
Limitations
......................................................................................................
23
5
Conclusions
.......................................................................................................................................
24
Bibliography
........................................................................................................................................
25
iii
Table of Equations
Equation 1: Linear Model for Blurring ........................................................................................... 3
Equation 2: Error Term in Blurring Equation ................................................................................. 4
Equation 3: Rank-k Truncated Matrix ............................................................................................ 5
Equation 4: Out-of-Focus Blur ....................................................................................................... 6
Equation 5: Atmospheric Blur ........................................................................................................ 6
Equation 6: Astronomical Telescope Blur ...................................................................................... 6
Equation 7: Deblurring with Kronecker Decomposition ................................................................ 8
Equation 8: Discrete Cosine Transform .......................................................................................... 8
Equation 9: Discrete Fourier Transform ......................................................................................... 8
Equation 10: TSVD Method for Spectral Filtering......................................................................... 9
Equation 11: Tikhonov Method for Spectral Filtering ................................................................... 9
Equation 12: An Example Low Pass Filter ................................................................................... 15
Equation 13: Adding Random Noise to the Blurred Image .......................................................... 16
iv
Table of Figures
Figure 1: Original Image of Pumpkins ........................................................................................... 4
Figure 2: Blurred Image of Pumpkins ............................................................................................ 4
Figure 3: Deblurred with Gaussian Elimination ............................................................................. 4
Figure 4: Rank-k Deblurring with k=800 (Out of 169744) ............................................................ 5
Figure 5: Examples of Tikhonov and TSVD Filtering ................................................................. 10
Figure 6: Original Image ............................................................................................................... 11
Figure 7: Gaussian Blur with s=2 ................................................................................................. 12
Figure 8: Applying the FFT Algorithm ........................................................................................ 12
Figure 9: Center Area of Original Image ...................................................................................... 13
Figure 10: Gaussian Blur with s=10 ............................................................................................. 13
Figure 11: Central Part of Deblurred Image, to Showcase Graininess (s=10) ............................. 13
Figure 12: Gaussian Blur with s=15 ............................................................................................. 14
Figure 13: Central Part of Deblurred Image, to Showcase Graininess (s=15) ............................. 14
Figure 14: Applying a Low Pass Filter to the Grainy Deblurred Image ....................................... 15
Figure 15: A PSF Using the Gaussian Method and s=2 ............................................................... 16
Figure 16: Deblurred Image with 1% Noise ................................................................................. 17
Figure 17: Deblurred Image with 1e-12 Noise ............................................................................. 17
Figure 18: Deblurred Image with 1e-13 Noise ............................................................................. 18
Figure 19: Deblurred Image with 1e-14 Noise ............................................................................. 18
Figure 20: TSVD Filtering with tol = 0.5*err ............................................................................... 19
Figure 21: TSVD Filtering with tol = 0.5 ..................................................................................... 19
Figure 22: TSVD Filtering with err=0.1% and tol=0.01 .............................................................. 20
Figure 23: TSVD Filtering with err=0.01% and tol=0.01 ............................................................ 20
Figure 24: Tikhonov Filtering with alpha=0.1 and err=0.01 (top) and 0.001 (bottom)................ 21
Figure 25: Photo of Plant, Sharp Foreground, Blurry Background .............................................. 22
Figure 26: Photo of Plant, Blurry.................................................................................................. 22
Figure 27: Deblurring the Photo of a Plant Using TSVD Filtering .............................................. 22
1 Introduction
Image deblurring is an old problem in the realm of image processing, one that
continues to garner attention from academics and businesses alike. It has applications in
many different real-world problems and serves as an easy way to visualize examples of a
larger range of inverse problems in many fields [8]. Image deblurring seeks to take a
blurry image and restore it to its original form algorithmically. When deblurring images,
a mathematical description of how it was blurred is very important to maximizing the
effectiveness of the deblurring process. With real-world photos, we do not have the
luxury of knowing the mathematical function by which the image was blurred. However,
there exist methods to approximate how blur occurred.
There are many different sources of blur in a photograph, such as motion blur,
camera shake and long exposure times. Even a relatively small amount of any of these
effects can be enough to ruin an otherwise good photograph. Image deblurring tools and
research seek to solve this problem by taking the blurred images and attempting to restore
or alter them to a sharper, clearer state. Because every picture taken comes out blurry to
some degree, image deblurring is fundamental in making pictures sharp and useful [1].
Image deblurring and similar techniques are also applicable in commercial
settings. Image enhancement techniques were used to enhance the original Star Wars
trilogy for its DVD release [2]. Image deblurring techniques can also be applied in fields
ranging from the analysis of astronomical data to barcode readers at supermarkets or
shipping companies [3]. Because of image deblurring techniques ability to be used for
valuable purposes such as enhancing videos for higher definition releases, there is a lot of
interest in developing algorithms and models that can be used to improve their
effectiveness.
Most modern cameras come with some form of hardware-based image
stabilization feature. This allows the camera to reduce the noise caused by blur in the
image without needing to depend on a software level solution. The most common form
used in camera is called Optical Image Stabilization (OIS) [9]. Different companies use
slightly different methods to achieve this result, but it allows the same likelihood of a
sharp image for shutter speeds 8-16 times slower than without any OIS [10]. Other
hardware-based stabilization techniques include gyroscopically stabilized gimbals, which
are both heavy and expensive, and boosting the film speed, which can result in a noisy
(grainy) photograph. Due to technological limitations, digital cameras are unable to
perform software level deblurring on pictures.
Through this investigation, we seek to improve our understanding of the
techniques and limitations surrounding image-deblurring technologies, and to explore
their effectiveness in simple, controlled cases. We also seek to use this investigation to
develop a more sophisticated understanding and appreciation for some of the linear
algebra techniques used in these methods.
In the following section, some background information on image deblurring and
associated techniques will be presented. Following that, we will discuss results that were
obtained, as well as discussing some of the weaknesses and limitations in the
investigation.
2 Background
In order to be able to better understand the methods of image deblurring that were
applied during this project, research was done into the mechanism behind some of the
nave approaches to image deblurring. The investigation went through three stages of
deblurring techniques of increasing sophistication in order to build up an understanding
of the basic methods and weaknesses of image deblurring. For the rest of this report,
unless otherwise stated, all images will be assumed to be in a gray scale.
! !! =
Equation 1: Linear Model for Blurring
Where ! and ! are the column and row blurring matrices, !! is the transpose, B is the
blurred image, and X is the original image. B and X are matrices in !! (The set of mby-n matrices of real numbers), ! is in !! and ! is in !! . We would expect
!!
!"!" = !!
! ! to be the case, but a fairly simple example shows that the resulting
output bears no resemblance to the original image. Because of factors like noise and
imprecision, knowing the exact blurring matrices for X is not sufficient to restore the
image. A simple example is provided below [1].
= +
Equation 2: Error Term in Blurring Equation
Unfortunately, the value of the noise term is unknown, and so the goal of more
!!
sophisticated methods is to minimize the influence of the inverted noise !!
! ! . The
reason that the noise causes the deblurred image to be unrecognizable is because its high4
frequency components are amplified when the small singular values of the blurring
matrices are inverted [1]. The simplest way to reduce this effect is by creating rank-k
versions of the row and column matrices from their singular value decompositions.
!!
= v!
v!
!!
u!!
u!!
While this method provides a better restoration than the previous solution, the noise still
has a large effect on the resulting image using vec = !!
! vec .
same way, we can imagine applying A to a black image with a single pixel of light,
known as a point source. The resulting image that describes how A acts on individual
pixels is called the Point Spread Function (PSF) [1]. Depending upon the type of blurring
that is being described, we have different functions that generate the PSF around a central
point of , . A few example PSFs are included below [1].
!,! =
1 !
0
if ! +
elsewhere
= exp
, = +
The function for atmospheric blur is commonly known as the two-dimensional Gaussian
function [4], and the function used for astronomical telescopes is called the Moffat
function [5]. By convoluting the generated PSF and the original image X, we can
generate the blurred image without needing to construct the larger blurring matrix.
Convolution can also be used to smooth noise or enhance edges in a picture with an
appropriate PSF.
Alone, the PSF only allows us to increase the speed at which we reach a poorly
deblurred image. In order to improve the accuracy of image reconstructions, we need to
take into account that the image we are looking at does not exist in a vacuum, but rather,
extends infinitely beyond the captured area. As such, noise can be generated from outside
the boundary of the image. In order to counteract this effect, the use of boundary
conditions allows us to make a simple assumption about the state of the world outside of
the images frame.
There are three types of boundary conditions commonly used for this purpose,
with different situations where they are used. For each boundary condition, the matrix X
is placed in the center of a matrix three times as large as it. In the zero boundary
condition, all other entries are left as zero. Zero boundary conditions assume that all
space beyond the image is empty, which is a good assumption to make when working
with astronomical data. A periodic boundary condition assumes that the image repeats
infinitely, and so tiles the original image in each of the blocks of the larger matrix. The
reflexive boundary condition assumes that the image is mirrored outside of the border,
and is created by flipping the original image depending on what block it is in, vertically if
it is above or below, horizontally if it is to the left or right, and both if it is diagonal of the
middle block where the original image was placed.
For certain states of the PSF and assumptions of the boundary condition, we can
use fast algorithms in order to calculate the blurred image and a nave solution from it.
When the PSF is separable1, under any boundary condition, we can interpret the blurring
matrix A as a Kronecker product of two smaller matrices. Using the PSF, we are able to
calculate the Ac and Ar whose Kronecker product form the blurring matrix A without
ever having to construct A. We can then apply singular value decomposition to these
= !
= ! !
! = ! ! !!
! = ! ! !!
!"!" = ! !!! !! ! !!! !!
Equation 7: Deblurring with Kronecker Decomposition and SVD
For PSFs that are doubly symmetric, under the reflexive boundary condition we can
apply the two-dimensional Discrete Cosine Transform (DCT) algorithm, which computes
the eigenvalues of the blurring matrix A from the PSF and blurs/deblurs the image
without constructing A [1].
!
! = !
! cos
!!!
2 1 1
2
The two-dimensional Fast Fourier Transform (FFT) algorithm can be applied whenever
there is a periodic boundary condition and works in a similar manner to the DCT
algorithm.
! =
!!!
! !!!"
!!! !!! !
For both the FFT and DCT, we compose their results over each dimension to obtain the
resulting matrix [4][6][7]. These algorithms allow us to compute efficiently the blur and
nave restoration of large images efficiently, but are still incredibly vulnerable to noise in
the data corrupting the image.
1
0
= 1, ,
= + 1, ,
Another method for generating the filter factors is the Tikhonov Method.
!!
! !
! + !
Equation 11: Tikhonov Method for Spectral Filtering
The in the Tikhonov Method equation is called the regularization parameter; this
parameter behaves in much the same way as the choice of k does in TSVD in damping
small eigenvalues that would increase the inverted noise.
In the above example [1], Tikhonov regularization (bottom left) and TSVD (bottom right)
were selected for the most visually pleasing result.
While these methods allow a much better reconstruction of the original image,
more sophisticated methods of image deblurring are required in order to generate more
accurate deblurred images.
10
3 Experimental Results
The results reported below were obtained under a single set of conditions. All
PSFs were generated using Gaussian (atmospheric) blur with = 0 and ! = ! , and all
images were manipulated under periodic boundary conditions with the Fast Fourier
Transform method. This was done both to prevent getting bogged down in the differences
between different algorithms on a given image and because the FFT method had the
fasted clock time for images of any size. Furthermore, we will be only applying
deblurring methods to single, blurred images to attempt to retrieve restored images.
Unless otherwise noted, the images worked with are all based off of a sharp original
image that is blurred through a given method so that the PSF of the blur is known.
The following picture was taken at the primary researchers home, using a tripod
for stabilization. The focus of the image was on the bowl of fruit and the picture was
taken with an aperture of 5.5. For comparison purposes, some of the deblurred images
will focus on just the small section containing the tabletop and fruit bowl itself in order to
convey differences that may not be easily apparent with the full image at this scale.
11
It was determined that as the radius of the PSF increased, the noise that was observed in
the restored image would increase as well.
13
We observe that even with a Gaussian blur using s=10, which leaves the blurred image
such that the fruit bowl is simply an unidentifiable blur, there is very little graininess in
the restored image. What noise is noticeable from this deblurring is concentrated around
edges in the photo (such as the edges of the table). When the blur is increased to use
s=15, the blurred image becomes even less recognizable, and in the full-sized image we
are able to see a noticeable amount of grain.
14
Simply by convoluting the grainy image with a low pass filter3, we can smooth the
graininess from the restored image; however, we receive a slightly blurry image in
exchange. The low pass filter used and resultant image are shown below.
1
1
1
1
2
1
1
1 10
1
Figure 14: Applying a Low Pass Filter to the Grainy Deblurred Image
As can be seen, an image that has been blurred without any random noise added in can be
blurred to a high degree without significant loss or noise in the restored image.
A low pass filter averages each pixel with its neighbors using some weighting, and the
sum of its elements is 1
15
following images are based on the fruit bowl image which has been blurred by a PSF
with s=2. Below is the Gaussian PSF in a 20-by-20 grid centered on the blurring point.
= +
In Equations 13, E is a matrix containing normalized random noise of the same size as B,
and
is the Frobenius norm4 of B. We can see that the amount of error noise that is
added to the image must be incredibly small in order to not cause the deblurred image to
be overcome with inverted noise to the point of there being nothing but noise left in the
image.
=
16
We can observe that the condition number of the eigenvalue matrix S is 2.68e+18. This
means that even a very small amount of noise is magnified to massive degrees when
inverted in the deblurring process. When S element-wise divides the Fourier transform of
the noisy blurred image, the noise is magnified by an enormous factor, dominating the
image. By testing this out with different degrees of error, we find that we require
reducing the amount of error to 1e-12 in order for less than half of the image to be
dominated by noisy patches.
17
Stepping beyond that, we observe that reducing the noise by a factor of ten reduces the
effect of inverted noise to a level where we must look closer at any section of the restored
image to notice the corruption due to inverted noise. Going one step further to an error
factor of 1e-14 we are able to more or less purge the inverted noise from the image.
18
For PSFs with s=2, we found that with the TSVD filtering approach the tolerance needed
to be between 0.5 and 0.5 times the error in order to produce a reasonable image. The
range is smaller as the amount of blur induced in the image increases;for s=4, a range of
0.05 to the error amount produces a good reconstruction. This implies that the range of
19
tolerances that will allow for the deblurred image to be good quality will shrink as the
blurred image gets blurrier.
Despite the limitations on how restrictive the tolerance parameter can be in order
to reconstruct an image into something that is both recognizable and of reasonable
quality, we are able to apply this filtering technique and obtain good results with error
noise as high as 0.1%.
The tolerance here is ten times the error parameter, and the resultant image is of good
quality, though the image is made choppy by the noise that is not canceled. Reducing the
error by one order of magnitude while keeping the tolerance at 0.01 reduces the amount
of inverted noise even more.
20
Applying the Tikhonov method to filter the singular values generates results that have
less noise than their TSVD counterparts, but are blurrier than them as well. This is due to
the different way the singular values are trimmed by the different filters.
Figure 24: Tikhonov Filtering with alpha=0.1 and err=0.01 (top) and 0.001 (bottom)
As can be seen with the above examples, this approach is insufficient to create a truly
sharp restored image from the blurry, noise added image.
21
As we can see, the deblurring leaves much to be desired when compared to a sharper
image, though it is sharper than the starting image. However, given that the PSF
parameter that best represented the image was so high, the reconstruction did fairly well.
22
23
5 Conclusions
Advances in image deblurring and similar techniques are important both to the
development of modern photography and to the restoration of images and videos that are
not as sharp as they can be. The ability to remove noise from images captured in highly
technical fields such as astronomy and medicine is critically important to increasing the
ability of related professionals to fulfill their jobs in the most effective manner possible.
This provides a strong reason for research into improving the accuracy and sophistication
of image deblurring techniques in both academic and corporate settings.
Though our investigation was limited to some of the more basic and illconditioned methods of deblurring, the results demonstrate the value in image deblurring
techniques and in looking into study of more sophisticated techniques. Investigation into
the more advanced and effective techniques of image deblurring would certainly be worth
doing in order to obtain a stronger understanding of the problems and algorithms in
image deblurring.
Through this study, we were able to gain a better understanding of both the
implementation and the concepts behind how images are handled in the deblurring
process. Additionally, through learning about these methods, a stronger understanding of
linear algebra concepts such as Fourier transforms was developed. Further, through this
investigation, we learned more about the importance of techniques such as singular value
decomposition and gained an appreciation for the value of spectral filtering techniques
for eliminating the small singular values that are inverted in the reconstruction.
Additionally, we gained an understanding about some of the limitations that are present
in image deblurring due to the weakness of nave methods.
24
Bibliography
[1] P. C. Hansen, J. Nagy, and D. OLeary, Deblurring Images: Matrices, Spectra
and Filtering, SIAM, Philadelphia, 2008.
[2] L. Joyeux, S. Boukir, B. Besserer and O. Buisson. Reconstruction of degraded
image sequences. Application to film restoration. Image and Vision Computing,
19:504-516, 2000.
[3] T. Wittman. Lost in the supermarket: Decoding blurry barcodes. SIAM News,
37(7):16, September 2004.
[4] A. K. Jain. Fundamentals of Digital Image Processing. Prentice-Hall, Englewood
Cliffs, NJ, 1989.
[5] A. F. J. Moffat. A theoretical investigation of focal stellar images in the
photographic emulsion and applications to photographic photometry. Astronom.
Astrophys. 3:455-461, 200.
[6] P. J. Davis. Circulant Matrices. Wiley, New York, 1979.
[7] C. F. Van Loan. Computational Frameworks for the Fast Fourier Transform.
SIAM, Philadelphia, 1992.
[8] S. Reeves, Image Deblurring Introduction. 2007,
http://blogs.mathworks.com/steve/2007/08/13/image-deblurring-introduction/.
[9] M. orel, Recent advances in deblurring and image stabilization. Czech Academy
of Sciences , 2008, http://zoi.utia.cas.cz/files/ASI2008slides.pdf.
[10] K. Rockwell, Image Stabilization (IS) and Vibration Reduction (VR), 2006,
http://www.kenrockwell.com/tech/image-stabilization.htm.
[11] R. Fergus, B. Singh, A Hertzmann, S. T. Roweis, W. T. Freeman, Removing
Camera Shake from a Single Photograph. MIT CSAIL, University of Toronto,
2006.
25