Copyright (c) 2010 IEEE. Personal use is permitted.

For any other purposes, Permission must be obtained from the IEEE by emailing [email protected].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
1
Efcient Fourier-wavelet Super-resolution
M. Dirk Robinson

, Cynthia A. Toth

, Joseph Y. Lo

, Sina Farsiu

Ricoh Innovations, Menlo Park, CA, Email: [email protected], Phone: 650-496-5703

Depts. of Ophthalmology and Biomedical Engineering, Duke University, Durham, NC,

Email: [email protected], Phone: 919-684-6611, Fax: 919-681-9523

Depts. of Radiology and Biomedical Engineering, Duke University, Durham, NC, Email:
[email protected], Phone: 919-684-7763, Fax: 919-681-7122

Depts. of Ophthalmology and Biomedical Engineering, Duke University, Durham, NC, Email:
[email protected], Phone: 919-684-6642, Fax: 919-681-9523 (corresponding Author)
Abstract
Super-resolution is the process of combining multiple aliased low-quality images to produce a high-
resolution high-quality image. Aside from registration and fusion of low-resolution images, a key process
in super-resolution is the restoration and denoising of the fused images. We present a novel extension
of the combined Fourier-wavelet deconvolution and denoising algorithm ForWarD to the multi-frame
super-resolution application. Our method rst uses a fast Fourier-base multiframe image restoration to
produce a sharp, yet noisy estimate of the high-resolution image. Our method then applies a space-variant
nonlinear wavelet thresholding that addresses the non-stationarity inherent in resolution-enhanced fused
images. We describe a computationally efcient method for implementing this space-variant processing
that leverages the efciency of the Fast Fourier Transform (FFT) to minimize complexity. Finally, we
demonstrate the effectiveness of this algorithm for regular imagery as well as in digital mammography.
1
Index Terms
Super-Resolution, Digital X-ray Imaging, Multi-Frame Deblurring, Wavelets, Denoising
EDICS Category: ISR-SUPR
1
Preliminary results of this work were presented at ICIP, October 2008 [1].
March 18, 2010 DRAFT
Authorized licensed use limited to: Annai Mathammal Sheela Engineering College. Downloaded on June 11,2010 at 01:09:37 UTC from IEEE Xplore. Restrictions apply.
Copyright (c) 2010 IEEE. Personal use is permitted. For any other purposes, Permission must be obtained from the IEEE by emailing [email protected].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
2
I. INTRODUCTION
Super-resolution (SR) is the process of combining multiple aliased low-resolution (LR) images to
produce a high-resolution high-quality image at a resolution greater than the sampling rate of the detector.
Such processing, commonly called super-resolution (SR), has received much attention in recent years in
the image processing community. We refer the interested reader to [2], [3], [4] for a broad review of
recent algorithmic development in this area.
Aside from registration and fusion of low-resolution images, a key process in super-resolution is
the restoration and denoising of fused images. In this paper, we propose an efcient restoration and
denoising method that is a novel multiframe extension of the Fourier Wavelet Regularized Deconvolution
(ForWarD) algorithm [5], which considers the non-stationarity of the multiframe reconstruction process.
The algorithms efciency stems from separating the multiframe deconvolution or restoration step from
the wavelet-based denoising step allowing us to achieve nonlinear denoising in a non-iterative fashion.
Furthermore, we describe how to efciently implement the algorithm to address the computational
complexity associated with the non-stationary noise processes inherent to multiframe reconstruction.
In this paper, we also explore an application of this algorithm to digital mammography. While relatively
new, digital mammography is rapidly replacing lm-based mammography for the screening and diagnosis
of early carcinomas in women. Solid-state detectors have demonstrated improved performance in terms
of specicity and sensitivity over lm-based imaging for certain groups of women such as those with
dense breast tissue, women under the age of fty, and pre-menopausal women [6].
Unlike lm-based mammography, digital mammography provides the opportunity to directly apply
sophisticated digital processing techniques without the need for a secondary lm scanning process. An
ideal digital mammography system exposes the patient to the minimum amount of radiation required to
accomplish the screening task. Digital mammography systems face the same design tradeoff between
image resolution, signal to noise ratio (SNR), and illumination or radiation exposure level as those found
in any digital imaging system. Shrinking the pixel dimension at the detector increases sampling resolution
at the expense of dynamic range and SNR. While improved SNR and dynamic range may be obtained
by combining multiple images, increasing total radiation beyond the standard dosage is undesirable for
the safety of the patient. Alternatively, using large detector pixels improves both the dynamic range and
the SNR of the system at the obvious expense of resolution. Digital mammography imaging systems
typically choose the highest resolution which supports a minimum required SNR.
To overcome the said quality tradeoffs, we propose digitally combining multiple low-dosage images,
March 18, 2010 DRAFT
Authorized licensed use limited to: Annai Mathammal Sheela Engineering College. Downloaded on June 11,2010 at 01:09:37 UTC from IEEE Xplore. Restrictions apply.
Copyright (c) 2010 IEEE. Personal use is permitted. For any other purposes, Permission must be obtained from the IEEE by emailing [email protected].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
3
a: 226 mAs b: 11.3mAs c: Restored
Fig. 1. Mammogram X-ray images from a phantom breast containing a pentagram-shaped set of micro-calcication cluster.
(a) High dosage at 226mAs, faintly showing the nodules (PSNR 13dB). (b) Extremely low-dosage at 11.3 mAS used in the
proposed multi-frame reconstruction scheme (PSNR 3dB), where the nodules are almost undetectable. The total dosage of
using 15 of these frames (15 11.3 = 170mAs) is still less than the high dosage. (c) Restoration combining the 15 low-dosage
frames, clearly demonstrating the pentagram-shaped set of micro-calcication cluster.
each containing spatial shifts. This motion may be the result of patient movement, intentional dithering
of the detector, vibration in the imaging system, or small movement of the imaging gantry. In practice,
the motion contained in the captured images is a combination of all such sources necessitating accurate
registration of the aliased low-resolution (LR) images.
Applying SR algorithms to digital mammography has two inherent challenges. The captured low-
resolution images are typically of size 10 megapixels and larger. Thus, algorithmic efciency is very
important due to the sheer size of the reconstructed images, which could range from 40 to 160 megapixels
depending on the resolution enhancement factor. Also, to minimize total radiation exposure, we must use
lower than normal dosages of X-ray exposure for capturing each frame. Therefore, the captured data has
extremely low peak SNR (PSNR). For example, Fig. 1 compares a high dosage X-ray image (computed
PSNR
2
13 dB) with the very low exposure images (computed SNR 3 dB) used in our multi-frame
scheme. We demonstrate that the effectiveness of the two-stage restoration and denoising algorithm
allows us to provide high-resolution, high contrast, and low noise images at very low radiation dosages.
Furthermore, our results suggest improved detection rates of texture resembling small calcication in
breast tissue. These results suggest new tradeoffs in designing digital mammogram systems.
In Section II, we describe the forward imaging model and the problem of super-resolution (SR). In
2
In this work, the PSNR was computed numerically as PSNR= 20log
10
s
n
. In experiments on real images, s is the grayscale
difference between the minimum and maximum signal regions and n is the noise standard deviation estimated from at regions.
In simulated experiments, n is the RMSE error between the estimated and ground truth image.
March 18, 2010 DRAFT
Authorized licensed use limited to: Annai Mathammal Sheela Engineering College. Downloaded on June 11,2010 at 01:09:37 UTC from IEEE Xplore. Restrictions apply.
Copyright (c) 2010 IEEE. Personal use is permitted. For any other purposes, Permission must be obtained from the IEEE by emailing [email protected].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
4
Section III, we describe the multiframe variant of the ForWarD algorithm we employ to restore and
denoise the reconstructed images. Section IV presents experimental results using this new approach and
Section V outlines some future directions of this work.
II. BACKGROUND: SR AND EFFICIENT DEBLURRING
In this section, we establish the background required for the development of our efcient multi-frame
SR algorithm in Section III. We introduce our imaging model and since our method is a combination of
the multi-frame SR and ForWarD deblurring algorithms, we also review these two concepts. Moreover,
we review a novel multi-frame motion estimation algorithm [7] that is used to produce the results in
Section IV. We believe that due to some algorithmic similarities, this brief review of the motion estimation
technique facilitates the study of the method described in Section III.
A. Imaging Model
The captured LR X-ray images are often very large and may contain complicated relative motions due
to patient motion. However, following several other popular SR methods [8], [9], [10], we consider the
translational (or pure rotational) motion models. To better justify and extend the application of this model,
in [11], we introduced a novel joint motion estimation and SR approach in a tile-based fashion. That is,
instead of considering a global translational motion model, we assume that each LR image is made of
a set of small sized tiles (blocks). These tiles move independently in a translational (or pure rotational)
model. When using tile-based processing, the relative shifts between data sets is better approximated
by the translational (or rotational) motion models. Furthermore, applying the algorithm to small tiles
reduces the memory requirements of the multiframe reconstruction algorithm. The motion estimation
process involves identifying image tiles from different LR images corresponding to a particular region of
interest. In [11], we introduced a joint estimation technique, in which matching blocks of different LR
frames are optimally detected and registered in a multi-frame joint estimation process. To simplify the
notations, without the loss of generality, all formulas used in this paper correspond to the reconstruction
of a single HR tile (a full image is reconstructed by stitching a set of such HR tiles).
We denote the raster scanned version for each of the K+1 LR image tiles by the vector y
k
. These noisy
LR input image tiles are blurry, translated, and downsampled versions of an unknown high-resolution
image tile denoted by x. The forward model relating these captured image tiles to the unknown high-
resolution tile is given by
y
k
= DHS(v
k
)x +e
k
, (1)
March 18, 2010 DRAFT
Authorized licensed use limited to: Annai Mathammal Sheela Engineering College. Downloaded on June 11,2010 at 01:09:37 UTC from IEEE Xplore. Restrictions apply.
Copyright (c) 2010 IEEE. Personal use is permitted. For any other purposes, Permission must be obtained from the IEEE by emailing [email protected].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
5
in which the vector y
k
represents B B (assumed square without loss of generality) samples of the
captured image y
k
(m

1
, m

2
), where m

1
, m

2
[0, B 1], are ordered as a (B)
2
1 vector. The captured
image is undersampled with respect to an unknown high-resolution image x(m
1
, m
2
), where m
1
, m
2

[0, DB 1], by a factor of D in each dimension. The vector x represents samples of the unknown
DBDB high-resolution image tile x(m
1
, m
2
) similarly ordered. The matrix H represents the blurring
associated with the imaging system. In X-ray imaging, this blurring is due to the geometry of the radiating
illumination as well as the scattering of light in the object material. In each set of tiles, we approximate
this effect by a spatially-invariant point spread function (PSF) h(m
1
, m
2
). Although, the PSF can be
different in different tile sets. The warping operator S(v
k
) of size (DB)
2
(DB)
2
represents the sub-
pixel spatial shifts between similar tiles in the captured images. The spatial shifting is described by the
vector v
k
= [v
k
1
, v
k
2
] for the kth frame. In our model, we assume that these spatial shifts are continuous
values in the range of [D, D]. This corresponds to the range of sub-pixel motions in the captured images.
Without loss of generality, we assume that the tile y
0
denes the coordinate system of the high-resolution
image and hence we only have to estimate the unknown motion parameters for the remaining K images.
The downsampling operator D of size B
2
(DB)
2
captures the undersampling of the detector. Finally,
e
k
of size B
2
1 represents the noise inherent in the analog-to-digital conversion. For our purposes, we
assume this noise to be uncorrelated zero-mean noise with standard deviation .
B. Classic Maximum A-Posteriori Super-Resolution Reconstruction
The general problem of SR is to combine K +1 captured LR images and estimate the high-resolution
image x. An important category of solutions to the SR problem is the maximum a posteriori (MAP)
and closely related methodologies, which is discussed in this section. The MAP methods are based on
the construction of a cost function (), which is the summation of two distinct terms. One is the data
penalty term
d
, which measures the closeness of data to the estimates. The other is the regularization
term
p
, which represents the prior information about the unknown high-resolution (HR) image (x).
Early MAP-based SR methods assumed that the motion vectors were accurately estimated in a separate
process and the noise model was Gaussian [12], [13], which justies the application of quadratic data
penalty terms. As for the regularization term, these techniques most frequently employed the quadratic
Tikhonov style regularization despite its tendency to reduce edge contrast. The resulting cost function is
in the form of
(x) =
K

k=0
y
k
DHS(v
k
)x
2
+x
T
C
1
x
x, (2)
March 18, 2010 DRAFT
Authorized licensed use limited to: Annai Mathammal Sheela Engineering College. Downloaded on June 11,2010 at 01:09:37 UTC from IEEE Xplore. Restrictions apply.
Copyright (c) 2010 IEEE. Personal use is permitted. For any other purposes, Permission must be obtained from the IEEE by emailing [email protected].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
6
where C
1
x
is often a spatial high-pass operator and is the weighting scalar. When C
x
is the exact
covariance of the unknown HR image, then this cost function produces the ideal Wiener lter estimate
of the unknown image. This MAP functional has the advantage of being quadratic, which means that the
estimate image is a linear function of the input measurements and is thus easy to compute.
Through the years, application of more advanced prior functions
p
such as Adaptive Kernel regression
[14] which generalizes popular priors such as Tikhonov and Bilateral Total-Variation (B-TV) [9], have
produced higher quality estimates. For example, the B-TV cost function is dened as

B-TV
(x) =
K

k=0
y
k
DHS(v
k
)x
2
2
+
L

t1,t2=L

|t
1
|+|t
2
|
x S(t)x
1
, (3)
where t = [t
1
, t
2
] is a set of integer pixel shifts and 0 < 1 is a constant [9]. The parameter L denes
the size of the corresponding Bilateral lter kernel. The Bilateral lter and its parameters are extensively
discussed in [15], [9].
These advanced regularization functions are not quadratic (nonlinear estimators) and hence require more
computationally-complex iterative minimization strategies. Such nonquadratic functionals can, however,
preserve many important features of images such as edges. Also, MAP-based robust SR techniques (e.g.
[9], [16], [17]) are able to reduce the effect of outliers such as motion estimation error.
Practical tests show that using a separate motion estimation process, specially in low-SNR cases, is
suboptimal. Therefore, the critical issue of joint SR and motion estimation problem has been the topic
of several papers (e.g. [18], [19], [20], [21], [22]). A simplied MAP formulation of this problem has
the form
(x, v) =
K

k=0
y
k
DHS(v
k
)x
2
+x
T
C
1
x
x, (4)
where v = [v
1
, . . . , v
k
]
T
. Note that, additional priors on motion vector distribution may also be added to
the above cost function [18]. The above approaches are commonly solved in an iterative fashion and are
relatively computationally expensive. While the joint estimation techniques are generally computationally
more complex than robust SR solutions, they are more effective when the number of LR images is small
or when the motion of most LR frames are estimated erroneously. As noted in [20], it is only by jointly
estimating the unknown motion vectors and the aliasing free image that estimators can avoid the bias
associated with registering images containing aliasing artifacts.
C. Problem of Joint Motion and Image Estimation
In this subsection, we briey review an alternative approach for estimating the image shifts between
aliased images using the Variable-Projection principal [23], which we described in detail in our recent
March 18, 2010 DRAFT
Authorized licensed use limited to: Annai Mathammal Sheela Engineering College. Downloaded on June 11,2010 at 01:09:37 UTC from IEEE Xplore. Restrictions apply.
Copyright (c) 2010 IEEE. Personal use is permitted. For any other purposes, Permission must be obtained from the IEEE by emailing [email protected].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
7
publication [7]. While motion-estimation is not the focus of this paper, study of this technique provides
intuition and simplies the material described in later sections of this paper.
Considering the PSF and motion assumptions in Section II-A, we may reverse the order of the shifting
and blur operators in (1) [8] and rewrite the imaging model as
y
k
= DS(v
k
)Hx +e
k
= DS(v
k
)z +e
k
, (5)
where z = Hx is the unknown HR blurry image. The optimization process will then be formulated as
(z, v) =
K

k=0
y
k
DS(v
k
)z
2
2
+z
T
C
1
z
z, (6)
where C
z
is the covariance matrix of the unknown signal z, which is typically assumed to be stationary.
A typical solution to the above problem is the cyclic coordinate-descent method [18], in which in each
iteration one unknown variable is updated based on the estimate of the other unknown variable in the
previous iteration.
D. Efcient Joint Estimation using Variable Projections
Noting that (6) is known in numerical analysis literature as the Separable Non-linear Least Squares
problem [23], in our Variable-Projection technique, we momentarily assume that the non- linear parameters
(motion-vectors) are known. Consequently, the estimate of the set of linear parameters (z) is computed
as
z =
_
Q(v) +C
1
z
_
1
g(v), (7)
where
Q(v) =
1

2
K

k=0
S
T
(v
k
)D
T
DS(v
k
), (8)
g(v) =
1

2
K

k=0
S
T
(v
k
)D
T
y
k
. (9)
We plug the parametric estimate of the blurry HR image ( z) into the MAP functional (6) and after some
algebraic simplications, we get a new (maximization) cost function that only relies on the motion-vectors:
(v) = g(v)
T
_
Q(v) +C
1
z
_
1
g(v). (10)
Note that, unlike the cyclic coordinate-descent method, we require no iterations between the sets of
parameters since we do not explicitly calculate (7). Indeed, a direct approach to maximize (10) involves
inverting a large matrix of size (DB)
2
(DB)
2
which is computationally challenging for even small
March 18, 2010 DRAFT
Authorized licensed use limited to: Annai Mathammal Sheela Engineering College. Downloaded on June 11,2010 at 01:09:37 UTC from IEEE Xplore. Restrictions apply.
Copyright (c) 2010 IEEE. Personal use is permitted. For any other purposes, Permission must be obtained from the IEEE by emailing [email protected].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
8
image tiles. In [7], we described a series of numerical tricks to speed up the process. One trick is solving
the problem in the Fourier domain and taking advantage of the spectral folding phenomenon in aliased
images.
To simplify the derivation, we reformulate the image capture model (19) in the Fourier domain. We
use the over-script to denote the Fourier domain representation. For example, the kth LR image
is given by y
k
(m

1
, m

2
), where the LR spatial frequencies are indexed by m

1
and m

2
according to

i
= e
j2m

i
B
= f
m

i
. The f

= e
j2
B
term is the base of the discrete fourier transform of the observed
image y
k
(m

1
, m

2
). Similarly, the high-resolution spatial frequency coordinates are indexed by m
1
and
m
2
, where
i
= e
j2m
i
DB
= f
m
i
.
Because the shift S operator is spatially-invariant, its Fourier representation is a diagonal matrix dened
as

S(v
k
) = diag
_
f
m1vk
1
+m2vk
2
_
. (11)
The downsampling operator D is not spatially-invariant and so its Fourier representation is not diagonal.
The downsampling operator is, however, periodic and is conveniently represented by

D = (1
T
D
I
B
) (1
T
D
I
B
), (12)
where represents the Kronecker matrix product, 1
D
represents a [D 1] vector of all ones, and I
B
represents the identity matrix of B dimension. In the Fourier domain, the vector y
k
corresponds to
samples of the spectrum of the captured image y
k
(m

1
, m

2
).
The form of matrix

D justies a decoupling of the observed spatial frequency components in the
Fourier domain. In other words, we can consider each LR spatial frequency component (indexed by m

1
and m

2
) as an independent observation model given by
y
k
(m

1
, m

2
) = 1
T
D
2

S
m

1
,m

2
(v
k
) z
m

1
,m

2
+ e
k
(m

1
, m

2
), (13)
where the D
2
D
2
matrices are constructed as

S
m

1
,m

2
(v
k
) = diag
_
f
m

1
v
k
1
+m

2
v
k
2
, f
(m

1
+B)v
k
1
+m

2
v
k
2
, . . . , f
(m

1
+(D1)B)v
k
1
+(m

2
+(D1)B)v
k
2
_
(14)
and the vectors are constructed according to
z
m

1
,m

2
=
_
z(m

1
, m

2
), z(m

1
+B, m

2
), . . . , z(m

1
+ (D 1)B, m

2
+ (D 1)B)

T
.
A single spatial frequency in the captured image is a function of the original signal content and summation
of the D
2
1 aliased spectral components. This demonstrates that the reconstruction can be applied to
March 18, 2010 DRAFT
Authorized licensed use limited to: Annai Mathammal Sheela Engineering College. Downloaded on June 11,2010 at 01:09:37 UTC from IEEE Xplore. Restrictions apply.
Copyright (c) 2010 IEEE. Personal use is permitted. For any other purposes, Permission must be obtained from the IEEE by emailing [email protected].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
9
each collection of aliased spectral components independently. Thus, a singlet set of D
2
high-resolution
spatial frequency components z
m

1
,m

2
can be estimated as

z
m

1
,m

2
=
_

Q
0,0
(v) +

C
1
z,m

1
,m

2
_
1
g
m

1
,m

2
(v), (15)
where

Q
0,0
(v) =
1

2
K

k=0

S
T
0,0
(v
k
)1
D
21
T
D
2

S
0,0
(v
k
), (16)

C
z,m

1
,m

2
= diag
_

Z(m

1
, m

2
),

Z(m

1
+B, m

2
), . . . ,

Z(m

1
+ (D 1)B, m

2
+ (D 1)B)
_
,
g
m

1
,m

2
(v) =
1

2
K

k=0

S
T
m

1
,m

2
(v
k
)1
D
2 y
k
(m

1
, m

2
), (17)
in which

Z(m
1
, m
2
) are samples of the signals power spectral density (PSD) function.
Finally, the motion estimation function of (10) simplies to
(v)=

1
,m

2
g
m

1
,m

2
(v)
T
_

Q
0,0
(v) +

C
1
z,m

1
,m

2
_
1
g
m

1
,m

2
(v). (18)
Estimating the motion vectors using (18) and the high-resolution image using (15) is signicantly faster
than using the direct matrix form of (10) and (7). The simplied form requires inverting B
2
small matrices
of size D
2
D
2
as opposed to inverting one very large D
2
B
2
D
2
B
2
matrix of (7). A set of similar
acceleration techniques was described in [7].
So far, ignoring the PSF effects, we have studied a computationally efcient multi-frame joint motion
estimation and SR technique in the presence of aliasing. In the next subsection, ignoring the aliasing
effects, we study an efcient single frame restoration (deblurring and denoising) method. We combine
these two techniques in Section III.
E. Single Frame Deblurring using ForWarD Algorithm
In this section, we consider the problem of restoring the contrast lost due to the blurring (PSF)
inherent to the imaging system. To achieve this, we briey review the fast two-step ForWarD deblurring
algorithm [5]. The ForWarD algorithm combines a Fourier-based regularized deconvolution algorithm
with a wavelet-based denoising post processing step. As described in the following, in the rst step, a
Fourier domain implementation of the Wiener lter reduces blur artifacts while suboptimally magnifying
the noise. In the second step, a wavelet-based denoising process reduces the noise artifacts.
The basic ForWarD algorithm addresses the problem of restoring an unknown image signal, which has
been blurred by a spatially-invariant operator such as a point spread function (PSF) and corrupted by a
March 18, 2010 DRAFT
Authorized licensed use limited to: Annai Mathammal Sheela Engineering College. Downloaded on June 11,2010 at 01:09:37 UTC from IEEE Xplore. Restrictions apply.
Copyright (c) 2010 IEEE. Personal use is permitted. For any other purposes, Permission must be obtained from the IEEE by emailing [email protected].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
10
stationary noise distribution as in the forward model
z
e
= Hx +e. (19)
The noise is assumed to have a stationary distribution. The rst step of the ForWarD algorithm involves
inverting the spatially-invariant blurring operator H using a regularized sharpening lter such as the
Wiener lter R [5], [24]. Such estimation process is formulated as
x = Rz
e
(20)
R =
_
H
T
C
1
e
H+C
1
x
_
1
H
T
, (21)
where C
e
is the covariance matrix of the stationary measurement noise (typically C
e
=
2
I) and
is a weighting factor [25]. For example, in the case of the normal and low-dosage images shown in
Fig. 1, the noise standard deviation corresponds to about 20 percent and 65 percent of the maximum
signal intensity, respectively. Indeed, under certain conditions the Wiener lter can be regarded as a
Tikhonov regularization functional [26], which was described in Section II-B. While in the common
Wiener lter implementation = 1, in ForWarD algorithm, weighting factor is chosen such that < 1.
Such perceivably suboptimal choice of the weighting factor often enhances sharpness at the expense of
substantial noise amplication.
The Wiener lter can be implemented efciently in the Fourier domain using Fast Fourier Transforms
(FFT). Implementation of the Wiener lter in the Fourier domain is founded on the assumption that the
blurring operator represented by H is spatially-invariant and that both the noise and signal are stationary
random processes. In this case, the blurring operator can be represented by its transfer function

H(m
1
, m
2
)
computed as the Fourier transform of the systems PSF. Also, the signal statistics are represented by a
power spectral density function (PSD)

X(m
1
, m
2
). The application of the Wiener lter is performed in
the Fourier domain producing an estimate of the original signal spectrum x according to

x(m
1
, m
2
) =

H

(m
1
, m
2
) z
e
(m
1
, m
2
)
|

H(m
1
, m
2
)|
2
+

2

X(m1,m2)
. (22)
This estimate is converted back into the spatial domain using FFT operators to obtain the estimate
x(m
1
, m
2
).
The second step in the ForWarD process applies a nonlinear denoising lter to the wavelet transform of
the sharpened image ( x) for the purpose of eliminating the residual noise amplied by the restoration lter
( e = Re). This nonlinear denoising step is based on adaptive thresholding in the wavelet domain. The
redundant (no downsampling) wavelet transform [27] is applied to the sharpened image by convolving
March 18, 2010 DRAFT
Authorized licensed use limited to: Annai Mathammal Sheela Engineering College. Downloaded on June 11,2010 at 01:09:37 UTC from IEEE Xplore. Restrictions apply.
Copyright (c) 2010 IEEE. Personal use is permitted. For any other purposes, Permission must be obtained from the IEEE by emailing [email protected].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
11
the image with a set of scaling and wavelet functions, represented by the matrices
i
and
l
, producing
a set of scale coefcient images s
i
and wavelet coefcient images w
l
. Reducing the wavelet coefcients,
or wavelet shrinkage, eliminates the noisy artifacts introduced during the deblurring step according to
w
l
(m
1
, m
2
) = w
l
(m
1
, m
2
)
1
1 +

2
l
(m
1
,m
2
)

2
l
(m
1
,m
2
)
, (23)
where (m
1
, m
2
) identies the pixel location,
2
l
(m
1
, m
2
) represents the noise variance at the lth wavelet
space for the (m
1
, m
2
) pixel, and
2
l
(m
1
, m
2
) is the wavelet signal power. After shrinking the wavelet
coefcients in this manner, the inverse wavelet transform produces the nal denoised estimate of the
original image x. We refer the reader to the original work of [5] for a more complete explanation of this
process including visualization of the images at the various steps in the algorithm.
Conceptually similar to the Wiener lter, the wavelet shrinkage reduces the observed wavelet coef-
cients as a function of the local SNR of the wavelet coefcients. The performance of the wavelet shrinkage
depends on the ability to predict the SNR for the wavelet coefcients. Because of the wavelets spatial
locality property, the wavelet shrinkage provides efcient local, signal-dependent denoising.
The value of the noise variance for each wavelet function is given by the covariance of the residual
noise e in the wavelet domain. The covariance matrix of the noise in the wavelet domain is given by

T
l

C
e

l
. Because the Wiener lter is spatially-invariant, the residual noise covariance

C
e
is circulant.
In other words, the residual noise is stationary due to the spatial invariance of the Wiener lter. Since the
redundant wavelet lter [27] is also spatially-invariant, the residual noise power in a particular wavelet
space is uniform over the entire image [5]. The residual noise power in the wavelet domain is computed
efciently in the Fourier domain by way of

2
l
=
2
B

m
1
,m
2
=0
|

l
(m
1
, m
2
)|
2
|

H(m
1
, m
2
)|
2
_
|

H(m
1
, m
2
)|
2
+

2

X(m
1
,m
2
)
_
2
. (24)
This must be computed once for each wavelet lter used in the thresholding.
The accuracy of the wavelet thresholding depends on accurate estimates of the local signal statistics
in the wavelet domain. In practice, this signal information must be estimated from the captured image
data. In [5], the authors propose a simple process for estimating the signal power term
2
l
(m
1
, m
2
). The
authors estimate the wavelet signal power by rst very coarsely denoising the sharpened image x. They
propose a simple hard thresholding wavelet denoising approach to obtain the coarsely denoised image
using a different set of scaling and wavelet functions

i
and wavelet

l
than those used for the soft
thresholding. We denote the wavelet coefcient images for this different wavelet transform by w

l
. A hard
March 18, 2010 DRAFT
Authorized licensed use limited to: Annai Mathammal Sheela Engineering College. Downloaded on June 11,2010 at 01:09:37 UTC from IEEE Xplore. Restrictions apply.
Copyright (c) 2010 IEEE. Personal use is permitted. For any other purposes, Permission must be obtained from the IEEE by emailing [email protected].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
12
thresholding applied to the wavelet and scaling coefcients according to

l
(m
1
, m
2
) =
_
_
_
w

l
(m
1
, m
2
), |w

l
(m
1
, m
2
)| >
l
0, |w

l
(m
1
, m
2
)|
l
, (25)
provides the coarsely denoised image. The term is the input threshold ([28] describes the choices for ).
The inverse wavelet transform applied to the hard thresholded wavelet coefcients produces the coarsely
denoised image

x

(we refer the interested reader to [5] for more information).

Next, the wavelet transform using the original wavelet function
i
and wavelet
l
is applied to the
coarsely denoised image

x

to produce the coefcients w

l
,
w
l
=
l

x

. (26)
The spatially-varying standard deviation of the signals wavelet coefcients is estimated to be the value
of the coefcients of the coarsely denoised image, or

l
(m
1
, m
2
) = w
l
(m
1
, m
2
). (27)
This estimate of the signal power is used in (23). The ForWarD approach to deconvolution has been
applied to several deconvolution problems with success and has been used as a benchmark for evaluating
the success of other single frame deconvolution algorithms[14]. In the next section, we propose a multi-
frame extension of this algorithm, which considers the aliasing and issues inherent to the SR problem.
III. MULTIFRAME IMAGE RESTORATION AND WAVELET DENOISING
The goal of the multiframe SR problem, aside from reducing the aliasing artifacts, is restoring the
contrast lost due to the blurring inherent to the imaging system. To achieve this, we derive a novel
multiframe variant of the fast two-step ForWarD method [5]. The authors of the original ForWarD
algorithm have recently proposed a new version of this algorithm that addresses the multi-frame deblurring
problem [24] for non-aliased images with stationary noise model. In this section, we introduce an
extension of this multiframe algorithm, which considers the non-stationarity inherent to the SR problem.
Figure 2 presents a ow chart representation of this multiframe ForWarD (MForWard) SR process.
A. MForWarD
The efciency of the original ForWarD algorithm [5], as well as the more recent multiframe version
[24], is founded on the stationarity of the noise as well as the spatial invariance of the blurring operators.
The stationarity assumption breaks down in the case of multiframe SR.
March 18, 2010 DRAFT
Authorized licensed use limited to: Annai Mathammal Sheela Engineering College. Downloaded on June 11,2010 at 01:09:37 UTC from IEEE Xplore. Restrictions apply.
Copyright (c) 2010 IEEE. Personal use is permitted. For any other purposes, Permission must be obtained from the IEEE by emailing [email protected].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
13
Fig. 2. A ow chart representation of the MForWard algorithm.
In the case of SR, we must deconvolve the collection of blurry and aliased images onto a higher
resolution sampling grid. The multiframe Wiener lter producing a sharp estimate of x is a variant of
(21) given by
x = B
1
( v)H
T
g( v), (28)
where
B( v) = H
T
Q( v)H+C
1
x
(29)
and Q and g are dened in (8) and (9). Here, we assume that the motion vectors ( v) are estimated from
(10). Similar to the ForWarD algorithm, we typically use values of < 1, which tend to sharpen the
March 18, 2010 DRAFT
Authorized licensed use limited to: Annai Mathammal Sheela Engineering College. Downloaded on June 11,2010 at 01:09:37 UTC from IEEE Xplore. Restrictions apply.
Copyright (c) 2010 IEEE. Personal use is permitted. For any other purposes, Permission must be obtained from the IEEE by emailing [email protected].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
14
images at the expense of increased noise amplication and ringing artifacts in the resolution-enhanced
image x. After applying the multiframe resolution enhancement lter of (28), the covariance matrix of
the residual noise error e is
C
e
= B
1
( v)H
T
Q( v)HB
1
( v). (30)
Effective denoising and artifact removal using the wavelet shrinkage method requires accurate estimates
of the residual noise power in the different wavelet lter domains. The noise covariance matrix in the
wavelet coefcient domain of the wavelet function
l
is given by
T
l
C
e

l
. In the case of the stationary
model, the residual noise power
2
l
is constant along the diagonal of this covariance matrix. Estimating
this single value is computed efciently in (24) using FFT operations. Unlike the standard ForWarD
algorithm, the multiframe estimate of the deblurred image x has a residual random error eld which is
not stationary. In the case of multiframe reconstruction, ignoring the border pixels, the residual noise
eld e is periodic with a period of D. For each wavelet lter, we must compute the residual noise
powers {
2
l
}
d
1
,d
2
along the diagonal of
T
l
C
e

l
which correspond to the D
2
HR grid locations. We
use (d
i
{1, 2, ..., D}, i {1, 2}) to index the HR sampling locations (note Fig. 3).
The spatially-varying residual noise power depends on the collection of motion vectors v. That is,
even though the noise elds of the captured LR images e
k
are stationary, different pixels in the high-
resolution reconstructed image x have varying amounts of data. Because of this, the residual noise powers

2
l
(m
1
, m
2
) in the wavelet domain are not uniform and cannot be computed using (24).
To see this effect more clearly, we present a simple example. Suppose that we capture twelve LR
images with a downsampling factor of D = 4 in both the vertical and horizontal dimensions. Furthermore,
suppose that the captured images are offset by integer numbers of pixels in the high-resolution grid with
the number of offsets per HR grid location shown in Fig. 3. To simplify the analysis, we assume that
the signal covariance matrix is given by C
x
=
1

I and that the imaging system is free of blur (H = I).

In this case, after applying the multiframe Wiener lter, the covariance matrix of the residual noise eld
C
e
is a diagonal matrix. The terms along the diagonal correspond to the residual noise variance in the
reconstructed image. The HR pixels in the sharpened image x lacking LR measurements will have much
higher noise variance as indicated on the right side of Fig. 3. Indeed, in the more general case, the motion
vectors do not fall perfectly onto grid locations.
Once we obtain the D
2
spatially-varying noise powers, we employ the same pattern of the traditional
ForWarD algorithm using the coarse denoising by the hard thresholding of (25) to estimate the wavelet
coefcient energies followed by the soft thresholding of (23). The only difference being that when we
March 18, 2010 DRAFT
Authorized licensed use limited to: Annai Mathammal Sheela Engineering College. Downloaded on June 11,2010 at 01:09:37 UTC from IEEE Xplore. Restrictions apply.
Copyright (c) 2010 IEEE. Personal use is permitted. For any other purposes, Permission must be obtained from the IEEE by emailing [email protected].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
15
0 0 0 0
0 0 0 0
2 4 1 1
1 1 1 1
0.5 0.25 1 1
1 1 1 1
inverse
d
2
d
1
d
2
d
1
Fig. 3. The grid on the left shows the 44 grid of HR image sample locations and the number of LR measurements (assuming
integer sampling offsets) per sample location (no blur case as in the shift-and-add image reconstrution described in [9]). In
the sharpened image x, the locations with fewer measurements will have higher residual noise variance. For example, the grid
on the right shows the residual noise variance for an image reconstructed using an unregularized inverse of the system. The
locations without any measurements have innite noise variance.
apply the hard or soft thresholding of (23) and (25), we do not use the same noise variance for every
pixel. Instead, we use the spatially-varying noise powers corresponding to the HR grid locations. In this
way, we incorporate the non-stationary SNR properties of the sharpened image x when performing the
wavelet denoising. The next subsection explains a computationally efcient approach to this issue.
B. Efcient Fourier MForWarD
One key advantage of the original ForWarD algorithm is its efcient implementation by way of FFT
operations. Using FFTs to compute (22) and (24) eliminates the need to explicitly construct the extremely
large matrices to deconvolve the image and to estimate the residual noise variances. The MForWarD
algorithm must also support similar computational efciency to have any practical value. For example,
computing the covariance matrix in (30) directly is computationally prohibitive due to the size of the
images. We now describe an efcient implementation of the MForWarD algorithm which leverages FFT
operations, analogous to the method used in the case of the stationary ForWarD approach.
We operate in the the Fourier domain as we did in Sec. II-C, where we originally dened many of
the matrices. The only additional matrix is that of the blur operator H which is spatially-invariant and
hence is diagonal in the Fourier domain

H = diag{

H(m
1
, m
2
)}. (31)
In the case of multiframe Wiener ltering, we again consider each spatial frequency component indexed
by m

1
and m

2
as an independent observation model given by
y
k
(m

1
, m

2
) = 1
T
D
2

H
m

1
,m

S
m

1
,m

2
(v
k
) x
m

1
,m

2
+ e
k
(m

1
, m

2
), (32)
March 18, 2010 DRAFT
Authorized licensed use limited to: Annai Mathammal Sheela Engineering College. Downloaded on June 11,2010 at 01:09:37 UTC from IEEE Xplore. Restrictions apply.
Copyright (c) 2010 IEEE. Personal use is permitted. For any other purposes, Permission must be obtained from the IEEE by emailing [email protected].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
16
where

H
m

1
,m

2
= diag
_

H(m

1
, m

2
),

H(m

1
+B, m

2
), . . . ,

H(m

1
+ (D 1)B, m

2
+ (D 1)B)
_
x
m

1
,m

2
=
_
x(m

1
, m

2
), x(m

1
+B, m

2
), . . . , x(m

1
+ (D 1)B, m

2
+ (D 1)B)

T
.
A single spatial frequency in the captured image is a function of the original signal content and summation
of the D
2
1 aliased spectral components. This demonstrates that the reconstruction can be applied to
each collection of aliased spectral components independently.
Estimates of the D
2
spatial frequency components in the high-resolution image from the multiple
measurements of the observed spatial frequencies

1
,

2
are obtained via

x
m

1
,m

2
=

B
1
m

1
,m

2
(v)

H
T
m

1
,m

2
g
m

1
,m

2
(v), (33)
where

B
m

1
,m

2
(v) =

H
T
m

1
,m

Q
0,0
(v)

H
m

1
,m

2
+

C
1
x,m

1
,m

2
,

C
x,m

1
,m

2
= diag
_

X(m

1
, m

2
),

X(m

1
+B, m

2
), . . . ,

X(m

1
+ (D 1)B, m

2
+ (D 1)B)
_
and

Q is dened in (8). As in Section II-C, estimating the high-resolution image in this fashion is
signicantly faster than using the direct matrix form of (28). Implementing the multiframe Wiener lter
in the Fourier domain requires inverting B
2
small matrices of size D
2
D
2
as opposed to inverting one
very large D
2
B
2
D
2
B
2
matrix of (28). In a practical sense, any images larger than 200 by 200 pixels
would require inordinate (for todays machines) amounts of processing to invert the B matrix if using
the direct matrix formulation. This approach enables efcient implementation of a multiframe resolution
enhancement in the Fourier domain. After estimating the sharpened image

x, the image is converted
back into the spatial domain using an inverse FFT to obtain the multiframe ltered image in the spatial
domain x.
We can also apply a similar technique to accelerate the calculation of the D
2
residual noise powers in
the wavelet spaces
2
l
(d
1
, d
2
), which are required for the wavelet denoising step. Because the residual
noise eld is not stationary, its statistics are not completely characterized by a power spectral density
function. The residual noise spectral components are correlated. The D
2
D
2
covariance matrix of the
residual noise eld spectral components associated with the LR spatial frequency set indexed by m

1
, m

2
is

C
e,m

1
,m

2
=

B
1
m

1
,m

2
(v)

H
T
m

1
,m

Q( v)

H
m

1
,m

B
1
m

1
,m

2
(v). (34)
March 18, 2010 DRAFT
Authorized licensed use limited to: Annai Mathammal Sheela Engineering College. Downloaded on June 11,2010 at 01:09:37 UTC from IEEE Xplore. Restrictions apply.
Copyright (c) 2010 IEEE. Personal use is permitted. For any other purposes, Permission must be obtained from the IEEE by emailing [email protected].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
17
To estimate the residual noise power for a given wavelet lter
l
at a particular grid location d
1
, d
2
, we
compute

2
l
(d
1
, d
2
) =

1
,m

2
Tr
_

T
l,m

1
,m

S
T
m

1
,m

2
(d)

C
e,m

1
,m

S
m

1
,m

2
(d)

l,m

1
,m

2
_
, (35)
where d = [d
1
, d
2
]
T
, Tr represents the trace operator, and the diagonal matrix

l,m

1
,m

2
= diag
_

(m

1
, m

2
),

(m

1
+B, m

2
), . . . ,

(m

1
+ (D 1)B, m

2
+ (D 1)B)
_
(36)
represents the samples of the wavelet lter spectral response. Equation 35 provides an efcient method
for computing the residual noise powers in the wavelet domain for use in wavelet denoising. This step
must be performed twice; once for the soft thresholding wavelet functions
l
and once for the hard
thresholding wavelet set

l
.
IV. EXPERIMENTAL RESULTS
In this section, we perform two sets of experiments demonstrating the capability of the MForWarD
algorithm. The rst section describes experiments using simulated general image data. These results
demonstrate the broad applicability of the MForWarD algorithm. The second section describes some
experimental results using real data captured by a digital mammogram system on a phantom breast.
A. Simulation-based experiments
For the rst set of experiments, we construct a simple simulation example which highlights the
advantages of the MForWarD algorithm for general imaging. We construct a simulated data sequence by
downsampling the 400 by 400 pixel portion of a high-resolution shown in Fig. 4(a) using the sub-sample
location shown in Fig. 3. We show only a cropped portion of the image to highlight the detail in the
experiment. We simulate the optical blur using a simple heavy-tailed point spread function of the form
h(m
1
, m
2
) =
1
c
e

m
2
1
+m
2
2
, (37)
where the term controls the rate of decay of the PSF, and c is the normalizing constant. We use a
value of 1.2 pixels. We also added noise to the LR images to create an effective SNR of about 26 dB.
An example of the simulated LR image is shown in Fig. 4(b).
We reconstructed the HR image using three different approaches representing the different classes of
SR algorithms. In all these simulated cases, the motion was assumed to be perfectly known. The rst
method is the completely linear approach embodied by the multiframe Wiener lter of (28), or the Fourier
March 18, 2010 DRAFT
Authorized licensed use limited to: Annai Mathammal Sheela Engineering College. Downloaded on June 11,2010 at 01:09:37 UTC from IEEE Xplore. Restrictions apply.
Copyright (c) 2010 IEEE. Personal use is permitted. For any other purposes, Permission must be obtained from the IEEE by emailing [email protected].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
18
a: Original Image b: Captured Image
c: Linear, strong prior d: Linear, weak prior
Fig. 4. Comparison of linear SR techniques for increasing the resolution of 12 simulated LR frames by a factor of D = 4.
implementation of (33). This is perhaps the least computationally complex approach to multiframe SR.
Using (33) produces an estimate of the HR image in a single, non-iterative step and requires inverting
several very small matrices. The drawback to this linear approach, however, is that the regularization
of the multiframe linear lter trades off sharpness in the nal image for noise reduction. For example,
Fig. 4(c) shows an example of the linear reconstruction when using a power spectral density (PSD)
function of the form

X(m
1
, m
2
) =

2
|m
1
|
1
+|m
2
|
2
, (38)
with = 1.2. The regularization weighting of (33) is = 6. The reconstructed image shows poor contrast
to maintain minimal noise amplication. Alternatively, if we apply the linear reconstruction with a weaker
prior ( = 0.02), the reconstructed image shows improved contrast restoration at the expense of noise
amplication as seen in Fig. 4(d). This is the classic tradeoff inherent to linear restoration algorithms.
The black curve of Fig. 5 shows the PSNR tradeoff as a function of regularization weighting strength.
For a small , the PSNR is quite poor due to weak regularization of the poorly conditioned system.
March 18, 2010 DRAFT
Authorized licensed use limited to: Annai Mathammal Sheela Engineering College. Downloaded on June 11,2010 at 01:09:37 UTC from IEEE Xplore. Restrictions apply.
Copyright (c) 2010 IEEE. Personal use is permitted. For any other purposes, Permission must be obtained from the IEEE by emailing [email protected].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
19
0 1 2 3 4 5
22
22.5
23
23.5
24
24.5
Regularization Weighting ()
P
S
N
R
Experimental PSNR versus Regularization Weighting


Linear Regularization
MForWarD,=0.3
MForWarD,=0.9
MForWarD,=1.5
Fig. 5. The solid curve shows the PSNR performance versus regularization using standard linear regularization. The optimal
weighting is around = 4.5 for the linear regularization. The dashed curves compare the PSNR performance versus regularization
for the MForWarD algorithm using different thresholding values . The MForWarD algorithm shows superior peak PSRN
performance over the linear regularization.
When increases beyond 4.5, the PSNR slowly degrades as the estimate becomes overly smoothed. If
the signals PSD was perfectly dened by (38), then the PSNR-maximizing weighting parameter would be
near = 1. The dashed curves show PSNR versus regularization weighting for the MForWarD algorithm
using different thresholding parameters . While the performance varies considerably for large values of
, the peak PSNR near = 1 for the MForWarD algorithm shows reasonable stability with respect to
the thresholding parameter . When we apply our MForWarD algorithm, we employ 2-tap Daubechie
lters for the the soft thresholding wavelet functions and 6-tap Daubechie lters for the coarse denoising
by way of hard wavelet coefcient thresholding.
Figure 6 compares the PSNR-optimal parameter settings for three different algorithms. Fig. 6(a) shows
the PSRN-optimal linear regularization setting, which has a PSNR of 23.90 dB at = 4.5. The image
shows a reasonable balance between contrast and noise gain, but still maintains a considerable amount
of noise in the at sky region. Fig. 6(b) shows the PSNR-optimal B-TV algorithm of (3) having 23.96
dB. In this experiment, L = 2, = 0.7, and = 0.0003. The algorithm does a reasonably good job
of preserving contrast while eliminating the noise in the at regions. Fig. 6(c) shows the MForWarD
algorithm at 24.3 dB using a thresholding value of = 1.1. The image preserves much of the contrast
while eliminating the noise.
March 18, 2010 DRAFT
Authorized licensed use limited to: Annai Mathammal Sheela Engineering College. Downloaded on June 11,2010 at 01:09:37 UTC from IEEE Xplore. Restrictions apply.
Copyright (c) 2010 IEEE. Personal use is permitted. For any other purposes, Permission must be obtained from the IEEE by emailing [email protected].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
20
a: Linear Regularization b: B-TV
c: MForWarD
Fig. 6. Comparison of image quality for different regularization techniques using PSNR-optimal settings. (a) Linear
regularization (PSNR=23.90 dB), (b) B-TV (PSNR=23.96 dB), and (c) MForWarD (PSNR=24.3 dB).
One advantage of the proposed Fourier-wavelet SR algorithm is the minimal computational overhead
required for the wavelet denoising. Running on an Intel Core-2-Duo 2.2 GHz processor, the Fourier
restoration requires 4.3 seconds of computation time. Performing the wavelet-based denoising requires
an additional 2.3 seconds for a total of about 6.6 seconds. In contrast, the B-TV algorithm, representative
of the large class of iterative techniques, requires only 0.5 seconds per iteration, but requires at least 60
steepest descent iterations (30 seconds) before approaching a limiting image quality.
B. Real X-ray data experiments
In this section, we apply our multiframe reconstruction and restoration algorithm to real images
captured on an experimental X-ray imaging system. Our experimental imaging system is based on
a Mammomat NovationTOMO digital mammography prototype system (Siemens Medical Solutions,
March 18, 2010 DRAFT
Authorized licensed use limited to: Annai Mathammal Sheela Engineering College. Downloaded on June 11,2010 at 01:09:37 UTC from IEEE Xplore. Restrictions apply.
Copyright (c) 2010 IEEE. Personal use is permitted. For any other purposes, Permission must be obtained from the IEEE by emailing [email protected].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
21
Erlangen Germany)
3
, stationed at the Duke University Medical Center. The system uses a stationary
selenium-based detector of 85 m pixels. Pixels with this size correspond to a Nyquist sampling rate of
5.6 line pairs per millimeter (lp/mm). We use a CIRS model 11A breast phantom (CIRS Inc., Norfolk
VA) to test our SR algorithms. We introduce shifts in the image by two methods. First, we allow the
x-ray tube to rotate by 1 degree. Second, we manually move the breast phantom to introduce motion
into the system. This manual motion is completely uncontrolled. Our dataset consists of 15 frames at
the low dosage level of 11.3 mAs at 28 kVp tube voltage. As a point of reference, we also acquire a
single frame at a more typical dosage of 226 mAs at 28 kVp tube voltage (Fig. 1). The breast phantom
includes several testing features including a pair of resolution bar charts. We focus on the results of the
test resolution chart to explore the contrast performance of the multiframe imaging system. We apply
our algorithm to 100 100 pixel tiles in the captured image to estimate 400 400 pixel high-resolution
images (enhancement D = 4).
We modeled our system PSF as a heavy-tailed exponential energy distribution with = 1.5. To get a
measure of the PSNR, we calculated the standard deviation in a textureless region of the phantom. We
also measured the difference in grayscale values between for the registration bars in the resolution chart
to get an approximate PSNR value of 3 dB. We t the terms of (38) to the periodogram obtained by
averaging the spectral energy distribution over the collection of LR images. We employed 2-tap Daubechie
lters for the the soft thresholding wavelet functions and 6-tap Daubechie lters for the coarse denoising
by way of hard wavelet coefcient thresholding.
Figure 7 shows a scatter plot of the set of estimated motions { v
k
} on the HR image grid. The grid
reects the number of image samples per pixel in the reconstructed image. The example shows some
pixel estimates combining as many as four measurements, whereas other pixels have no measurements
underscoring the spatial variability of the residual noise variance.
Fig. 8 gives a visual example of the SNR for an image obtained by increasing the radiation of a single
exposure versus the SNR after combining multiple low-exposure images. We focus on the portion of the
resolution chart beyond the Nyquist rate for the imaging system (5.6 lp/mm). The numbers indicate the
resolution in terms of line pairs per millimeter (lp/mm). The rst image Fig. 8(a) shows an example of
an image captured at a typically high radiation dosage of 226 mAs. The bar targets clearly show aliasing
3
Caution: Investigational Device. Limited by US Federal law to investigational use. The information about this product is
preliminary. The product is under development and is not commercially available in the US; and its future availability cannot
be ensured.
March 18, 2010 DRAFT
Authorized licensed use limited to: Annai Mathammal Sheela Engineering College. Downloaded on June 11,2010 at 01:09:37 UTC from IEEE Xplore. Restrictions apply.
Copyright (c) 2010 IEEE. Personal use is permitted. For any other purposes, Permission must be obtained from the IEEE by emailing [email protected].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
22
1
m
2
m
Translation Estimates
4
3
2
1
4 3 2 1
Fig. 7. The scatter plot shows the set of estimated motions { v
k
} on the HR image grid. Some high-resolution pixels have as
many as four measurements whereas others have none underscoring the need for a spatially-varying denoising approach.
artifacts. The second image Fig. 8(b) shows the same portion of the resolution chart captured at a much
lower dosage (11.3 mAs). The image demonstrates the extremely poor SNR of the captured images at such
low illuminating radiation. The third image Fig. 8(c) shows the result of averaging the motion compensated
LR frames without enhancing the resolution or sampling rate of the system. Interestingly, the multiframe
average appears to have an approximately equivalent SNR to the single frame captured at a normal
radiation dosage, although it contains some blur as a result of averaging frames with subpixel sampling
offsets. The fourth image Fig. 8(d) shows the reconstructed image z after registering the collection of
images using the multiframe algorithm described in Section II. The reconstructed image shows a restored
resolution above the Nyquist rate inherent to the detector. The image is, however, still noisy and has low
contrast. The effective SNR seems comparable to that of the single image captured under high dosage
Fig. 8(a).
Figure 9 shows the resulting images after applying the sharpening and denoising steps of Section III.
The left image Fig. 9(a) shows the result x after applying the multiframe Wiener sharpening lter of (28).
The image shows improved contrast with increased sharpness, but also amplied noise. The image Fig.
9(b) shows the result after the traditional ForWarD algorithm applied to the Wiener sharpened image Fig.
9(a). The hard threshold value was chosen to be = 6. The resulting image preserves the contrast around
the bar chart signal locations while eliminating much of the noise in the signal-free portions of the image.
The image still contains some residual noise. The nal image Fig. 9(c) shows the result after applying
March 18, 2010 DRAFT
Authorized licensed use limited to: Annai Mathammal Sheela Engineering College. Downloaded on June 11,2010 at 01:09:37 UTC from IEEE Xplore. Restrictions apply.
Copyright (c) 2010 IEEE. Personal use is permitted. For any other purposes, Permission must be obtained from the IEEE by emailing [email protected].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
23
a: High dosage LR b: Low dosage LR
c: Average d: z
Fig. 8. (a) High dosage image (226 mAs) (b) Low-dosage image (11.3 mAs) (c) Motion compensated average of LR frames
(no resolution enhancement). (d) Multiframe reconstruction image z of (7).
the MForWarD algorithm including the non-stationary noise power computations. The hard threshold for
the MForWarD algorithm was also = 6. Including the spatially-varying noise powers improves the
noise removal over the basic ForWarD algorithm Fig. 9(b).
To get an another perspective on the effects of the multiframe restoration and denoising, we plot slices
through the resolution test chart region as indicated in Fig. 10 (top). Fig. 10 shows slices through several
images. The top curve shows the slice through the average of the captured images containing only subpixel
motion. The slice shows some aliasing as well as lost contrast for the bars about the Nyquist sampling
rate of 5.6 lp/mm. The second graph shows a slice through the multiframe reconstructed image z. The
reconstruction eliminates the aliasing artifacts and effectively restores contrast beyond the sampling rate
of the detector. The signal strength above 8 lp/mm, however, is very weak due to the blurring inherent to
March 18, 2010 DRAFT
Authorized licensed use limited to: Annai Mathammal Sheela Engineering College. Downloaded on June 11,2010 at 01:09:37 UTC from IEEE Xplore. Restrictions apply.
Copyright (c) 2010 IEEE. Personal use is permitted. For any other purposes, Permission must be obtained from the IEEE by emailing [email protected].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
24
a: x b: Basic ForWarD x
c: MForWarD x
Fig. 9. Different restoration techniques applied on the low-dosage sequence illustrated in Fig. 8(b). (a) Multiframe sharpened
image x (b) Basic ForWarD applied to multiframe sharpened image (c) MForWarD x. The MForWarD algorithm provides
superior noise removal while preserving contrast.
the imaging system. The third slice is from the multiframe restoration result ( x). The sharping restores
contrast out to the 12 lp/mm, more than twice the Nyquist rate, but at the expense of noise amplication.
The fourth slice is from the basic Forward reconstruction, which still has some noise amplied regions.
The bottom slice shows x after multiframe wavelet denoising. We observe that the contrast is preserved
while signicantly eliminating the noise in between the bar chart signal regions.
The nal goal of digital mammography is the detection and diagnosis of cancerous lesions in the breast
tissue. The breast phantom contains small grains of calcium for predicting the diagnostic capability of
the imaging system for calcications in the breast. The calcium grains range from 400 m down to 196
m. Figure 11 shows the performance of the MForWarD algorithm for enhancing the contrast of these
March 18, 2010 DRAFT
Authorized licensed use limited to: Annai Mathammal Sheela Engineering College. Downloaded on June 11,2010 at 01:09:37 UTC from IEEE Xplore. Restrictions apply.
Copyright (c) 2010 IEEE. Personal use is permitted. For any other purposes, Permission must be obtained from the IEEE by emailing [email protected].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
25

100 200 300 400 500 600
100 200 300 40 0 50 0 600
5 lp/mm 6 lp/mm 7 lp/mm 8 lp/mm 9 lp/mm 10 lp/mm 11 lp/mm 12 lp/mm
Multiframe Average
Multiframe Reconstruction
Multiframe Restoration
Basic ForWarD
Multiframe ForWarD
Fig. 10. Top: An input low-dosage image (a zoomed in version of which was illustrated in Fig. 8(b)). The arrow marks the
sampling slice through the resolution chart. Bottom: The ve curves show slices through the horizontal resolution bar charts.
The Nyquist rate of the system corresponds to 5.6 lp/mm. The top slice shows a slice through an interpolated average of the
captured images showing aliasing artifacts and lost contrast. The second slice through z shows enhanced resolution beyond
the Nyquist rate, but poor contrast. The third slice through x shows restored contrast but with noise amplication. The rst
three left ellipsoids in the fourth slice through the basic Forward reconstruction mark the amplied noise regions. The rightmost
ellipsoid marks the lost resolution region. The bottom slice through the MForWarD estimate x shows contrast preservation with
signicantly less noise.
small deposits. The MForWarD algorithm enhances the contrast of even the smallest grains of calcium
carbonite. In fact, the grains are visible even when the grains are nearly indistinguishable from noise in
the single images captured at a standard dosage.
March 18, 2010 DRAFT
Authorized licensed use limited to: Annai Mathammal Sheela Engineering College. Downloaded on June 11,2010 at 01:09:37 UTC from IEEE Xplore. Restrictions apply.
Copyright (c) 2010 IEEE. Personal use is permitted. For any other purposes, Permission must be obtained from the IEEE by emailing [email protected].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
26

400 um
275 um
230 um
196 um
Low-dosage
image
Multiframe
Average
Standard-
dosage
image
Multiframe
ForWarD
Fig. 11. The table of images shows the low-dosage image (rst column), multiframe average (second column), standard dosage
image (third column) and MForWarD image of calcium carbonite deposits with decreasing grain size. The synthetic calcications
are clearly visible in all of the MForWarD images.
V. CONCLUSION
In this paper, we have proposed a novel method for restoring and denoising super-resolved low dosage
X-ray images in a fast multiframe variant of the ForWarD algorithm of [5]. The proposed Fourier
multiframe restoration and wavelet denoising algorithm provides high contrast super-resolved images
while improving the extremely poor SNR of low-dosage images. The experimental results conrm
that multiframe imaging can provide an alternative in the SNR versus resolution tradeoff for digital
mammography. We note that our restoration algorithm can be easily modied to further enhance the
quality of other Shift-and-Add based SR techniques [2], [29].
The design of future X-ray imaging systems would benet from a systematic analysis of the resolution
and SNR required for mammographic screening and diagnosis. Currently, there are no publicly available
databases of multi-frame aliased digital mammography images, only digitized analog mammography
images. Upon approval of an institutional review board (IRB), we intend to apply this technique to
imagery collected from patients in the clinical setting. At this point, we hope to demonstrate the ability
March 18, 2010 DRAFT
Authorized licensed use limited to: Annai Mathammal Sheela Engineering College. Downloaded on June 11,2010 at 01:09:37 UTC from IEEE Xplore. Restrictions apply.
Copyright (c) 2010 IEEE. Personal use is permitted. For any other purposes, Permission must be obtained from the IEEE by emailing [email protected].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
27
of this technique to improve image quality on real digital mammography images.
In the future, one might explore the fundamental tradeoffs between radiation exposure, number of
frames, and reconstruction performance. Furthermore, extensions to the ForWarD algorithm which include
more sophisticated redundant wavelet techniques such as curvelets [30] or ridgelets [31] might show even
better performance. Recent research has shown that use of more sophisticated wavelets have been found
to improve the quality in other medical imaging applications [32]. Future research might also include
exploration of the effects of such multiframe imaging on the higher-level segmentation or detection tasks
associated with digital mammography.
VI. ACKNOWLEDGMENTS
Our MATLAB software implementation of the noted algorithms is in part based on the ForWarD soft-
ware, developed by Dr. Ramesh D. Neelamani of the Digital Signal Processing group at Rice University
(available at http://www.dsp.rice.edu/software/ward.shtml). We would like to thank Prof. Peyman Milanfar
of the University of California Santa Cruz for collaborating with us in the original multi-frame motion
estimation publication.
REFERENCES
[1] M. D. Robinson, S. Farsiu, J. Y. Lo, and C. A. Toth, Efcient restoration and enhancement of super-resolved x-ray
images, in Image Processing, 2008. ICIP 2008. 15th IEEE International Conference on, Oct. 2008, pp. 629632.
[2] S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, Advances and challenges in super-resolution, International Journal of
Imaging Systems and Technology, vol. 14, no. 2, pp. 4757, October 2004.
[3] S. Borman and R. L. Stevenson, Super-resolution from image sequences - a review, in Proc. of the 1998 Midwest
Symposium on Circuits and Systems, vol. 5, Apr. 1998.
[4] S. Park, M. Park, and M. G. Kang, Super-resolution image reconstruction, a technical overview, IEEE Signal Process.
Magazine, vol. 20, no. 3, pp. 2136, May 2003.
[5] R. Neelamani, H. Choi, and R. Baraniuk, Forward: Fourier-wavelet regularized deconvolution for ill-conditioned systems,
IEEE Trans. Signal Process., vol. 52, no. 2, pp. 418433, February 2004.
[6] E. Pisano, C. Gatsonis, E. Hendrick, M. Yaffe, J. Baum, S. Acharyya, E. Conant, L. Fajardo, L. Bassett, C. DOrsi, R. Jong,
and M. Rebner, Diagnostic performance of digital versus lm mammography for breast cancer screening- the results of
the american college of radiology imaging network (ACRIN) digital mammographic imaging screening trial (DMIST),
New England Journal of Medicine, October 2005.
[7] D. Robinson, S. Farsiu, and P. Milanfar, Optimal registation of aliased images using variable projection with applications
to super-resolution, The Computer Journal, vol. 52, January 2009.
[8] M. Elad and Y. Hel-Or, A fast super-resolution reconstruction algorithm for pure translational motion and common space
invariant blur, IEEE Trans. Image Process., vol. 10, no. 8, pp. 11861193, August 2001.
March 18, 2010 DRAFT
Authorized licensed use limited to: Annai Mathammal Sheela Engineering College. Downloaded on June 11,2010 at 01:09:37 UTC from IEEE Xplore. Restrictions apply.
Copyright (c) 2010 IEEE. Personal use is permitted. For any other purposes, Permission must be obtained from the IEEE by emailing [email protected].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
28
[9] S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, Fast and robust multi-frame super-resolution, IEEE Trans. Image
Process., vol. 13, no. 10, pp. 13271344, October 2004.
[10] R. Hardie, A fast image super-resolution algorithm using an adaptive wiener lter, IEEE Trans. Image Process., vol. 16,
no. 12, pp. 29532964, Dec. 2007.
[11] D. Robinson, S. Farsiu, J. Y. Lo, P. Milanfar, and C. A. Toth, Efcient multiframe registration of aliased X-ray images,
Proc. of the 41th Asilomar Conference on Signals, Systems, and Computers, pp. 215219, Nov. 2007.
[12] R. R. Schultz and R. L. Stevenson, Extraction of high-resolution frames from video sequences, IEEE Trans. Image
Processing, vol. 5, no. 6, pp. 9961011, June 1996.
[13] M. Elad and A. Feuer, Restoration of single super-resolution image from several blurred, noisy and down-sampled
measured images, IEEE Trans. Image Process., vol. 6, no. 12, pp. 16461658, Dec. 1997.
[14] H. Takeda, S. Farsiu, and P. Milanfar, Deblurring using regularized locally adaptive kernel regression, IEEE Trans. Image
Process., vol. 17, no. 4, pp. 550563, April 2008.
[15] C. Tomasi and R. Manduchi, Bilateral ltering for gray and color images, in Proc. of IEEE Int. Conf. on Computer
Vision, Jan. 1998, pp. 836846.
[16] V. Patanavijit and S. Jitapunkul, A lorentzian stochastic estimation for a robust iterative multiframe super-resolution
reconstruction with Lorentzian-Tikhonov regularization, EURASIP J. Adv. Signal Process, vol. 2007, no. 2, 2007.
[17] N. A. El-Yamany and P. E. Papamichalis, Robust color image superresolution: an adaptive m-estimation framework, J.
Image Video Process., vol. 8, no. 2, pp. 112, 2008.
[18] R. Hardie, K. Barnard, and E. Armstrong, Joint map registration and high-resolution image estimation using a sequence
of undersampled images, IEEE Trans. Image Process., vol. 6, no. 12, pp. 16211633, 1997.
[19] L. D. Alvarez, J. Mateos, R. Molina, and A. K. Katsaggelos, Advances and challenges in super-resolution, International
Journal of Imaging Systems and Technology, vol. 14, no. 2, pp. 5866, Oct. 2004.
[20] D. Robinson and P. Milanfar, Statistical performance analysis of super-resolution, IEEE Trans. Image Process., vol. 15,
no. 6, pp. 14131428, June 2006.
[21] P. Vandewalle, S. Susstrunk, and M. Vetterli, A frequency domain approach to registeration of aliased images with
application to super-resolution, EURASIP Journal on Applied Signal Processing, p. Article ID 71459, 2006.
[22] N. Woods, N. Galatsanos, and A. Katsaggelos, Stochastic methods for joint registration, restoration, and interpolation of
multiple undersampled images. IEEE Trans. Image Process., vol. 15, no. 1, pp. 201213, Jan 2006.
[23] G. Golub and V. Pereyra, Separable nonlinear least squares: the variable projection method and its applications, Institute
of Physics Inverse Problems, vol. 19, pp. R1R26, 2003.
[24] R. Neelamani, M. Deffenbaugh, and R. Baraniuk, Texas Two-Step: A Framework for Optimal Multi-Input Single-Output
Deconvolution, IEEE Trans. Image Process., vol. 16, no. 11, pp. 27522765, Nov. 2007.
[25] A. K. Jain, Fundamentals of Digital Image Processing, 1st ed. Englewood Cliffs, New Jersey: Prentice Hall, 1989.
[26] A. Murli, L. DAmore, and V. De Simone, The wiener lter and regularization methods for image restoration problems,
Image Analysis and Processing, 1999. Proceedings. International Conference on, pp. 394399, Sept. 1999.
[27] J. Fowler, The redundant discrete wavelet transform and additive noise, IEEE Signal Process. Letters, vol. 12, no. 9, pp.
629632, Sept. 2005.
[28] S. Mallat, A Wavelet Tour of Signal Processing. NY:Academic Press, pp.442, 1998.
[29] H. Takeda, S. Farsiu, and P. Milanfar, Kernel regression for image processing and reconstruction, IEEE Trans. Image
Process., vol. 16, no. 2, pp. 349366, Feb. 2007.
March 18, 2010 DRAFT
Authorized licensed use limited to: Annai Mathammal Sheela Engineering College. Downloaded on June 11,2010 at 01:09:37 UTC from IEEE Xplore. Restrictions apply.
Copyright (c) 2010 IEEE. Personal use is permitted. For any other purposes, Permission must be obtained from the IEEE by emailing [email protected].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.
29
[30] J.-L. Starck, E. Candes, and D. Donoho, The curvelet transform for image denoising, IEEE Trans. Image Process.,
vol. 11, no. 6, pp. 670684, June 2002.
[31] M. Do and M. Vetterli, The nite ridgelet transform for image representation, IEEE Trans. Image Process., vol. 12,
no. 1, pp. 1628, January 2003.
[32] A. Khare and U. S. Tiwary, A new method for deblurring and denoising of medical images using complex wavelet
transform, Proc. IEEE Conference Engineering in Medicine and Biology, pp. 18971900, Sept. 2005.
March 18, 2010 DRAFT
Authorized licensed use limited to: Annai Mathammal Sheela Engineering College. Downloaded on June 11,2010 at 01:09:37 UTC from IEEE Xplore. Restrictions apply.