IEEE SIGNAL PROCESSING LETTERS, VOL. 22, NO.

1, JANUARY 2015

115

An Optimized Pixel-Wise Weighting Approach

for Patch-Based Image Denoising
Jianzhou Feng, Li Song, Xiaoming Huo, Xiaokang Yang, and Wenjun Zhang

AbstractMost existing patch-based image denoising algorithms filter overlapping image patches and aggregate multiple
estimates for the same pixel via weighting. Current weighting
approaches always assume the restored estimates as independent
random variables, which is inconsistent with the reality. In this
letter, we analyze the correlation among the estimates and propose
a bias-variance model to estimate the Mean Squared Error (MSE)
under various weights. The new model exploits the overlapping
information of the patches; it then utilizes the optimization to try
to minimize the estimated MSE. Under this model, we propose
a new weighting approach based on Quadratic Programming
(QP), which can be embedded into various denoising algorithms.
Experimental results show that the Peak Signal to Noise Ratio
(PSNR) of algorithms like K-SVD and EPLL can be improved by
around 0.1 dB under a range of noise levels. This improvement is
promising, since it is gained independent to which image model is
used, especially when the gain from designing new image models
becomes less and less.
Index TermsEPLL, image denoising, K-SVD.

I. INTRODUCTION

MAGE denoising is one of the most classical image processing problems; it aims to restore an image under random
additive white Gaussian noise. Many state-of-the-art image denoising algorithms are based on image patches [1], [2], [3], [4],
[5], [6], [7], [8], [9]. Their denoising methods can be interpreted
as an iteration of a so-called Filtering and Weighting (F&W)
process. Under the F&W process, local image patches are firstly
restored through filtering, and then multiple estimates of the
same pixel from overlapping restored patches are weighted to
derive the final estimate. For the filtering method, sophisticated
patch-based image models have been applied to generate the
filters, e.g., the sparse coding model [1], [3], [5], [6], [9], the
Gaussian Mixture Model [7], [8], and the non-local similarity

Manuscript received July 11, 2014; accepted August 17, 2014. Date of publication August 20, 2014; date of current version August 27, 2014. This work
was supported by the National Science Foundation of China (NSFC) under
Grants 61221001 and 61025005, the 111 Project under Grant B07022, and by
the Shanghai Key Laboratory of Digital Media Processing and Transmissions.
The associate editor coordinating the review of this manuscript and approving
it for publication was Prof. Tolga Tasdizen.
J. Feng, L. Song, X. Yang, and W. Zhang are with Future Medianet
Innovation Center, Shanghai Jiao Tong University, Shanghai, 200240
(e-mail: [email protected]; [email protected]; [email protected];
[email protected]).
X. Huo is with School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, GA 30332 USA, (e-mail: [email protected].
edu).
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/LSP.2014.2350032

model [2], [4]. In contrast, the weighting methods are still

somewhat straightforward, either using simple averaging or
deriving the weight independently based on certain transform
coefficients of the corresponding image patch itself [2], [4], [9].
This form of weighting method is optimal when the estimates
for weighting are independent random variables. However,
the estimates can be heavily correlated due to overlapping of
the patches, which violates the assumption of independence.
Therefore, we may further improve the denoising performance
by analyzing the correlation among the estimates using the
overlapping information.
Based on the above idea, in Section II, we describe the F&W
process precisely, analyze the Mean Squared Error (MSE) under
various weights, and derive a bias-variance model to estimate
it accurately. We also show that optimizing the weight under
the proposed model yields the minimum MSE with the help of
the overlapping information. In Section III, we propose a new
weighting approach to solve the optimization problem under
the bias-variance model via Quadratic Programming (QP). In
Section IV, we introduce the proposed weighting approach into
the K-SVD algorithm and the EPLL algorithm. It indicates that
the Peak Signal to Noise Ratio (PSNR) of both algorithms can
be improved by around 0.1 dB under a range of noise levels. Finally, Section V concludes the letter.
II. THE BIAS-VARIANCE MODEL
In this section, we first formulate the degradation model of
image denoising and describe the F&W process in an analytic
way. Then we propose a bias-variance model to characterize the
correlation of the restored estimates under the F&W process.
The new model can estimate the MSE under various weights
faithfully by exploiting the overlapping information of the
restored patches. Therefore, optimizing the weight under this
model is nearly equivalent to minimizing the real MSE.
A. The Degradation Model and the F&W Process
The degradation model of image denoising can be formulated
as
(1)
where denote the (vectorized) clean image, is its noisy version, and represents the additive white Gaussian noise with
variance .
Under the above notations, one can represent the F&W
process for each pixel as:

1070-9908 2014 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

116

IEEE SIGNAL PROCESSING LETTERS, VOL. 22, NO. 1, JANUARY 2015

Fig. 1. Two examples of and

. In the image of (the left one), the red point indicates pixel and the blue block represents the -th local region. In the
(the right one), the lighter the pixel, the larger the corresponding element in
. (a) A smooth region. (b) A region contains an edge.
image of

Filtering within Local Regions: Suppose there are

local
regions (also known as patches) that share pixel . In the -th
is estimated as
region,
(2)
) can be seen as a low-pass global filter. Various
where (
denoising algorithms compute (
) in their own way, but
the values are just slightly different. For example, in EPLL [7],
suppose pixel is at the -th place of the -th (vectorized) local
region
, where
is a selection matrix, then the patch
is Wiener filtered to be
(3)
where and are the parameter of a Gaussian distribution. In
this case,
equals to the -th column of

B. Two Components in MSE

in (6) is comUnder the F&W process, we assume
puted exactly using the original filtering method of a denoising
algorithm. Therefore, is formulated as a function of , and
is formulated as

(7)
is the number of pixels in and is denoted as the
where
concatenation of all s.
is a random variable depend on noise ,
Since
we propose a bias-variance model, which estimates it by its expectation under . For mathematical derivation simplicity, we
assume that
s are independent to . Hence, the expectation can be estimated as

(4)
and
equals to the -th element of
.
Due to the property of
,
is a sparse vector with nonzero elements only in the -th local region, and its -th element
reflects the closeness between and . As illustrated in Fig. 1,
the -th element of
is always the largest, and if the local
region in contains two smooth areas like in Fig. 1(b), the -th
is close to 0 when pixel is in the other area;
element of
is a bias term of the filter.
Weighting Throughout Local Regions: The
estimates are
weighted to derive the final estimate of as
(5)
where the weights
Denote

s are nonnegative and sums to one.

,
and
, we have
(6)

All the denoising algorithms in [1], [2], [3], [4], [5], [6], [7],
[8], [9] fit the F&W process quite well. As for the Non-local
Means algorithm [10], though it can be seem as a weighting algorithm without filtering, the weights actually reflect the closes do under the
ness among pixels, which is mainly what
F&W process. Hence, NLM is more proper to be interpreted as
a global filtering process with only one estimate for each pixel.

(8)
where
(9)
is the bias of

to

, and
(10)

is the variance.
In reality,
s are derived from , which makes
them still correlated to , i.e., the assumption that leads to
(8) may be violated. To evaluate the appropriateness of using
to approximate
, we compute
their ratio
(11)
is a constant under all s, then we
under various s. If
can conclude that optimizing (8) is equivalent to minimizing the
true value of
. Experimental results done on several standard images under three representative denoising algorithms, K-SVD [1], EPLL [7], and BM3D [2] validate this
guess. As shown in Table I, under each image and denoising algorithm combination, the values of
under an averaging
and a uniformly sampled are really close. We only list these
two values for illustration due to space limitation, the value of

FENG et al.: OPTIMIZED PIXEL-WISE WEIGHTING APPROACH

117

TABLE I
UNDER (IMAGE, DENOISING ALGORITHM) COMBINATIONS. IN
AND THE
EACH COMBINATION, THE LEFT ONE USE AVERAGING
RIGHT ONE USE A UNIFORMLY SAMPLED

tion of two weights, each minimizes the bias and the variance
component separately, with a practically derived combination
coefficient.
A. The Approximation Profile

under other s are also quite close to the presented ones.

Therefore, we denote the objective function as

. Hence,
There is unknown pixel values contained in
before optimizing
, we need to approximate
first based
and
s. For simplicity, like previous weighting
on
methods assume
as a diagonal matrix, we assume
here as a diagonal matrix , while still retain the overlapping
. The -th diagonal entry of
information of patches in
is
(15)

(12)

We approximate it as

where

(16)
(13)

)s, for
In current weighting approaches, (
, are assumed to be independent random variables with zero
means, so that their covariance matrix is assumed to be a diagonal matrix. Among them, the most promising one computes
the weight as the inverse of the sparsity of the transform coefficients [2], [9]. Though it has been validated under a shift-invariant DCT transform based denoising algorithm [9], experimental results show that the same weighting method doesnt
work for K-SVD and EPLL, which use simple averaging originally. The reason is: on one side, in [9], some estimates are not
so good when the block DCT basis can not represent the patches
sparsely, so that the weighting strategy performs well by giving
such estimates small weights; on the other side, in K-SVD and
EPLL, all the estimates are comparable since the transform is
more adaptive and always yields sparse representations, which
makes this strategy ineffective.
While under the bias-variance model, there is no independence assumption, and the covariance matrix is derived analytically as
(14)
is superior than any diagonal maas shown in (12). Such
trix because it retain the overlapping information of different
local regions by computing
. It is easy to see that each elis a inner product of two
s. As mentioned in
ement of
Section II-A,
has zero elements outside local region and
the value of its nonzero elements is mostly dependent on how
close is the corresponding to . Therefore, the inner product
and
can be seen as the total squared closeness to
of
of the overlapped pixels in local region
and .
III. THE QP BASED WEIGHTING APPROACH
In this section, we propose a Quadratic Programming (QP)
. This approach
based weighting approach for optimizing
contains two profiles. In Section III-A, we propose the approxwith an approximation
imation profile, which optimizes
matrix
. In Section III-B, we propose the practical profile, which computes the optimal weight as a linear combina-

where
(17)
is the mean of all the
s and is a small parameter to ensure
the entry to be positive.
Under this approximation, the optimal weight is

(18)

can be solved independently via

It is easy to see that each
QP. We also note that using the Lagrangian multiplier method
may lead to negative elements in , which
based on
means constraint
is non-trivial.
B. The Practical Profile
The approximation profile can be interpreted under a more
general linear combination framework. We can see that
in
(18) can be approximated by
(19)
with a certain

, where
(20)

and
(21)
under the same constraints as in (18).
In the practical profile, we compute the optimal weight as
, where
is a practically learned real scalar that
yields the minimum averaged MSE on a training image set.
is a good approximation of
,
would be within
When

118

IEEE SIGNAL PROCESSING LETTERS, VOL. 22, NO. 1, JANUARY 2015

TABLE II
PSNR COMPARISON UNDER K-SVD. UNDER EACH NOISE LEVEL, THE
LEFT COLUMN USES THE ORIGINAL WEIGHT, THE RIGHT COLUMN
USES WEIGHT OF THE PRACTICAL PROFILE

Fig. 2. The PSNR gain by using

under
for K-SVD. Each curve
represent one training image and the circle indicates the position of the optimal
that leads to the maximum gain for that image.

so that both profiles can improve the denoising performance. Otherwise,

may be negative and only the practical
profile performs well under this case.
In practice, we find that s are quite likely to be a positive
scalar times the identity matrix, which makes to be the averaging weight. Thus we modify to be the original weight of
a certain denoising algorithm. This reduces the computational
cost of solving (20) and makes the original algorithm as a spe.
cial case under the practical profile by setting
IV. NUMERICAL EXPERIMENTS
In this section, we test the proposed weighting approach
under three representative denoising algorithms: K-SVD,
EPLL, and BM3D. Due to space limitation, we list the experimental results done on frequently used standard images
in image denoising domain here, and list the other supportive
results on our webpage.
For the K-SVD algorithm, we find the denoising performance can be improved most significantly under high noise
levels using the practical profile with negative s. For each
noise level,
is pre-learned from a training set with three
as an example, as shown in
standard images. Taking
for
,
Fig. 2, we compute the PSNR gain of using
and find that
can lead to almost the maximum
PSNR gain for any of the training images. Therefore, we set
under
to denoise all the images. After the
training process, we apply the practical profile to the other 8
standard images under from 30 to 50, and compare the PSNR
with the original K-SVD algorithm. As shown in Table II, the
averaged PSNR gain increases from about 0.1 dB to 0.2 dB as
increases.
For the EPLL algorithm, we apply the proposed weighting
approach for multiple times since it is an iterative algorithm.
We find both of the two profiles are effective for moderate noise
levels, while the approximation profile performs even better.
Therefore, we choose the approximation profile to improve the
EPLL algorithm. As shown in Table III, for noise level from
to
, the averaged PSNR gain can reach around
0.1dB. The proposed weighting approach is not effective for
high noise levels probably because: It is designed to minimize

TABLE III
PSNR COMPARISON UNDER EPLL. UNDER EACH NOISE LEVEL, THE
LEFT COLUMN USES THE ORIGINAL WEIGHT, THE RIGHT COLUMN
USES WEIGHT OF THE APPROXIMATION PROFILE

the MSE under only one F&W process. When it is used for multiple times, minimizing the MSE within each iteration may not
be the optimal. As the noise level increases, the number of iteration also increases, which enlarges the impact of the misleading
objective.
For the BM3D algorithm, we find the PSNR improvement
by using the proposed weighting approach is insignificant, no
matter which profile is used. This is probably because BM3D
has much more estimates for the same pixel compare to K-SVD
and EPLL, and their correlation is also more complicated, which
in (14)
makes approximating the hidden covariance matrix
accurately very hard. Therefore, we need to design more sophisticated profiles for BM3D in the future.
V. CONCLUSION
In this letter, we propose a bias-variance model to estimate
the MSE accurately by analyzing the correlation among the estimates. We then propose a new weighting approach that contains
two profiles using QP. The proposed weighting approach optimize the weights by preserving the overlapping information of
restored patches. Experimental results show that the PSNR gain
of K-SVD and EPLL can be improved by about 0.1 dB under
a range of noise levels. The 0.1 dB improvement is promising,
since it is independent to which image model is used, especially
when the gain from designing new image models becomes less
and less.
This work setup a novel bias-variance model that formulates
the selection of weights as an optimization problem. The proposed two profiles for solving this optimization problem can be
seen as a stepping stone, and better profiling methodology may
be proposed with more sophisticated techniques.

FENG et al.: OPTIMIZED PIXEL-WISE WEIGHTING APPROACH

REFERENCES
[1] M. Elad and M. Aharon, Image denoising via sparse and redundant representations over learned dictionaries, IEEE Trans. Image
Process., vol. 15, no. 12, pp. 37363745, Dec. 2006.
[2] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, Image denoising
by sparse 3d transform-domain collaborative filtering, IEEE Trans.
Image Process., vol. 16, no. 8, pp. 20802095, Aug. 2007.
[3] J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman, Non-local
sparse models for image restoration, in Proc. IEEE Int. Conf. Computer Vision, 2009, pp. 22722279.
[4] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, Bm3d image denoising with shape-adaptive principal component analysis, in Proc.
Workshop on Signal Processing with Adaptive Sparse Structured Representations (SPARS09, 2009), 2009.

119

[5] G. Yu, G. Sapiro, and S. Mallat, Image modeling and enhancement

via structured sparse model selection, in Proc. IEEE Int. Conf. Image
Process, Sep. 2010, pp. 16411644.
[6] W. Dong, X. Li, L. Zhang, and G. Shi, Sparsity-based image denoising
via dictionary learning and structural clustering, in Proc. IEEE Int.
Conf. Computer Vision and Pattern Recognition, Jun. 2011.
[7] D. Zoran and Y. Weiss, From learning models of natural image
patches to whole image restoration, in Proc. IEEE Int. Conf. Computer Vision, 2011, pp. 479486.
[8] J. Feng, L. Song, X. Huo, X. Yang, and W. Zhang, Image restoration
via efficient gaussian mixture model learning, in Proc. IEEE Int. Conf.
Image Process, Sep. 2013.
[9] O. G. Guleryuz, Weighted averaging for denoising with overcomplete dictionaries, IEEE Trans. Image Process., vol. 16, no. 12, pp.
30203034, 2007.
[10] A. Buades, B. Coll, and J. Morel, A non-local algorithm for image denoising, in Proc. IEEE Int. Conf. Computer Vision and Pattern Recognition, Jun. 2005.