A non-local algorithm for image denoising

Antoni Buades, Bartomeu Coll Jean-Michel Morel

Dpt. Matemàtiques i Informàtica, UIB CMLA, ENS Cachan
Ctra. Valldemossa Km. 7.5, 61, Av du Président Wilson
07122 Palma de Mallorca, Spain 94235 Cachan, France
[email protected], [email protected] [email protected]

Abstract Formally we define a denoising method Dh as a decom-

position
We propose a new measure, the method noise, to evalu- v = Dh v + n(Dh , v),
ate and compare the performance of digital image denois- where v is the noisy image and h is a filtering parame-
ing methods. We first compute and analyze this method ter which usually depends on the standard deviation of the
noise for a wide class of denoising algorithms, namely the noise. Ideally, Dh v is smoother than v and n(Dh , v) looks
local smoothing filters. Second, we propose a new algo- like the realization of a white noise. The decomposition
rithm, the non local means (NL-means), based on a non lo- of an image between a smooth part and a non smooth or
cal averaging of all pixels in the image. Finally, we present oscillatory part is a current subject of research (for example
some experiments comparing the NL-means algorithm and Osher et al. [10]). In [8], Y. Meyer studied the suitable func-
the local smoothing filters. tional spaces for this decomposition. The primary scope of
this latter study is not denoising since the oscillatory part
contains both noise and texture.
1. Introduction The denoising methods should not alter the original im-
age u. Now, most denoising methods degrade or remove the
fine details and texture of u. In order to better understand
The goal of image denoising methods is to recover the this removal, we shall introduce and analyze the method
original image from a noisy measurement, noise. The method noise is defined as the difference be-
tween the original (always slightly noisy) image u and its
v(i) = u(i) + n(i), (1) denoised version.
We also propose and analyze the NL-means algorithm,
where v(i) is the observed value, u(i) is the “true” value and which is defined by the simple formula
n(i) is the noise perturbation at a pixel i. The best simple

way to model the effect of noise on a digital image is to add 1 (Ga ∗|u(x+.)−u(y+.)|2 )(0)
N L[u](x) = e− h2 u(y) dy,
a gaussian white noise. In that case, n(i) are i.i.d. gaussian C(x) Ω
values with zero mean and variance σ 2 .
 (Ga ∗|u(x+.)−u(z+.)|2 )(0)
Several methods have been proposed to remove the noise where x ∈ Ω, C(x) = Ω e− h2 dz is a
and recover the true image u. Even though they may be very normalizing constant, Ga is a Gaussian kernel and h acts
different in tools it must be emphasized that a wide class as a filtering parameter. This formula amounts to say that
share the same basic remark : denoising is achieved by ave- the denoised value at x is a mean of the values of all points
raging. This averaging may be performed locally: the Gaus- whose gaussian neighborhood looks like the neighborhood
sian smoothing model (Gabor [7]), the anisotropic filtering of x. The main difference of the NL-means algorithm with
(Perona-Malik [11], Alvarez et al. [1]) and the neighbor- respect to local filters or frequency domain filters is the sys-
hood filtering (Yaroslavsky [16], Smith et al. [14], Tomasi tematic use of all possible self-predictions the image can
et al. [15]), by the calculus of variations: the Total Varia- provide, in the spirit of [6]. For a more detailed analysis on
tion minimization (Rudin-Osher-Fatemi [13]), or in the fre- the NL-means algorithm and a more complete comparison,
quency domain: the empirical Wiener filters (Yaroslavsky see [2].
[16]) and wavelet thresholding methods (Coiffman-Donoho Section 2 introduces the method noise and computes its
[5, 4]). mathematical formulation for the mentioned local smooth-

Authorized licensed use limited to: J.R.D. Tata Memorial Library Indian Institute of Science Bengaluru. Downloaded on July 26,2024 at 10:58:15 UTC from IEEE Xplore. Restrictions apply.
 ing filters. Section 3 gives a discrete definition of the NL- 2.2. The anisotropic filtering
means algorithm. In section 4 we give a theoretical result
on the consistency of the method. Finally, in section 5 we The anisotropic filter (AF ) attempts to avoid the blurring
compare the performance of the NL-means algorithm and effect of the Gaussian by convolving the image u at x only
the local smoothing filters. in the direction orthogonal to Du(x). The idea of such filter
goes back to Perona and Malik [11]. It is defined by
2. Method noise

Du(x)⊥
AFh u(x) = Gh (t)u(x + t )dt,
|Du(x)|
Definition 1 (Method noise) Let u be an image and Dh a
denoising operator depending on a filtering parameter h. for x such that Du(x) = 0 and where (x, y)⊥ = (−y, x)
Then, we define the method noise as the image difference and Gh is the one-dimensional Gauss function with vari-
ance h2 . If one assumes that the original image u is twice
u − Dh u. continuously differentiable (C 2 ) at x, it is easily shown by
a second order Taylor expansion that
The application of a denoising algorithm should not al-
ter the non noisy images. So the method noise should be Theorem 2 The image method noise of an anisotropic filter
very small when some kind of regularity for the image is AFh is
assumed. If a denoising method performs well, the method
1
noise must look like a noise even with non noisy images u(x) − AFh u(x) = − h2 |Du|curv(u)(x) + o(h2 ),
and should contain as little structure as possible. Since 2
even good quality images have some noise, it makes sense where the relation holds when Du(x) = 0.
to evaluate any denoising method in that way, without the
traditional “add noise and then remove it” trick. We shall By curv(u)(x), we denote the curvature, i.e. the signed
list formulas permitting to compute and analyze the method inverse of the radius of curvature of the level line passing by
noise for several classical local smoothing filters: the Gaus- x. This method noise is zero wherever u behaves locally like
sian filtering [7], the anisotropic filtering [1, 11], the Total a straight line and large in curved edges or texture (where
Variation minimization [13] and the neighborhood filtering the curvature and gradient operators take high values). As a
[16]. The formal analysis of the method noise for the fre- consequence, the straight edges are well restored while flat
quency domain filters fall out of the scope of this paper. and textured regions are degraded.
These method noises can also be computed but their inter-
pretation depends on the particular choice of the wavelet 2.3. The Total Variation minimization
basis.
The Total Variation minimization was introduced by
2.1. The Gaussian filtering Rudin, Osher and Fatemi [13]. Given a noisy image v(x),
these authors proposed to recover the original image u(x)
The image isotropic linear filtering boils down to the as the solution of the minimization problem
convolution of the image by a linear symmetric kernel. The

paradigm of such kernels is of course the gaussian kernel TVFλ (v) = arg min T V (u) + λ |v(x) − u(x)|2 dx
|x|2 u
1 − 4h2
x → Gh (x) = (4πh 2) e . In that case, Gh has standard
deviation h and it is easily seen that where T V (u) denotes the total variation of u and λ is a
given Lagrange multiplier. The minimum of the above min-
Theorem 1 (Gabor 1960) The image method noise of the imization problem exists and is unique. The parameter λ
convolution with a gaussian kernel Gh is is related to the noise statistics and controls the degree of
filtering of the obtained solution.
u − Gh ∗ u = −h2 ∆u + o(h2 ),
Theorem 3 The method noise of the Total Variation mini-
for h small enough. mization is
1
The gaussian method noise is zero in harmonic parts of u(x) − TVFλ (u)(x) = − curv(TVFλ (u))(x).
2λ
the image and very large near edges or texture, where the
Laplacian cannot be small. As a consequence, the Gaussian As in the anisotropic case, straight edges are maintained
convolution is optimal in flat parts of the image but edges because of their small curvature. However, details and tex-
and texture are blurred. ture can be over smoothed if λ is too small.

Authorized licensed use limited to: J.R.D. Tata Memorial Library Indian Institute of Science Bengaluru. Downloaded on July 26,2024 at 10:58:15 UTC from IEEE Xplore. Restrictions apply.
 2.4. The neighborhood filtering

We call neighborhood filter any filter which restores a

pixel by taking an average of the values of neighboring pix-
els with a similar grey level value. Yaroslavsky (1985) [16]
averages pixels with a similar grey level value and belong-
ing to the spatial neighborhood Bρ (x),


1 |u(y)−u(x)|2
YNFh,ρ u(x) = u(y)e− h2 dy, (2)
C(x) Bρ (x)

 |u(y)−u(x)|2
where x ∈ Ω, C(x) = Bρ (x) e− h2 dy is the normal-
ization factor and h is a filtering parameter.
The Yaroslavsky filter is less known than more recent
versions, namely the SUSAN filter (1995) [14] and the Bilat- Figure 1. Scheme of NL-means strategy. Similar
eral filter (1998) [15]. Both algorithms, instead of consider- pixel neighborhoods give a large weight, w(p,q1)
ing a fixed spatial neighborhood Bρ (x), weigh the distance and w(p,q2), while much different neighborhoods
to the reference pixel x, give a small weight w(p,q3).


1 −
|y−x|2 |u(y)−u(x)|2
SNFh,ρ u(x) = u(y)e ρ2 e− h2 dy,
C(x) Ω
(3) as a decreasing function of the weighted Euclidean distance,
 − |y−x| 2
−
|u(y)−u(x)|2 v(Ni ) − v(Nj )22,a , where a > 0 is the standard deviation
where C(x) = Ω e ρ e 2 h2 dy is the normaliza-
of the Gaussian kernel. The application of the Euclidean
tion factor and ρ is now a spatial filtering parameter. In prac-
distance to the noisy neighborhoods raises the following
tice, there is no difference between YNFh,ρ and SNFh,ρ . If
equality
the grey level difference between two regions is larger than
h, both algorithms compute averages of pixels belonging to E||v(Ni ) − v(Nj )||22,a = ||u(Ni ) − u(Nj )||22,a + 2σ 2 .
the same region as the reference pixel. Thus, the algorithm
does not blur the edges, which is its main scope. In the This equality shows the robustness of the algorithm since in
experimentation section we only compare the Yaroslavsky expectation the Euclidean distance conserves the order of
neighborhood filter. similarity between pixels.
The problem with these filters is that comparing only The pixels with a similar grey level neighborhood to
grey level values in a single pixel is not so robust when v(Ni ) have larger weights in the average, see Figure 1.
these values are noisy. Neighborhood filters also create ar- These weights are defined as,
tificial shocks which can be justified by the computation of
2
its method noise, see [3]. 1 − ||v(Ni )−v(N j )||2,a
w(i, j) = e h2 ,
Z(i)
3. NL-means algorithm
where Z(i) is the normalizing constant
Given a discrete noisy image v = {v(i) | i ∈ I}, the  ||v(Ni )−v(Ni )||2
2,a
estimated value N L[v](i), for a pixel i, is computed as a Z(i) = e− h2

weighted average of all the pixels in the image, j


N L[v](i) = w(i, j)v(j), and the parameter h acts as a degree of filtering. It controls
j∈I
the decay of the exponential function and therefore the de-
cay of the weights as a function of the Euclidean distances.
where the family of weights {w(i, j)}j depend on the si- The NL-means not only compares the grey level in a sin-
milarity between the pixels i and  j, and satisfy the usual gle point but the the geometrical configuration in a whole
conditions 0 ≤ w(i, j) ≤ 1 and j w(i, j) = 1. neighborhood. This fact allows a more robust comparison
The similarity between two pixels i and j depends on than neighborhood filters. Figure 1 illustrates this fact, the
the similarity of the intensity gray level vectors v(Ni ) and pixel q3 has the same grey level value of pixel p, but the
v(Nj ), where Nk denotes a square neighborhood of fixed neighborhoods are much different and therefore the weight
size and centered at a pixel k. This similarity is measured w(p, q3) is nearly zero.

Authorized licensed use limited to: J.R.D. Tata Memorial Library Indian Institute of Science Bengaluru. Downloaded on July 26,2024 at 10:58:15 UTC from IEEE Xplore. Restrictions apply.
 (a) (b) (c)

(d) (e) (f)

Figure 2. Display of the NL-means weight distribution used to estimate the central pixel of every image. The
weights go from 1(white) to zero(black).

4. NL-means consistency (i) E[V (i) | Xi = x] = E[U (i) | Xi = x] for all i ∈ I

and x ∈ Rp .
Under stationarity assumptions, for a pixel i, the NL-
means algorithm converges to the conditional expectation (ii) The expected random variable E[U (i) | V (Ni \{i})]
of i once observed a neighborhood of it. In this case, the is the function of V (Ni \{i}) that minimizes the mean
stationarity conditions amount to say that as the size of the square error
image grows we can find many similar patches for all the
min E[U (i) − g(V (Ni \{i}))]2
details of the image. g
Let V be a random field and suppose that the noisy ima-
ge v is a realization of V . Let Z denote the sequence of Similar optimality theoretical results have been obtained in
random variables Zi = {Yi , Xi } where Yi = V (i) is real [9] and presented for the denoising of binary images. Theo-
valued and Xi = V (Ni \{i}) is Rp valued. The NL-means retical links between the two algorithms will be explored in
is an estimator of the conditional expectation r(i) = E[Yi | a future work.
Xi = v(Ni \{i})].
Theorem 4 (Conditional expectation theorem) Let Z = 5. Discussion and experimentation
{V (i), V (Ni \{i})} for i = 1, 2, . . . be a strictly sta-
tionary and mixing process. Let N Ln denote the In this section we compare the local smoothing filters
NL-means algorithm applied to the sequence Zn = and the NL-means algorithm under three well defined cri-
{V (i), V (Ni \{i})}ni=1 . Then, teria: the method noise, the visual quality of the restored
image and the mean square error, that is, the Euclidean dif-
|N Ln (j) − r(j)| → 0 a.s
ference between the restored and true images.
for j ∈ {1, . . . , n}. For computational purposes of the NL-means algorithm,
we can restrict the search of similar windows in a larger
The full statement of the hypothesis of the theorem and its
”search window” of size S × S pixels. In all the experimen-
proof can be found in a more general framework in [12].
tation we have fixed a search window of 21×21 pixels and a
This theorem tells us that the NL-means algorithm corrects
similarity square neighborhood Ni of 7 × 7 pixels. If N 2 is
the noisy image rather than trying to separate the noise (os-
the number of pixels of the image, then the final complexity
cillatory) from the true image (smooth).
of the algorithm is about 49 × 441 × N 2 .
In the case that an additive white noise model is assumed,
The 7 × 7 similarity window has shown to be large
the next result shows that the conditional expectation is the
enough to be robust to noise and small enough to take care
function of V (Ni \{i}) that minimizes the mean square er-
of details and fine structure. The filtering parameter h has
ror with the true image u.
been fixed to 10 ∗ σ when a noise of standard deviation σ
Theorem 5 Let V, U, N be random fields on I such that is added. Due to the fast decay of the exponential kernel,
V = U + N , where N is a signal independent white noise. large Euclidean distances lead to nearly zero weights acting
Then, the following statements are hold. as an automatic threshold, see Fig. 2.

Authorized licensed use limited to: J.R.D. Tata Memorial Library Indian Institute of Science Bengaluru. Downloaded on July 26,2024 at 10:58:15 UTC from IEEE Xplore. Restrictions apply.
 Figure 3. Denoising experience on a natural texture. From left to right: noisy image (standard deviation 35),
Gauss filtering, anisotropic filtering, Total variation, Neighborhood filtering and NL-means algorithm.

Figure 4. Method noise experience on a natural image. Displaying of the image difference u−Dh (u). From left to
right and from top to bottom: original image, Gauss filtering, anisotropic filtering, Total variation minimization,
Neighborhood filtering and NL-means algorithm. The visual experiments corroborate the formulas of section 2.

In section 2 we have computed explicitly the method struction of edges, texture and details.
noise of the local smoothing filters. These formulas are cor- Due to the nature of the algorithm, the most favorable
roborated by the visual experiments of Figure 4. This fig- case for the NL-means is the textured or periodic case. In
ure displays the method noise for the standard image Lena, this situation, for every pixel i, we can find a large set of
that is, the difference u − Dh (u), where the parameter h samples with a very similar configuration. See Figure 2 e)
is been fixed in order to remove a noise of standard devia- for an example of the weight distribution of the NL-means
tion 2.5. The method noise helps us to understand the per- algorithm for a periodic image. Figure 3 compares the per-
formance and limitations of the denoising algorithms, since formance of the NL-means and local smoothing filters for a
removed details or texture have a large method noise. We natural texture.
see in Figure 4 that the NL-means method noise does not Natural images also have enough redundancy to be re-
present any noticeable geometrical structures. Figure 2 ex- stored by NL-means. Flat zones present a huge number of
plains this property since it shows how the NL-means al- similar configurations lying inside the same object, see Fig-
gorithm chooses a weighting configuration adapted to the ure 2 (a). Straight or curved edges have a complete line
local and non local geometry of the image. of pixels with similar configurations, see Figure 2 (b) and
The human eye is the only one able to decide if the (c). In addition, natural images allow us to find many simi-
quality of the image has been improved by the denoising lar configurations in far away pixels, as Figure 2 (f) shows.
method. We display some denoising experiences compar- Figure 5 shows an experiment on a natural image. This ex-
ing the NL-means algorithm with local smoothing filters. perience must be compared with Figure 4, where we display
All experiments have been simulated by adding a gaussian the method noise of the original image. The blurred or de-
white noise of standard deviation σ to the true image. The graded structures of the restored images coincide with the
objective is to compare the visual quality of the restored noticeable structures of its method noise.
images, the non presence of artifacts and the correct recon- Finally Table 1 displays the mean square error for the

Authorized licensed use limited to: J.R.D. Tata Memorial Library Indian Institute of Science Bengaluru. Downloaded on July 26,2024 at 10:58:15 UTC from IEEE Xplore. Restrictions apply.
 Figure 5. Denoising experience on a natural image. From left to right and from top to bottom: noisy image
(standard deviation 20), Gauss filtering, anisotropic filtering, Total variation, Neighborhood filtering and NL-
means algorithm. The removed details must be compared with the method noise experience, Figure 4.

Image GF AF TVF YNF NL [6] A. Efros and T. Leung. Texture synthesis by non parametric
Lena 120 114 110 129 68 sampling. In Proc. Int. Conf. Computer Vision, volume 2,
Baboon 507 418 365 381 292 pages 1033–1038, 1999.
[7] M. Lindenbaum, M. Fischer, and A. Bruckstein. On gabor
contribution to image enhancement. Pattern Recognition,
Table 1. Mean square error table. A smaller mean 27:1–8, 1994.
square error indicates that the estimate is closer [8] Y. Meyer. Oscillating Patterns in Image Processing and
Nonlinear Evolution Equations, volume 22. AMS Univer-
to the original image.
sity Lecture Series, 2002.
[9] E. Ordentlich, G. Seroussi, M. W. S. Verdú, and T. Weiss-
man. A discrete universal denoiser and its application to bi-
nary images. In Proc. IEEE ICIP, volume 1, pages 117–120,
denoising experiments given in the paper. This numerical
2003.
measurement is the most objective one, since it does not [10] S. Osher, A. Sole, and L. Vese. Image decomposition and
rely on any visual interpretation. However, this error is not restoration using total variation minimization and the h−1
computable in a real problem and a small mean square error norm. Multiscale Modeling and Simulation, 1(3):349–370,
does not assure a high visual quality. So all above discussed 2003.
criteria seem necessary to compare the performance of al- [11] P. Perona and J. Malik. Scale space and edge detection using
gorithms. anisotropic diffusion. IEEE Trans. Patt. Anal. Mach. Intell.,
12:629–639, 1990.
[12] G. Roussas. Nonparametric regression estimation under
References mixing conditions. Stochastic processes and their applica-
tions, 36:107–116, 1990.
[13] L. Rudin, S. Osher, and E. Fatemi. Nonlinear total variation
[1] L. Alvarez, P.-L. Lions, and J.-M. Morel. Image selective
based noise removal algorithms. Physica D, 60:259–268,
smoothing and edge detection by nonlinear diffusion (ii).
1992.
Journal of numerical analysis, 29:845–866, 1992.
[14] S. Smith and J. Brady. Susan - a new approach to low level
[2] A. Buades, B. Coll, and J. Morel. On image denoising meth- image processing. International Journal of Computer Vi-
ods. Technical Report 2004-15, CMLA, 2004. sion, 23(1):45–78, 1997.
[3] A. Buades, B. Coll, and J. Morel. Neighborhood filters and [15] C. Tomasi and R. Manduchi. Bilateral filtering for gray and
pde’s. Technical Report 2005-04, CMLA, 2005. color images. In Proceedings of the Sixth Internatinal Con-
[4] R. Coifman and D. Donoho. Wavelets and Statistics, chapter ference on Computer Vision, pages 839–846, 1998.
Translation-invariant de-noising, pages 125–150. Springer [16] L. Yaroslavsky. Digital Picture Processing - An Introduc-
Verlag, 1995. tion. Springer Verlag, 1985.
[5] D. Donoho. De-noising by soft-thresholding. IEEE Trans-
actions on Information Theory, 41:613–627, 1995.

Authorized licensed use limited to: J.R.D. Tata Memorial Library Indian Institute of Science Bengaluru. Downloaded on July 26,2024 at 10:58:15 UTC from IEEE Xplore. Restrictions apply.