1

Image Processing using Smooth Ordering of its

Patches
Idan Ram, Michael Elad, Fellow, IEEE, and Israel Cohen, Senior Member, IEEE
AbstractWe propose an image processing scheme based on
reordering of its patches. For a given corrupted image, we
extract all patches with overlaps, refer to these as coordinates
in high-dimensional space, and order them such that they are
chained in the shortest possible path, essentially solving the
traveling salesman problem. The obtained ordering applied to
the corrupted image, implies a permutation of the image pixels
to what should be a regular signal. This enables us to obtain
good recovery of the clean image by applying relatively simple
one-dimensional (1D) smoothing operations (such as ltering or
interpolation) to the reordered set of pixels. We explore the use
of the proposed approach to image denoising and inpainting, and
show promising results in both cases.
Index Termspatch-based processing, traveling salesman,
pixel permutation, denoising, inpainting.
I. INTRODUCTION
In recent years, image processing using local patches has
become very popular and was shown to be highly effective
see [1-11] for representative work. The core idea behind these
and many other contributions is the same: given the image
to be processed, extract all possible patches with overlaps;
these patches are typically very small compared to the original
image size (a typical patch size would be 8 8 pixels).
The processing itself proceeds by operating on these patches
and exploiting interrelations between them. The manipulated
patches (or sometimes only their center pixels) are then put
back into the image canvas to form the resulting image.
There are various ways in which the relations between
patches can be taken into account: weighted averaging of
pixels with similar surrounding patches, as the NL-Means
algorithm does [1], clustering the patches into disjoint sets
and treating each set differently, as performed in [2], [3], [4]
[5], [6], seeking a representative dictionary for the patches
and using it to sparsely represent them, as practiced in [7],
[8], [9] and [10], gathering groups of similar patches and
applying a sparsifying transform on them [9], [11]. A common
theme to many of these methods is the expectation that every
patch taken from the image may nd similar ones extracted
elsewhere in the image. Put more broadly, the image patches
are believed to exhibit a highly-structured geometrical form in
the embedding space they reside in. A joint treatment of these
patches supports the reconstruction process by introducing a
non-local force, thus enabling better recovery.
I. Ram and I. Cohen are with the Department of Electrical Engi-
neering, Technion Israel Institute of Technology, Technion City, Haifa
32000, Israel. E-mail addresses: [email protected] (I. Ram), ico-
[email protected] (I. Cohen); tel.: +972-4-8294731; fax: +972-4-8295757.
M. Elad is with the Department of Computer Science, Technion Israel
Institute of Technology, Technion City, Haifa 32000, Israel. E-mail address:
[email protected]
In our previous work [12] and [13] we proposed yet another
patch-based image processing approach. We constructed an
image-adaptive wavelet transform which is tailored to sparsely
represent the given image. We used a plain 1D wavelet
transform and adapted it to the image by operating on a
permuted order of the image pixels
1
. The permutation we
proposed is drawn from a shortest path ordering of the image
patches. This way, the patches are leveraged to form a multi-
scale sparsifying global transform for the image in question.
In this paper we embark from our earlier work as reported in
[12] and [13], adopting the core idea of ordering the patches.
However, we discard the globality of the obtained transform
and the processing that follows, the multi-scale treatment, and
the sparsity-driven processing that follows. Thus, we propose
a very simple image processing scheme that relies solely on
patch reordering. We start by extracting all the patches of size

n

n with maximal overlaps. Once these patches are
extracted, we disregard their spatial relationships altogether,
and seek a new way for organizing them. We propose to refer
to these patches as a cloud of vectors/points in R
n
, and we
order them such that they are chained in the shortest possible
path, essentially solving the traveling salesman problem [15].
This reordering is the one we have used in [12] and [13],
but as opposed to our past work, our treatment from this
point varies substantially. A key assumption in this work is
that proximity between two image patches implies proximity
between their center pixels. Therefore if the image mentioned
above is of high-quality, the new ordering of the patches is
expected to induce a highly regular (smooth or at least a
piece-wise smooth) 1D ordering of the image pixels, being
the center of these patches. When the image is deteriorated
(noisy, containing missing pixels, etc.), the above ordering is
expected to be robust to the distortions, thereby suggesting
a reordering of the corrupted pixels to what should be a
regular signal. Thus, applying relatively simple 1D smoothing
operations (such as ltering or interpolation) to the reordered
set of pixels should enable good recovery of the clean image.
This is the core process we propose in this paper for a
given corrupted image, we reorder its pixels, operate on the
new 1D signal using simplied algorithms, and reposition the
resulting values to their original location. We show that the
proposed method, applied with several randomly constructed
orderings and combined with a proposed subimage averaging
scheme, is able to lead to state-of-the-art results. We explore
the use of the proposed image reconstruction scheme to image
denoising, and show that it achieves results similar to the
1
Note that the idea of adapting a wavelet transform to the image by
reordering its pixels appeared already in [14], but the scheme proposed there
did not use image patches, and targeted image compression only.
a
r
X
i
v
:
1
2
1
0
.
3
8
3
2
v
1


[
c
s
.
C
V
]


1
4

O
c
t

2
0
1
2
2
ones obtained with the K-SVD algorithm [7]. We also explore
the use of the proposed image processing scheme to image
inpainting, and show that it leads to better results compared to
the ones obtained with a simple interpolation scheme and the
method proposed in [16] which employs sparse representation
modeling via the redundant DCT dictionary. Finally, we draw
some interesting ties between this scheme and BM3D rationale
[11].
The paper is organized as follows: In Section II we in-
troduce the basic image processing scheme. In Section III
we explain how the performance of the basic scheme can be
improved using a subimage averaging scheme, and describe
the connection between the improved scheme and the BM3D
algorithm. In Section IV we explore the use of the proposed
approach to image denoising and inpainting, and present
experimental results that demonstrate the advantages of the
proposed scheme. We summarize the paper in Section IV with
ideas for future work along the path presented here.
II. IMAGE PROCESSING USING PATCH ORDERING
A. The Basic Scheme
Let Y be an image of size N
1
N
2
where N
1
N
2
= N,
and let Z be a corrupted version of Y, which may be noisy
or contain missing pixels. Also, let z and y be the column
stacked representations of Z and Y, respectively. Then we
assume that the corrupted image satises
z = My +v (1)
where the N N matrix M denotes a linear operator which
corrupts the data, and v denotes an additive white Gaussian
noise independent of y with zero mean and variance
2
. In
this work the matrix M is restricted to represent a point-
wise operator, covering applications such as denoising and
inpainting. The reason for this restriction is the fact that we
will be permuting the pixels in the image, and thus spatial
operations become far more complex to handle.
Our goal is to reconstruct y from z, and for this end we
employ a permutation matrix P of size N N. We assume
that when P is applied to the target signal y, it produces a
smooth signal y
p
= Py. We will explain how such a matrix
may be obtained using the image patches in Section II-B. We
start by applying P to z and obtain z
p
= Pz. Next, we take
advantage of our prior knowledge that y
p
should be smooth,
and apply a simple 1D smoothing operator H on z
p
, such
as 1D interpolation or ltering. Finally, we apply P
1
to the
result, and obtain the reconstructed image
y = P
1
H{Pz} . (2)
In order to better smooth the recovered image, we use an
approach which resembles the cycle spinning method [17].
We randomly construct K different permutation matrices P
k
,
utilize each to denoise the image z using the scheme described
above, and average the results. This can be expressed by
y =
1
K
K

k=1
P
1
k
H{P
k
z} . (3)
Fig. 1 shows the proposed image processing scheme. We next
describe how we construct the reordering matrix P.
permutations
inverse
permutations
1D
processing
1
P
K
P
z
1
1

P
1
K

P
y
1/ K H
H
Fig. 1: The basic image processing scheme.
B. Building the Permutation Matrix P
We wish to design a matrix P which produces a smooth
signal when it is applied to the target image y. When the
image Y is known, the optimal solution would be to reorder
it as a vector, and then apply a simple sort operation on the
obtained vector. However, we are interested in the case where
we only have the corrupted image Z. Therefore, we seek a
suboptimal ordering operation, using patches from this image.
Let y
i
and z
i
denote the ith samples in the vectors y and z,
respectively. We denote by x
i
the column stacked version of
the

n

n patch around the location of z

i
in Z. We assume
that under a distance measure
2
w(x
i
, x
j
), proximity between
the two patches x
i
and x
j
suggests proximity between the
uncorrupted versions of their center pixels y
i
and y
j
. Thus, we
shall try to reorder the points x
i
so that they form a smooth
path, hoping that the corresponding reordered 1D signal y
p
will also become smooth. The smoothness of the reordered
signal y
p
can be measured using its total-variation measure
y
p

TV
=
N

j=2
|y
p
(j) y
p
(j 1)|. (4)
Let {x
p
j
}
N
j=1
denote the points {x
i
}
N
i=1
in their new order.
Then by analogy, we measure the smoothness of the path
through the points x
p
j
by the measure
X
p
TV
=
N

j=2
w(x
p
j
, x
p
j1
). (5)
Minimizing X
p
TV
comes down to nding the shortest path that
passes through the set of points x
i
, visiting each point only
once. This can be regarded as an instance of the traveling sales-
man problem [15], which can become very computationally
expensive for large sets of points. We choose a simple approx-
imate solution, which is to start from a random point and then
continue from each point x
j0
to its nearest neighbor x
j1
with
a probability p
1
exp
_

w(xj
0
,xj
1
)

_
, or to its second nearest
neighbor x
j2
with a probability p
2
exp
_

w(xj
0
,xj
2
)

_
,
where is a design parameter, and x
j1
and x
j2
are taken
from the set of unvisited points.
We restrict the nearest neighbor search performed for each
patch to a surrounding square neighborhood which contains
B B patches. When no unvisited patches remain in that
neighborhood, we search for the nearest neighbor among all
the unvisited patches in the image. This restriction decreases
2
Throughout this paper we will be using variants of the squared Euclidean
distance.
3
0 100 200 300 400 500 600 700
0
0.005
0.01
0.015
0.02
0.025
0.03
Distances
P
r
o
b
a
b
ilit
y
0 10 20 30 40 50 60 70
0
0.01
0.02
0.03
0.04
0.05
0.06
0.07
Distances
P
r
o
b
a
b
ilit
y
(a) (b)
Fig. 2: Two normalized histograms of the spatial distances between
adjacent patches after the reordering, obtained for a noisy version
of the image Barbara with unrestricted (a) and restricted (b) search
areas. The lengths of the pathes in the spatial domain obtained with
the unrestricted and restricted searches were 4.2 10
7
and 6.1 10
6
,
respectively.
the overall computational complexity, and our experiments
show that with a proper choice of B it also leads to improved
results. The permutation applied by the matrix P is dened as
the order in the found path.
It is interesting to examine the characteristics of the patch
ordering in the spatial domain. To this end we apply the patch
ordering scheme described above to the patches of the noisy
Barbara image shown in Fig. 3(a) with both unrestricted and
restricted (B = 111) search neighborhoods, and with the
parameters

n = 8 and = 10
6
. We apply the obtained
permutations to the patches, and calculate two normalized
histograms of the spatial distances between adjacent patches,
shown in Figure 2. Fig. 2(a) shows that only about 3% of
neighboring patches in the path are also immediate spatial
neighbors,and that far away patches are often assigned as
neighbors in the reordering process. The histogram in Fig.
2(b) is limited to show only distances which are smaller or
equal to 78, the maximal possible distance within the search
window. It can be seen that despite the restriction to a smaller
search neighborhood, only about 6% of neighboring patches
in the path are also immediate spatial neighbors, and patches
all over the search neighborhood are assigned as neighbors in
the reordering process.
In order to facilitate the cycle-spinning method mentioned
above, we simply run the proposed ordering solver K times,
and the randomness (both in the initialization and in assigning
the neighbors) leads to different permutation results. We next
describe how the quality of the produced images may be
further improved using a subimage averaging scheme, which
can be seen as another variation of cycle spinning.
C. Subimage Averaging
Let X be an n (N
1


n + 1)(N
2


n + 1) matrix,
containing column stacked versions of all the

n

n
patches inside the image Z. We extract these patches column
by column, starting from the top left-most patch. When we
calculated P as described in the previous section, we assumed
that each patch is associated only with its middle pixel.
Therefore P was designed to reorder the signal composed of
the middle points in the patches, which reside in the middle
row of X. However, we can alternatively choose to associate
all the patches with a pixel located in a different position, e.g.,
the top left pixel in each patch. This means that the matrix P
can be used to reorder any one of the signals located in the
rows of X. These signals are the column stacked versions of
all the n subimages of size (N
1

n +1) (N
2

n +1)
contained in the image Z. We denote these subimages by Z
j
,
j = 1, 2, . . . , n. An example for two of them, Z
1
and Z
n
,
contained in a noisy version of the image Barbara, is shown
Fig 3(a).
We already observed in [12] and [13] that improved de-
noising results are obtained when all the n subimages of a
noisy image are employed in its denoising process. Here we
use a similar scheme in order to improve the quality of the
recovered image. In order to avoid cumbersome notations we
rst describe a scheme which utilizes a single ordering matrix
P. Let z
j
= R
j
z be the column stacked version of Z
j
, where
the matrix R
j
extracts the jth subimage from the image z. We
rst calculate the matrix P using the patches in X and apply
it to each subimage z
j
. Then we apply the operator H to
each of the reordered subimages z
p
j
= Pz
j
, apply the inverse
permutation P
1
on the result, and obtain the reconstructed
subimages
y
j
= P
1
H{Pz
j
} = P
1
H{PR
j
z}. (6)
We next reconstruct the image from all the reconstructed
subimages y
j
by plugging each subimage into its original
place in the image canvas and averaging the different values
obtained for each pixel. More formally, we obtain the recon-
structed image y as follows:
y = D
1
n

j=1
R
T
j
y
j
= D
1
n

j=1
R
T
j
P
1
H{PR
j
z} (7)
where the matrix R
T
j
plugs the estimated jth subimage into
its original place in the canvas, and
D =
n

j=1
R
T
j
R
j
(8)
is a diagonal weight matrix that simply averages the overlap-
ping contributions per each pixel. When K random matrices
P
k
are employed, we obtain the nal estimate by averaging
the images obtained with the different permutations
y =
1
K
K

k=1
_
_
D
1
n

j=1
R
T
j
P
1
k
H{P
k
R
j
z}
_
_
. (9)
This formula reveals two important properties of our
scheme: (i) the two summations that correspond to the two
cycle-spinning versions lead to an averaging of nK candidate
solutions, a fact that boosts the overall performance of the
recovery algorithm; and (ii) if H is chosen as linear, then the
overall processing is linear as well, provided that we disregard
the highly non-linear dependency of P on z.
D. Connection to BM3D
The above processing scheme can be described a little
differently. We start by calculating the permutation matrix P
from the image patches x
i
. We then gather the patches by
4
1
Z
n
Z
(a) (b)
Fig. 3: (a) Two subimages Z1 and Zn contained in a noisy version of
the image Barbara. (b) Classication of the pixels in a noisy version
of the image Barbara to centers of smooth (white) and non-smooth
(black) patches.
arranging them as the columns of a matrix X
p
in the order
dened by P. This matrix contains in its rows the reordered
subimages z
p
j
, therefore we next apply the operator H to its
rows, and shufe the columns of the resulting matrix according
to the permutation dened by P
1
. We obtain a matrix

X,
which contains in its rows the reconstructed subimages y
j
,
and in its columns reconstructed versions x
i
of the image
patches x
i
. We obtain the reconstructed image y from the
patches x
j
by plugging them into their original places in the
image canvas, and averaging the different values obtained for
each pixel. When K random matrices P
k
are employed, we
apply the aforementioned scheme with each of these matrices,
and average the obtained images.
Now, looking at the image processing scheme described
above, we can see some similarities to the rst stage of the
BM3D algorithm. Both algorithms stack the image patches
into groups, apply 1D processing across the patches, return the
patches into their place in the image, and average the results.
We note that the BM3D algorithm also applies a 2D transform
to the patches before performing 1D processing across them.
This feature can be easily added to our scheme if needed, and
we regard this as a preprocessing part of the operator H. On
the other hand, there are some key differences between the two
schemes. First, while the BM3D algorithm constructs a group
of neighbors for each patch, here we order all the patches
to one chain, which denes local neighbors. Furthermore,
this process is repeated K times, implying that our approach
consider K different neighbors assignments. Also, while in the
BM3D the patch order in each group is not restricted, ours is
carefully determined as it plays a major role in our scheme.
Finally, the 1D processing applied by the BM3D consists of
the use of a 1D transform, followed by thresholding and the
inverse transform, implying a specic denoising. Here we do
not restrict ourselves to any specic 1D processing scheme,
and allow the operator H to be chosen based on the application
at hand. We next demonstrate our proposed schemes for image
denoising and inpainting.
III. APPLICATIONS AND RESULTS
A. Image Denoising
The problem of image denoising consists of the recovery of
an image from its noisy version. In that case M = I and the
corrupted image satises z = y + v. The patches x
i
contain
noise, and we choose the distance measure between x
i
and x
j
to be the squared Euclidean distance divided by n, i.e
w(x
i
, x
j
) =
1
n
x
i
x
j

2
. (10)
In our previous works [12] and [13] we applied a complex
multi-scale processing on the ordered patches. Here we wish
to employ a far simpler scheme; we choose a 1D linear shift-
invariant lter, and as we show next, we learn this lter from
training images. Furthermore, we suggest to switch between
two such lters, based on the patch content.
We desire to treat smooth areas in the image differently than
areas with edges or texture, as our experiments show that this
approach leads to better results. More specically, we employ
different permutation matrices and lters in the smooth and
non smooth areas of the image. We rst divide the patches into
two sets: S
s
- which contains smooth patches, and S
e
- which
contains patches with edges or texture. Let std(x
i
) denote the
standard deviation of the patch x
i
and let C be a scalar design
parameter. Then we use the following classication rule: if
std(x
i
) < C then x
i
S
s
, otherwise x
i
S
e
. Fig 3(b)
demonstrates the application of this classication rule to the
noisy Barbara image shown in Fig. 3(a), where we use the
parameters

n = 8 and C = 1.2 which we will later use
in the denoising process of this image. White pixels are the
centers of smooth patches and black pixels are the centers of
patches containing texture or edges. It can be seen that the
obtained image indeed contains a rough classication of the
patches into smooth and non smooth sets.
We next divide each subimage z
j
into two signals: z
j,s
-
which contains the pixels corresponding to the smooth patches,
and z
j,e
- which contains the pixels corresponding to the
patches with edges and texture. We apply the nearest neighbors
search method described above to the patches in the sets S
s
and S
e
, and obtain two different permutation matrices P
s
and
P
e
. P
s
and P
e
extract from z
j
the signals z
j,s
and z
j,e
,
respectively, and then each apply a different permutation. We
apply P
s
and P
e
to z
j,s
and z
j,e
and obtain z
p
j,s
and z
p
j,e
,
respectively, which are the signals to which we apply the
lters. More formally,
_
z
p
j,s
z
p
j,e
_
=
_
P
s
{z
j,s
}
P
e
{z
j,e
}
_
=
_
P
s
P
e
_
z
j
= Pz
j
= z
p
j
(11)
where we dened the matrix
P =
_
P
s
P
e
_
. (12)
We next wish to nd the lters h
s
and h
e
applied to z
p
j,s
and z
p
j,e
, respectively. We denote the convolution matrices
corresponding to z
p
j,s
and z
p
j,e
by Z
p
j,s
and Z
p
j,e
, and obtain
the ltered subimages
y
j
= P
1
_
Z
p
j,s
h
s
Z
p
j,e
h
e
_
= P
1
_
Z
p
j,s
0
0 Z
p
j,e
_ _
h
s
h
e
_
= P
1
Z
p
j
h (13)
where we dened
Z
p
j
=
_
Z
p
j,s
0
0 Z
p
j,e
_
, h =
_
h
s
h
e
_
. (14)
5
The vector h stores the lter taps to be designed. We substitute
(13) in (7), and obtain the reconstructed image
y = D
1
n

j=1
R
T
j
P
1
Z
p
j
h. (15)
When K random matrices P
k
are employed, we obtain the
nal estimate by averaging the images obtained with the
different matrices
y =
1
K
K

k=1
_
_
D
1
n

j=1
R
T
j
P
1
k
Z
p
j
h
_
_
=
1
K
K

k=1
D
1
[R
T
1
, . . . , R
T
n
]
_

_
P
1
k
Z
p
1
.
.
.
P
1
k
Z
p
n
_

_h = Qh (16)
where we dened
Q =
1
K
K

k=1
D
1
[R
T
1
, . . . , R
T
n
]
_

_
P
1
k
Z
p
1
.
.
.
P
1
k
Z
p
n
_

_. (17)
Now let y
g
, j = 1, . . . , G be a training set which contains
the column stack versions of G clean images. For each such
image we create a noisy version z
g
by adding it noise with the
same statistics as the noise in z. Then we calculate for each
image z
g
a matrix Q
g
using (17), and learn the lters vector
h by minimizing

h = argmin
h
G

g=1
y
g
Q
g
h
2
=
_
G

g=1
(Q
g
)
T
Q
g
_
1
G

k=1
(Q
g
)
T
y
g
. (18)
Once we have the lters vector

h we can employ it to denoise
z by building Q using (17) and then calculating
y = Q

h. (19)
We can further improve our results by applying a second
iteration of our proposed scheme, in which all the processing
stages remain the same, but the permutation matrices are built
using patches extracted from the rst iteration clean result.
In order to assess the performance of the proposed image
denoising scheme we apply it to a test set containing noisy
versions of the images Lena, Barbara and House, with noise
standard deviations = 10, 25, 50. We learn the lters
vector h from a training set containing the images Man,
Peppers, Boat and Fingerprint. The parameters employed by
the proposed denoising scheme for the three noise levels are
shown in Table I. We note that the reason we chose a uniform
lter length of 25 samples for all noise levels can be justied
using Fig. 4. Fig. 4. shows the average of the PSNR values
obtained in the rst and second iterations for the 3 test images,
as a function of the lter length, for the different noise levels.
It can be seen that in both iterations the performance gain
obtained using lters longer than 25 samples is negligible.
The trained lters obtained in each iteration for the different
noise levels are shown in Fig. 5. First, it can be seen that
0 10 20 30 40 50
26
28
30
32
34
36
# Filter Length [samples]
P
S
N
R

[
d
B
]


=10
=25
=50
0 10 20 30 40 50
22
24
26
28
30
32
34
36
# Filter Length [samples]
P
S
N
R

[
d
B
]


=10
=25
=50
Fig. 4: Average of the PSNR values obtained for the 3 test images,
as a function of the lter length, for the different noise levels: (a)
First iteration. (b) Second Iteration.
0 5 10 15 20 25
0.2
0.1
0
0.1
0.2
0.3
0.4
0.5
Samples
A
m
p
lit
u
d
e


=10
=25
=50
0 5 10 15 20 25
0.2
0.1
0
0.1
0.2
0.3
0.4
0.5
Samples
A
m
p
lit
u
d
e


=10
=25
=50
0 5 10 15 20 25
0.04
0.02
0
0.02
0.04
0.06
0.08
0.1
Samples
A
m
p
lit
u
d
e


=10
=25
=50
0 5 10 15 20 25
0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
Samples
A
m
p
lit
u
d
e


=10
=25
=50
Fig. 5: The trained lters learned from noisy images: Left column -
the lters hs obtained in the rst (top) and second (bottom) iterations.
Right column - the lters he obtained in the rst (top) and second
(bottom) iterations.
lters h
s
and h
e
indeed look different. It can also be seen
that in the rst iteration the shape of the lter h
s
does not
change much as the noise level increases, and in the second
iteration the lters obtained for the higher noise levels are
similar, but very different from the lter obtained for = 10.
On the other hand, in both iterations the shape of the lter h
e
changes greatly as the noise level increases.
For comparison, we also apply the K-SVD algorithm [7].
The PSNR values of the results obtained with this algorithm,
and two iterations of our denoising scheme are shown in Table
II. The noisy and recovered images obtained with our scheme
for = 25 are shown in Fig. 6. First, it can be seen that the
second iteration improves the results of our proposed scheme
in all the cases but one. It can also be seen that the results
obtained with two iterations of our scheme are comparable to
the ones of the K-SVD for = 10, but are much better than
the ones of the K-SVD for = 25 and = 50.
B. Image Inpainting
The problem of image inpainting consists of the recovery
of missing pixels in the given image. Here we handle the case
where there is no additive noise, therefore v = 0, and M
is a diagonal matrix of size N N which contains ones
and zeroes in its main diagonal corresponding to existing
and missing pixels, correspondingly. Each patch may contain
6
PSNR=31.58 dB PSNR=31.81 dB
PSNR=30.46 dB PSNR=30.54 dB
PSNR=32.48 dB PSNR=32.65 dB
Fig. 6: Denoising results (PSNR) for the images Lena, Barbara and House ( = 25, input PSNR=20.18 dB): Left column - noisy images,
Center column - 1 iteration results, and Right column - 2 iterations results.
TABLE I: Parameters used in the denoising experiments.
Iteration K

n B C Filters Length
10
1 10 6 111 1.6 10
6
25
2 10 4 441 0.8 10
6
25
25
1 10 8 111 1.2 10
6
25
2 10 4 441 0.4 10
6
25
50
1 10 12 111 1.1 10
6
25
2 10 5 441 0.2 10
6
25
TABLE II: Denoising results (PSNR in dB) of noisy versions of
the images Lena, Barbara and House,obtained with the K-SVD
algorithm and the proposed scheme. For each image and noise level
the best result is highlighted.
Image Method
/PSNR
10/28.14 25/20.18 50/14.16
Lena
K-SVD 35.49 31.36 27.82
proposed (1 iter.) 35.33 31.58 28.54
proposed (2 iter.) 35.41 31.81 29.00
Barbara
K-SVD 34.41 29.53 25.4
proposed (1 iter.) 34.48 30.46 27.17
proposed (2 iter.) 34.46 30.54 27.45
House
K-SVD 36.00 32.12 28.15
proposed (1 iter.) 35.83 32.48 29.37
proposed (2 iter.) 35.94 32.65 29.93
missing pixels, and we denote by S
i
the set of indices of
non-missing pixels in the patch x
i
. We choose the distance
measure between patches x
i
and x
j
to be the average of
squared differences between existing pixels that share the same
location in both patches, i.e.
w(x
i
, x
j
) =

kSiSj
(x
i
[k] x
j
[k])
2
|S
i
S
j
|
. (20)
We start by calculating the matrix P according to the
scheme described in Section II-B, with a minor difference:
when a patch does not share pixels with any of the unvisited
patches, the next patch in the path is chosen to be its nearest
spatial neighbor. We next apply the obtained matrix to the
subimages z
j
, and observe that the permuted vectors z
p
j
= Pz
j
contain missing values. We bear in mind that the target signals
y
p
j
= Py
j
should be smooth, and therefore apply on the
subimages z
p
j
an operator H which recovers the missing values
using cubic interpolation. We apply the matrix P
1
on the
resulting vectors and obtain the estimated subimages y
j
. The
nal estimate is obtained from these subimages using (7). We
improve our results by applying two additional iterations of
a modied version of this inpainting scheme, where the only
difference is that we rebuild P using reconstructed (and thus
full patches).
We demonstrate the performance of our proposed scheme
on corrupted versions of the images Lena, Barbara and House,
obtained by zeroing 80% percents of their pixels, which are
selected at random. The parameters employed in each of the
three iterations are shown in Table III. In order to demon-
strate the advantages of our method over simpler interpolation
schemes we compare our results to the ones obtained by the
7
TABLE III: Parameters used in the inpainting experiments.
Iteration K

n B
1 10 16 9 10
2
2 10 8 43 10
4
3 10 5 55 10
8
TABLE IV: Inpainting results (PSNR in dB) of corrupted versions
of the images Lena, Barbara and House with 80 percents of their
pixels missing, obtained using triangle-based cubic interpolation
(tri. cubic), overcomplete DCT dictionary, 1 and 3 iterations of the
proposed scheme. For each image the best result is highlighted.
Image Method PSNR
Lena
tri. cubic 30.25
DCT 29.97
proposed (1 iter.) 30.25
proposed (2 iter.) 31.8
proposed (3 iter.) 31.96
Barbara
tri. cubic 22.88
DCT 27.15
proposed (1 iter.) 27.56
proposed (2 iter.) 29.34
proposed (3 iter.) 29.71
House
tri. cubic 29.21
DCT 29.69
proposed (1 iter.) 29.03
proposed (2 iter.) 32.1
proposed (3 iter.) 32.71
matlab function griddata which performs cubic interpolation
of the missing pixels based on Delaunay triangulation [18],
[19]. We also compare our results to the ones obtained using
the algorithm described in chapter 15 of [16], which employs
a patch-based sparse representation reconstruction algorithm
with a DCT overcomplete dictionary to recover the image
patches. We use a patch size of 16 16 pixels in order to
improve the results this method produces. We note that we
do not employ the K-SVD based algorithm which was also
described in this chapter, as our experiments showed that it
produces comparable or only slightly better results than the
redundant DCT dictionary, at a higher computational cost.
The PSNR values of the results obtained with the different
algorithms are shown in Table IV. Fig.7 shows the corrupted
and the reconstructed images, with the corresponding PSNR
values, obtained using triangle-based cubic interpolation, over-
complete DCT dictionary, 1 and 3 iterations of the proposed
scheme. First, it can be seen that the second and third iterations
greatly improve the results of our proposed algorithm. It can
also be seen that the results obtained with three iterations of
our proposed scheme are much better than those obtained with
the two other methods.
C. Computational Complexity
We next evaluate the computational complexity of a single
iteration of the two image processing algorithms described
above. We note that for the image denoising scheme, we
assume that the lters training has been done beforehand, and
exclude it from our calculations. First, building the matrix X
which contains the image patches requires O(nN) operations.
We assume that when the nearest-neighbor search described
above is used with a search window of size B B, most of
the patches do not require to calculate distances outside this
neighborhood. Therefore, as calculating each of the distance
measures (10) and (20) requires O(n) operations, the number
of operations required to calculate a single reordering matrix
P
k
can be bounded by O(NB
2
n). Next, applying the matrices
P
k
and P
1
k
to the n subimages z
j
require O(nN) operations,
and so does applying either one of the operators H described
above to the n subimages z
p
j
. Finally , constructing an estimate
image by averaging the pixel values obtained with the different
subimages also requires O(nN) operations, and averaging the
estimates obtained with the different matrices P
k
requires
O(KN) operations. Therefore when K permutation matrices
are employed, the total complexity is
O((n +K)N) +K
_
O(NB
2
n) +O(nN)

= O(NKB
2
n)
(21)
operations, which means that, as might be expected, the overall
complexity is dominated by the creation of the permutation
matrices. For a typical case in our experiments, N = 512
2
,
K = 10, n = 64 and B = 111, and the above amounts to
1.86 10
10
operations. We note that while in our experiments
we employed exact exhaustive search, approximate nearest
neighbor algorithms may be used to alleviate the computa-
tional burden.
IV. CONCLUSIONS
We have proposed a new image processing scheme which is
based on smooth 1D ordering of the pixels in the given image.
We have shown that using a carefully designed permutation
matrices and simple and intuitive 1D operations such as linear
ltering and interpolation, the proposed scheme can be used
for image denoising and inpainting, where it achieves high
quality results.
There are several research directions to extend this work
that we are currently considering. The rst is to make use
of the distances between the patches not only to nd the
ordering matrices, but also in the reconstruction process of
the subimages. These distances carry additional information
which might improve the obtained results. A different direction
is to develop new image processing algorithms which involve
optimization problems in which the 1D image reorderings act
as regularizers. These may both improve the image denoising
and inpainting results, and allow to tackle other applications
such as image deblurring, where the operator M is no longer
restricted to be point-wise local. Additionally, the proposed
image denoising scheme may be improved by dividing the
patches to more than two types, and treating each type differ-
ently. Finally, we note that in our work we have not exhausted
the potential of the proposed algorithms, and the choice of
different parameters (e.g., B, ) for each set of patches may
also improve the produced results.
REFERENCES
[1] A. Buades, B. Coll, and J. M. Morel, A review of image denoising
algorithms, with a new one, Multiscale Modeling and Simulation, vol. 4,
no. 2, pp. 490530, 2006.
[2] P. Chatterjee and P. Milanfar, Clustering-based denoising with locally
learned dictionaries, Image Processing, IEEE Transactions on, vol. 18,
no. 7, pp. 14381451, 2009.
8
PSNR= 6.65 dB PSNR=30.25 dB PSNR=29.97 dB PSNR=30.25 dB PSNR=31.96 dB
PSNR= 6.86 dB PSNR=22.88 dB PSNR=27.15 dB PSNR=27.56 dB PSNR=29.71 dB
PSNR= 5.84 dB PSNR= 29.21 dB PSNR= 29.69 dB PSNR= 29.03 dB PSNR= 32.71 dB
Fig. 7: Inpainting results (PSNR) of corrupted versions of the images Lena, Barbara and House with 80% of their pixels missing (input
PSNR=6.41 dB), obtained with different reconstruction methods: : First column - corrupted images, Second column - triangle-based cubic
interpolation, Third column - overcomplete DCT dictionary, Fourth column - 1 iteration of the proposed scheme, Fifth column - 3 iterations
of the proposed scheme.
[3] G. Yu, G. Sapiro, and S. Mallat, Image modeling and enhancement via
structured sparse model selection, in Image Processing (ICIP), 2010
17th IEEE International Conference on. IEEE, 2010, pp. 16411644.
[4] , Solving inverse problems with piecewise linear estimators: from
gaussian mixture models to structured sparsity, Image Processing, IEEE
Transactions on, no. 99, pp. 11, 2010.
[5] W. Dong, X. Li, L. Zhang, and G. Shi, Sparsity-based image denoising
via dictionary learning and structural clustering, in Computer Vision
and Pattern Recognition (CVPR), 2011 IEEE Conference on. IEEE,
2011, pp. 457464.
[6] D. Zoran and Y. Weiss, From learning models of natural image patches
to whole image restoration, in Computer Vision (ICCV), 2011 IEEE
International Conference on. IEEE, 2011, pp. 479486.
[7] M. Elad and M. Aharon, Image denoising via sparse and redundant rep-
resentations over learned dictionaries, IEEE Trans. Image Processing,
vol. 15, no. 12, pp. 37363745, 2006.
[8] J. Mairal, M. Elad, G. Sapiro et al., Sparse representation for color
image restoration, IEEE Transactions on Image Processing, vol. 17,
no. 1, p. 53, 2008.
[9] J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman, Non-local
sparse models for image restoration, in Computer Vision, 2009 IEEE
12th International Conference on. IEEE, 2009, pp. 22722279.
[10] R. Zeyde, M. Elad, and M. Protter, On single image scale-up using
sparse-representations, Curves and Surfaces, pp. 711730, 2012.
[11] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, Image denoising by
sparse 3-d transform-domain collaborative ltering, IEEE Trans. Image
Processing, vol. 16, no. 8, pp. 20802095, 2007.
[12] I. Ram, M. Elad, and I. Cohen, Generalized Tree-Based Wavelet
Transform , IEEE Trans. Signal Processing, vol. 59, no. 9, pp. 4199
4209, 2011.
[13] , Redundant Wavelets on Graphs and High Dimensional Data
Clouds, IEEE Signal Processing Letters, vol. 19, no. 5, pp. 291294,
2012.
[14] G. Plonka, The Easy Path Wavelet Transform: A New Adaptive Wavelet
Transform for Sparse Representation of Two-Dimensional Data, Mul-
tiscale Modeling & Simulation, vol. 7, p. 1474, 2009.
[15] T. H. Cormen, Introduction to algorithms. The MIT press, 2001.
[16] M. Elad, Sparse and Redundant Representations: From Theory to
Applications in Signal and Image Processing. Springer Verlag, 2010.
[17] R. R. Coifman and D. L. Donoho, Wavelets and Statistics. Springer-
Verlag, 1995, ch. Translation-invariant de-noising, pp. 125150.
[18] T. Yang, Finite element structural analysis. Prentice-Hall, 1986, vol. 2.
[19] D. Watson, Contouring: a guide to the analysis and display of spatial
data: with programs on diskette, 1992.