A METHOD FOR REASSEMBLING FRAGMENTS IN IMAGE RECONSTRUCTION

Francesco Amigoni, Stefan0 Gazzuni, Simone Podico

Artificial Intelligence and Robotics Laboratory; Dipartimento di Elettronica e Informazione

Politecnico di Milano; Piazza Leonard0 da Vinci 32,20133 Milano, Italy

ABSTRACT 2. REASSEMBLY O F T W O FRAGMENTS

The reassemblyof fragments to reconstruct images and objects is 2.1. Fragments Analysis
a problem often encounteredin several applications, ranging from
archeology to medicine. In this paper we present a novel method In this subsection we describe the step (I)above. The input to
for reassemblingtwo 2D fragments. The proposedmethod is based our method i s represented by two rectangular bitmap images of
on the information extracted from the outlines and from the color the fragments (havingclosed boundary curves) over a background.
contents of the fragments, without relying on any knowledge of The method requires two images with the same scale; this means
the final image. The techniques illustrated in this paper constitute that they must be digitally acquired from the same distance and
the core of a more general method for reassembling n fragments viewing angle.
we are developing. However, here we concentrate mainly on the Given the two images, we preliminary separate the fragments
description of the algorithms for matching two fragments and on from the background we use afloodflill based algorithm [4] that
their experimental analysis and validation. labels every pixel as fragment pixel or backgroundpixel.
The pixels that compose the border o f a fragment arc then
located as those belonging to the fragment that are adjacent to
1. INTRODUCTION a background pixel. We employ a simple algorithm that clock-
wise follows the border o f the fragment by moving from one bor-
The reassembly o f fragments to recompose images and objects der pixel to another: the next border pixel i s located by searching
i s a problem occurring in a number of fields: in archeology [I], among the eight neighbors of the current border pixel. The result
in art restoration 121, and in other disciplines including forensics, is a border urray containing the coordinates and the color o f every
computer-aided design, chemistry, and medicine [3]. pixel belonging to the fragment’s border.
In this paper we present a novel method for reassembling two To obtain a rotation- and translation-independentmodel o f the
2D fragments. We give only few preliminary considerations on border, we use the local curvature (as in [ 5 ] and 161). The local
the more general problem (on which we arc currently working) o f curvature o f the border in a pixel p is calculated as the magnitude
integrating n.fragments to reconstruct a complete image. o f the derivative o f the local tangent i n p. To make the computed
The outline o f the proposed method for reassemblingtwo frag- curvature independent from the images resolution, we introduce
ments follows. the parameter D that represents the distance from p to the two
neighboringborder pixels used to calculate the local tangent in p .
(I) Analysis o f the two fragments and extraction of their rep- The local curvature of the border pixels of the first and the second
resentations. A fragment i s representedby the sequence o f fragment are stored in real-values cumuIure mruys, called Shupel
its border pixels with associated information about coordi- and Shupe2, respectively.
nates, color, and local curvature. Since the number of pixels in two segments with the same
physical length i s different if the segments are not parallel, we
(2) Comparison o f the two sequences to identify the common use an adjusting algorithm that stretches the border and curvature
subsequences (with similar local curvature and satisfying arrays adding interpolated pixels depending on the slope o f the
constraints on color contents). The common subsequences local tangent.
represent the candidate matchingportions of the borders.
(3) Calculationo f the transformation (translation and rotation) 2.2. Fragments Match
that brings the first fragment to join the second one along
the best (longest) matchingportion of the borders. In this subsectionwe describe the steps (2) and (3) above. In order
to find the longest common portion of the borders, we developed
This paper i s structured as follows. In Section 2 we detail our an algorithm called runk dgorirhm, listed in Fig. 2, that represents
method for reassembling two fragments o f a 2D image. Section 3 the foundational part o f our method and the main contribution of
shows some of the results obtained from the experimental analy- this paper.
sis that validated the proposed method. In Section 4 we critically We make two preliminary considerations before describing the
compare our method with other similar techniques. Finally, Sec- core of the tank algorithm. First, one of the two curvature arrays
tion 5 concludes the paper. has to be flipped because the border pixels in both images have
been found in clockwise order. Second, since the fragments have
This paper is dedicated to the memnry of Marcn Samalvico, who has close boundary curves, we consider the curvature arrays as circular
been b t h teacher and friend. by duplicating them.

Authorized licensed use limited to: Ankara Bilim University. Downloaded on September 21,2021 at 10:37:44 UTC from IEEE Xplore. Restrictions apply.
 Require: Length(ShapeI)>Lenglh(Shapea)
ShapelDup t Duplicate(Shape1)
ShapeZDup c Duplicate(Flip(Shape2))
f o r i = 0 to Length(Shape1)do
MaxCommanLengthc 0; Counting t FALSE; ErrarTank c 0
for j = 0 to Length(Shapr2Dup)do
e t IShapel[(i+j) mod Length(ShapelDiip)l.Curvature~
ShapeZDupU1,Curvaturel {compute the error)
i f ( e < E ) v (Counting) then
ErrorTank t ErrorTaank+e
Counting t TRUE
i f ErrnrTank < ErrarTankCapacitythen
i r e < E then
Fig. 1. Various placements of the arrays ErrorTank t ErrorTank - R I

end if
-
el% i f P < 4 . E then
Errofrank ErrorTank .Rz
The tank algorithm requires that Shape1 is longer than Shupe2. {increase the common portion length)
The outer loop cycles the different placements of the curvature CommanLength c CommonLengthtl
arrays, as shown in Fig. 1. Initially they are positioned as in A, &e
then Shape2 is shifted by one position (B), and so on until the end { Erroilank overflowed}
if CommonLength > MaxCammonl.ength then
of the loop (C). The number o f evaluated placements is equal to {refine the candidate common portion}
the length 11 ofShupel. CammonLength c Trim(CommonPonion)
For each single placement, the curvature arrays are compared end if
element by element (pixel by pixel) from left to right (the double if CommonLength > MaxCommonLengththen
i f ColorTest=TRUE then
arrows of Fig. I). The total number of element comparisons for
{coiarcheck ok: new longest common partian found}
each placement is equal to the double of length 12 of Shupe2. This MaxCommonLength t CommonLength
means that the worst-case complexity of the tank algorithm (not R..ult[i] + CommonPortionData
considering the color test, see below) is O(1112).For each element end if
comparison the algorithm calculates the absolute value e o f the end if
difference of the two corresponding curvatures. When e is minor CommonLength c 0; Counting t FALSE ErrorTank c 0
than the parameter E , the curvatures are very similar and a common end i f
end i f
portion of the borders starts. When a common portion has started, end far
the e computed for the following elements is added to ErrorTank end for
(initially set to 0). When Errorrank exceeds the parameter Error-
TunkCupucily, the current common portion is ended. While exam- Fig. 2. The tank algorithm
ining a common portion, e is not only added to ErmrTonk but also
contributes to reduce the ErmrTunk value in particular cases. We
use two parameters called RIand R2. When e < E the Errorrank
The tank algorithm produces the array Result that contains in-
is reduced by being multiplied by RI (0 5 R I 5 1). otherwise
if e < 4 . E , ErmrTuonk is reduced by being multiplied with Rz formation about the longest common portion found for each place-
ment o f the borders. At the top of Fig. 3 is shown the normalized
(0 5 RI 5 Rz 5 1). Typical values for the two paranients are
graph of the length of the common portions for all possible place-
0.5 for RI (ErmrTonk becomes the 50% of its old value) and 0.8
ments of two fragments. For each common portion, call 1p the
(ErmrTunk becomes the 80% of its old value) for Rz. RI and Rz
angle by which the first fragment must be counterclockwise ro-
give a sort of reward when the curvature of two pixels are almost
tated to align with the second one along the common portion of
equal (RI) or enough similar (Rz). This makes the tank algorithm
the borders. This angle is calculated by comparing the slopes of
very robust to sporadic differences in the common portion of the
the segments that join the start and end points of the common por-
borders. Finally a refinement o f the common portion is done: the
tion in the two fragments. The length of the common portions for
elements from the end are trimmed until the error e is less than E
each angle 1p forms a new array of 360 elements (it is possible to
(this is necessary because. at the end of the search for the common
have approximations more accurate than one degree by using more
portion, the curvatures were different and ErmrTank overflowed).
elements). An example of this (normalized) array is shown at the
Once a common portion has been found, we perform a color bottom of Fig. 3; it is possible to notice that the best rotation angle
test by applying a modified version of the above part of the tank is 198O. The translation is simply calculated in order t o superim-
algorithm to the border arrays containing information about the pose the common portions of the borders.
RGB color values of the border pixels. The variable e now is
the Euclidean distance (in three-dimensional space RGB) between
the points that represent the colors of the corresponding pixels 3. EXPERIMENTAL RESULTS
(in the current common portion). If, during the coliir test, Error-
Tank exceeds the ErrorTunkCupucity the common portion is dis- In this section we present some o f the experimental results ob-
carded due to incompatibility in color of adjacent pixels. Other- tained from the implementation of the method described above.
wise the common portion is accepted and, if it is the longest for The first prototypes of the algorithms have been coded in Delphi
the current placement, it is labelled with its length (in pixel) and to preliminary assess the effectiveness of the method and to an-
with a weight that represents the color diversity, given by Ermr- alyze the role of the different parameters involved, working in a
Tonk/&mrT=nnkCupuciN. CUI environment that could be quickly modified. Given this non-

Ill - 582

Authorized licensed use limited to: Ankara Bilim University. Downloaded on September 21,2021 at 10:37:44 UTC from IEEE Xplore. Restrictions apply.
 Fig. 6. Two fragments of a jigsaw puzzle taken from the Web
site of the movie “Le fabuleux destin d’AmClie Poulain” and the
reassembled puzzle
rlDnn

Fig. 3. Length of the common portions for the placements (top)

and for the rotation angles (bottom) of the fragments of Fig. 4

Fig. 4. Two fragments of an image of Piazza dei Miracoli at Pisa

(Italy)

Fig. 1. Two fragments of an image o f Pisa (Italy) that are not

optimized implementation, we did not make a deep complexity correctly reassembled
analysis of our method.
The integration o f the two fragments of Fig. 4 is reported in
Fig. 5. Although it represents only a particular application, our common portion of the borders even when RI = Rz = 1(namely,
method has proven to be effective also in reassembling fragments when the tank level is never reduced) given that E is large enough.
ofjigsaw puzzles as shown in Fig. 6. Fig. 7 shows an example in For example, when E > 5 for the fragments of Fig. 4. Conversely,
which our method fails. In this case, the two fragments clearly can- if the tank level can be reduced (0 5 RI 5 Rz < l),then the cor-
not be correctly matched and the method returns the best match. rect common portion is found also for small E . For example, for
However, the result appears absolutely reasonable when the shape E = 1, RI = 0.5, and Rz = 0.8 for the fragments of Fig. 4. Ex-
and the color content of the fragments are considered (see the perimentally, it results that the longest correct common portions
clouds!). Only humans can recognize the erroneous reassembly of the borders (namely the hest matches) are often found when
because of the semantic content of the fragments. 10 5 E 5 30 and that, in these cases, the influence of the values of
The values of the parameters used in the above reassembling RI and Rz is small. Finally, the value of D depends on the resolu-
examples have been: D = 10, E = 10, R, = 0.5, RZ = 0.8, tion (the larger the resolution the larger D)and on the irregularities
and ErmrEhnkCapacity= 0.512 (remember that 12 is the length of in the borders.
the second, shorter, fragment). The time required (on a 300MHz In conclusion, the proposed method for reassembling frag-
Pentium I1 computer) for matching the previous fragments ranges ments has shown to he robust in producing the correct result also
from 1s to 5s. when some little pieces of the boundaries are missing.
It is impossible to report here the complete experimental anal-
ysis on the role of parameters in our method, but some main facts
are worth to be discussed. The tank algorithm correctly finds the 4. RELATED WORKS

A number of methods have been developed to reassembly frag-

ments in order to recompose both two-dimensional images and
three-dimensional objects. In the case of 2D images, when the
final result of integration is not known, the only information avail-
able to the methods is obtained from the fragments themselves.
Roughly, this information is related to the boundaries (edges or
fracture lines) or to the content of the fragments (their colors or
drawings).
The method presented in this paper exploits information about
Fig. 5. The reassembled image o f Pisa (Italy) both boundaries and color content of fragments. Another effec-

111 - 583

Authorized licensed use limited to: Ankara Bilim University. Downloaded on September 21,2021 at 10:37:44 UTC from IEEE Xplore. Restrictions apply.
 tive method but based only on boundary information is presented
in [6]. The outlines of the fragments are represented by curva-
ture strings (as in [SI), namely by a circular sequence of samples,
where each sample is the local curvature of the outline at the cor-
responding point. This representation is used also in our method.
The matches between fragments are found by a multiscale proce-
dure; while the refinement of matches is performed by a dynamic
programming algorithm. In our method we do not employ a mul-
tiscale procedure for matching the fragments and, in the current
version, we match only two fragments; thus we cannot directly
compare the two methods. Fig. U Two imagcs ofthe \ ~ I l Contmni-Camcrtni
a at Piuzzola rul
Although the representation is similar, our method for match- B r m a (1131)) rca<icmbled frum 1\10 *ots o f 5 ilcfi) and I; [right)
ing the outlines of the fragments differs from the one presented in fmgmenti
[5]. The latter manages noise by considering two curvature values
as equal if there is at most an e-difference between them (where
e is a “locall- measure of accuracy), while the former considers fragment is left. We aim to improve this elementary method for
a “global” measure of accuracy as implemented in the tank algo- reassembling n fragments by pre-classifying the fragments in the
rithm. Our method deals more robustly with noise since it allows pool (for example according to their color contents) and to experi-
scattered pairs of significantly different curvature points within a mentally test the method in real-case applications.
sequence of pairs of significantly equal curvatme points. (Note
that our method essentially reduces to that in [ 5 ] when RI =
Rz = 1.)The worst-case complexity ofthe method in [ 5 ] is O ( n 2 ) 6. REFERENCES
(where n is the length of the longer border representation), which
[ I ] M. Levoy, “Forma urbis romae project,” 2003, http:
is comparable to that of our method.
//formaurbis . stanford.edu/, last accessed January
Hori et al. [7] propose a method for detecting the joint among
Sth, 2003.
two 2D potsherds. Their method uses a segmentation of the con-
tours of the fragments and, then, finds match between pairs of seg- [2] Guide to the Recovery, Recomposition, and Restoration of
ments. Its applicability (and that of similar techniques) is limited Shattered Wall Paintings: Experience Gained at the Basilica
to the cases in which the fragments possess critical points (i.e., di San Francesco in Assisi, ICR, Rome, Italy, 2001.
comers) on which the segmentation can be based. Our method is,
in this respect, more general and a direct comparison is difficult. [3] G. Barequet and M. Sharir, “Parlial surface and volume
The object recognition techniques (see [8] for a survey) can matching in three dimensions,” IEEE Transactions on Pattern
provide hints to represent the boundaries and to develop algorithms Analysis and Machine Intelligence, vol. 19, no. 9, pp. 929-
for their matching. However, the object recognition problem is 948, September 1997.
structurally different from the fragment reassembly problem. In
the former it is usually the case when an object is known (the [4] MIT Computer Graphics Group, “lntro to computer
model) and the goal is to find, by matching, the occurrences of graphics:’ 2003, http://graphics.lcs.mit.edu/
the object in a scene. The fragment reassembly problem is more classes/6.837/FOl/Lecture03/index.html,last
“symmetric”, since the fragments to be matched are not subject to accessed January Sth, 2003.
any model-instance relation. Finally, although addressing a simi-
[5] H. Wolfson, “On curve matching:’ IEEE Transactions on
lar problem, the most effective techniques for automatic assembly
Pattern Analysis and Machine Intelligence, vol. 12, no. 5 , pp.
ofjigsaw puzzles (like [9]) cannot be directly applied to fragment
4 8 3 4 8 9 , May 1990.
reassembly since they usually heavily rely on the specific shape of
jigsaw puzzle fragments from which invariants are extracted. [6] H. C. de Gama Leiso and J. Stolfi, “A multiscale method for
the reassembly of two-dimensional fragmented objects,” IEEE
5. CONCLUSIONS Transactions on Pattern Analysis and Machine Intelligence,
vol. 24, no. 9,pp. 1239-1251, September2002.
In this paper we have presented some results of a novel method
171 K. Hori, M. Imai, and T. Ogasawara, “Joint detection for
for reassembling two fragments of a 2D image. The method has
potsherds o f broken earthenware,” in Proceedings of the
proved to be effective and robust in several experimental settings.
IEEE Conference on Computer Yision and Patlern Recogni-
Since our goal was to assess the validity of the proposed approach,
tion (CVPR), June 1999, vol. 2, pp. 440-445.
we made no attempt to optimize the algorithms but we aim to im-
prove their performance by implementing a more efficient C++ [8] A. Pope, “Model-based object recognition: A survey of recent
version of the code. Another direction of future work is to employ research:’ Tech. Rep. TR-94-04, University of Califomia at
the method presented in this paper to address thc reassembly of n. Berkeley, Berkeley, CA, USA, January 1994.
fragments. Some preliminary results are shown in Fig. 8. The re-
assembled images have been obtained by a quite simple algorithm: [9] G. Budea and H. Wolfson, “Solving jigsaw puzzles by a
given a p w l of fragments, we look for the two fragments with the robot,” IEEE Transactions on Robotics and Autonration, vol.
best match (the longest common portion of borders), we add to 5, no. 6, pp. 752-764, December 1989.
the pool the new fragment obtained by joining the two fragments
(which are removed from the pool), and we continue until a single

111 - 584

Authorized licensed use limited to: Ankara Bilim University. Downloaded on September 21,2021 at 10:37:44 UTC from IEEE Xplore. Restrictions apply.