Assignment 4

CAP 6419 - 3D Computer Vision

Fall 2013
Due 11-12-2013
Edward Aymerich
1 Implemented methods
1.1 Loading images
The function loadImg() was implemented to ease the loading of several images.
This function takes a string indicating the name of the images to load (for
example: test*.jpg) and returns a cell array with the images, ordered by
name and number (if any). Wildcards must be used in the le name to load
several images.
In order to correctly order numbered images, loadImg() uses the function
sort nat() from Douglas M. Schwarz
1
.
The function loadImg() has the limitation that it can only load images from
the current directory. If the le name contains a path (relative or absolute),
loadImg() will fail to load the images.
1.2 Feature point correspondence
The function ndHomography() was implemented to nd feature points in a
pair of images, and then match such points. The implementation used MAT-
LAB functions detectSURFFeatures() and extractFeatures()
2
to identify feature
points in the input images, and then the MATLAB function matchFeatures()
creates a correspondence between the feature points in both images.
Finally, the format of the matching points returned by matchFeatures() is
changed to nd an homography.
Originally, the number of feature points was limited to 100, i.e., if more than
100 points were found, then only the 100 points with the strongest features were
used. However, it was found that using this amount of points resulted in very
dierent homographies each time the algorithm was run (because of the random
nature in the RANSAC method used later). Increasing the amount of feature
points used also increased the stability of the homography, so in the current
implementation there is no limit to the amount of points used: all found feature
points are matched and used to compute the homography.
1
Available at: http://www.mathworks.com/matlabcentral/fileexchange/
10959-sortnat-natural-order-sort
2
Part of the Computer Vision System Toolbox.
1
1.3 Compute Innite Homography
The function ransacthomography() from Peter Kovesi
3
is used to estimate the
best homography for the matched feature points. This function uses RANSAC
to t a 2D homography. A value of 0.01 was used as distance threshold to decide
is a point is an inlier or not.
The call to ransacthomography() is done inside the implemented function
ndHomography(), so after it calculates the matching points, uses ransactho-
mography() to t the homography and then returns such homography.
1.4 Warping of images
With the tted homography, one of the input images must be warped to t into
the other image. Such warping is done by MATLAB function imtransform().
One problem with this function is that it automatically centers the warped
image, so any translation in our homography is lost, and since the objective
is to do a nal mosaic, then there is a high chance that we have translations
between the images.
To recover the lost translation information, imtransform() is called in the
following way:
[A

, xdata, ydata] = imtransform(A, H);

With A being the image to warp, H the tted homography, A

is the warped
image, and xdata and ydata store the translation information, which will be used
when the mixing of images is calculated.
1.5 Mixing images
An easy way to mosaic the images is to take the reference image (the one
that wasnt warped) and copy the warped image over it, using the translation
information to position the warped image correctly. This approach has one big
problem: the superimposing of images is very easy to see, and this is seen as
artifacts in the resulting mosaic:
3
Available at: http://www.csse.uwa.edu.au/
~
pk/Research/MatlabFns/
2
To prevent this, a mask is applied to each image before it is added into the
mosaic. To create such mask, an initial mask is created for each image, in a way
that it has the greater value at the center, and decreases linearly to the borders.
The value of all masks is added into the mosaic mask, and then the nal mask
for each image is determined by dividing its initial mask by the mosaic mask.
Simple mosaic of images A and B. Mosaic mask.
Initial mask for A. Initial mask for B.
Final mask for A. Final mask for B.
Final mosaic.
The initial mask for each image is calculated by the implemented function
3
getMask(), which calculates the inverse distance of each pixel to the image center
in each axis (X and Y), measuring this inverse distance as 1 in the center and
0 close to the borders. The nal value for each pixel in the mask is its inverse
X distance times its inverse Y distance.
The functions immix2() and immix fast() were implemented to handle the
mixing of images. immix2() is just a wrapper around immix fast() that facil-
itates the passing of parameters. This functions do not apply any warping to
the images, they only mix two images at given coordinates using the procedure
described before.
A initial implementation of this method was done in the function immix().
Due to the authors inexperience with MATLAB, a lot of for cicles were used
to calculate the masks, and some values were calculated twice, in the hopes
that recalculating some simple values was faster than storing and retrieving
them from memory (as it happens with some algorithms implemented in C).
However, immix() was very inecient, and most of the time of the whole mo-
saicing process was spend only mixing the images. The function immix fast()
is a re implementation of immix(), using no loops and relying in MATLABs
vectorization.
Using MATLABs Proler to measure the eciency of immix() and im-
mix fast(), it was found that the speedup of the new function is very signicant.
Using 5 images to produce a mosaic took 13.039 seconds using immix(), of which
12.519 seconds are used by immix(). In contrast, using immix fast() to produce
the same mosaic took 0.591 seconds, of which only 0.088 seconds were spend in
immix fast(). This represents a speedup of 22, with immix fast() using about
15% of the total time, which is a big improvement over the 96% of the total
time used by immix().
1.6 Creation of the mosaic
To put all the previous steps together, the function imfuse() was implemented.
This function receives two images, say A and B, and calls ndHomography()
to estimate the homography H that match A into B. Then it warps A into
A

using H and imtransform(), and nally mix A

with B using immix2() and

returns the resulting image.
Up to this point the process of warping and mixing two images together has
been described. One method to mosaic several images could be to estimate the
homography H between each pair of images, and then use backward mapping
methods to estimate the homography that match each image into the reference
image. This implementation doesnt follow this method directly, instead using
a dierent approach.
The function mosaic() was implemented to handle the mosaicing of several
images. To do this, mosaic() rst select the middle image in the cell array of
images given to it. Then it creates a mosaic mixing images from the beginning
of the cell array up to the middle, and another mosaic from the end of the
cell array up to the middle. Finally the two mosaics a mixed together into the
resulting mosaic. mosaic() allows the user to dene how many images around
4
the center (called neighbors) will be used in the nal mosaic. If no amount of
neighbors is dened, then mosaic() will use all the images.
To explain how mosaic() works, lets follow a small example. Suppose that
we want to mosaic 5 images, labeled [A,B,C,D,E], and we want to use C as
reference. mosaic() rst fuses images A and B, transforming A to t into B
and leaving B unwarped. Lets call this new image AB. Now image AB is fused
with C, warping AB to t into C and leaving C unwarped and producing image
ABC. A similar process, but in backward direction, happens on the other side:
E is warped into D to form ED, and ED is warped into C to produce EDC.
Finally, ABC and EDC are fused together. Since in both images C was not
warped, C will remain unwarped in the nal mosaic, serving as reference for the
other images.
To see why this method works, lets consider the process from As viewpoint.
In theory, if we estimate the homography H
AB
that warps A into B and the
homography H
BC
that warps B into C, then an homography H
AC
that warps A
into C can be calculated as H
AC
= H
AB
H
BC
. In our method, since we already
warped A into AB, then we dont have to worry about estimating H
AC
, because
A is already in Bs space. When AB is warped into C, it will carry on the
transformation applied to A into C space. In this method, the transformations
are not explicitly accumulating in a homography, but they are accumulating in
the images. Therefore we must only worry about nding the correct homography
to t the current image (which is the accumulation of previous images) into the
next image. An example of this process follows:
Image A Accumulated image A.
Image B Accumulated image AB.
Image C Accumulated image ABC.
5
Notice how image A is increasingly distorted as more and more images are
introduced into the accumulated image. This is due to the transformations
necessary to warp A into reference image C.
2 Results
2.1 Using set mov2
The set mov2 was provided for this homework. It consist of 21 pictures
apparently taken from the main entrance of the Harris Engineering Corporation
building at UCF.
mov2b 7 mob2b 8 mob2b 9 mob2b 10 mob2b 11
mov2b 12 mob2b 13 mob2b 14 mob2b 15 mob2b 16
mov2b 17 mob2b 18 mob2b 19 mob2b 20 mob2b 21
mov2b 22 mob2b 23 mob2b 24 mob2b 25 mob2b 26
mov2b 27
Using a dierent number of neighbors around the center image (mob2b 17)
the following mosaics were calculated:
6
Using 0 neighbors. (1 picture).
Using 1 neighbors.(3 pictures).
Using 3 neighbors. (7 pictures).
Using all set. (21 pictures).
7
2.2 Using set mov3
The set mov3 was provided for this homework. It consist of 17 pictures of an
indoor environment.
mov3 1 mov3 2 mov3 3 mov3 4 mov3 5
mov3 6 mov3 7 mov3 8 mov3 9 mov3 10
mov3 11 mov3 12 mov3 13 mov3 14 mov3 15
mov3 16 mov3 17
Using a dierent number of neighbors around the center image (mob3 9)
the following mosaics were calculated:
Using 0 neighbors. (1 picture).
Using 3 neighbors.(7 pictures).
8
Using 5 neighbors. (11 pictures).
Using all set. (17 pictures).
2.3 Using set mymov2
The mymov2 set of images was capture using a PTZ camera, in particular a
Sony SNC-RZ30N from the Computing Imaging Lab (CIL) at UCF. The images
were taken inside CIL, panning from left to right. Although the original set of
images consist of 19 pictures covering a 180 degrees of view, only a subset of this
images is used in this report, specically images from test6.jpg to test12.jpg
were used, for a total of 7 pictures.
The assumption for this set is that the PTZ camera doesnt move its camera
center. Therefore, if the camera is not moved, its center should remain at the
same place, even if we rotate the camera to do the panning.
test6.jpg test7.jpg test8.jpg test9.jpg
test10.jpg test11.jpg test12.jpg
9
Starting with the center image (test9.jpg) we show the results of the mosaic
with an increasing amount of neighbors around the center:
Using 0 neighbors. (1 picture).
Using 1 neighbors.(3 pictures).
Using 2 neighbors. (5 pictures).
Using 3 neighbors. (7 pictures).
10
2.4 Using set mymov3
The mymov3 set of images was also capture using the PTZ camera from CIL
(a Sony SNC-RZ30N) at UCF. The images were also taken inside CIL, but
this time the panning was diagonal, from lower left to upper right. Although
the original set of images consist of 19 pictures covering 180 degrees of view
horizontally, only a subset of this images is used in this report, specically
images from test6.jpg to test12.jpg were used, for a total of 7 pictures.
This set is intended to test how good is the matching of images when using
a diagonal panning.
test6.jpg test7.jpg test8.jpg test9.jpg
test10.jpg test11.jpg test12.jpg
Starting with the center image (test9.jpg) we show the results of the mosaic
with an increasing amount of neighbors around the center:
Using 0 neighbors. (1 picture).
Using 1 neighbors.(3 pictures).
11
Using 2 neighbors. (5 pictures).
Using 3 neighbors. (7 pictures).
2.5 Using set mymov4
This set of images was captured using a consumer grade point-and-shoot camera
(a Canon PowerShot A490), without the use of a tripod. The pictures were taken
in front of the Millican Hall at UCF, a couple hours before the Spirit Splash of
November 8, 2013. A total of 5 pictures were taken.
This set is intended to test how good the algorithm works when the assump-
tion that the camera center doesnt move is not certain. Since the camera was
held by hand, the human imprecision is almost a guaranteed that the camera
center moves.
IMG 4315.jpg IMG 4316.jpg
12
IMG 4317.jpg (center) IMG 4318.jpg
IMG 4319.jpg
The unied mosaic of all ve images follows:
13