Outline

Introduction
Methodology
Experiments and Results
Conclusion and Future Work

Towards Real-time Terrain Modeling

for Robot Navigation using a
Monocular Camera

Muhammad Tufail
[email protected]
M Eng. in Mechatronics
School of Engineering and Technology

Final Presentation
July 2007

Muhammad Tufail Asian Institute of Technology 1 / 41

 Outline
Introduction
Methodology
Experiments and Results
Conclusion and Future Work

Outline of Topics

1 Introduction
2 Methodology
3 Experiments and Results
4 Conclusion and Future Work

Muhammad Tufail Asian Institute of Technology 2 / 41

 Outline
Introduction Motivation
Methodology Problem Statement
Experiments and Results Objectives
Conclusion and Future Work

Motivation

For vision-based navigation of a mobile robot in an outdoor

environment, terrain modeling is one of the basic capabilities that
the mobile robot must have. Terrial modeling is obtained by 3D
scene reconstruction from depth maps of the environment. In order
to have enough information for robot navigation, dense depth
maps must be constructed from images of the scene. Similar
stereo vision based approach has been used by the Jet Propulsion
Laboratory, NASA in Mars exploratory missions to navigate a
rover, but to decrease cost, we want to do the same for a
monocular camera and sequential pictures.

Muhammad Tufail Asian Institute of Technology 3 / 41

 Outline
Introduction Motivation
Methodology Problem Statement
Experiments and Results Objectives
Conclusion and Future Work

Problem Statement
The need is to develop a wide-baseline monocular stereo vision
technique for landmine detecting mobile robots that combines local
motion estimation and robust image matching in order to build a
3D model of the terrain.

Muhammad Tufail Asian Institute of Technology 4 / 41

 Outline
Introduction Motivation
Methodology Problem Statement
Experiments and Results Objectives
Conclusion and Future Work

Objectives

What was expected from this thesis:

To compute projective geometry between two images
To perform planar rectification of the two images
To calculate a dense disparity map of the scene
To reconstruct the 3D structure of the scene

Muhammad Tufail Asian Institute of Technology 5 / 41

 Outline
Introduction Motivation
Methodology Problem Statement
Experiments and Results Objectives
Conclusion and Future Work

Objectives

What was expected from this thesis:

To compute projective geometry between two images
To perform planar rectification of the two images
To calculate a dense disparity map of the scene
To reconstruct the 3D structure of the scene

Muhammad Tufail Asian Institute of Technology 5 / 41

 Outline
Introduction Motivation
Methodology Problem Statement
Experiments and Results Objectives
Conclusion and Future Work

Objectives

What was expected from this thesis:

To compute projective geometry between two images
To perform planar rectification of the two images
To calculate a dense disparity map of the scene
To reconstruct the 3D structure of the scene

Muhammad Tufail Asian Institute of Technology 5 / 41

 Outline
Introduction Motivation
Methodology Problem Statement
Experiments and Results Objectives
Conclusion and Future Work

Objectives

What was expected from this thesis:

To compute projective geometry between two images
To perform planar rectification of the two images
To calculate a dense disparity map of the scene
To reconstruct the 3D structure of the scene

Muhammad Tufail Asian Institute of Technology 5 / 41

 Outline
Introduction
Methodology System Model
Experiments and Results
Conclusion and Future Work

Figure: System Model

Muhammad Tufail Asian Institute of Technology 6 / 41
 Outline
Introduction
Methodology System Model
Experiments and Results
Conclusion and Future Work

Camera Model

Figure: Camera model. Figure 5.1 from [Hartley and Zisserman (2000)].

Muhammad Tufail Asian Institute of Technology 7 / 41

 Outline
Introduction
Methodology System Model
Experiments and Results
Conclusion and Future Work

Camera Model
K
 
X
  z
 }| {  
x αx 0 x0 1 0 0 0  T 
 y = 0 R t  Y 
αy y0 0 1 0 0   
0T
 
3 1  Z 
1 0 0 1 0 0 0 1
| {z } | {z } 1
x P | {z }
X
(1)

X , Y and Z are three coordinates of a 3-D scene point X

x and y are the coordinates of the 2D image point x
αx and αy are the scaling factors along the horizontal and
vertical directions
(x0 , yo is the principle point
K is called the camera calibration matrix.
Muhammad Tufail Asian Institute of Technology 8 / 41
 Outline
Introduction
Methodology System Model
Experiments and Results
Conclusion and Future Work

Back project the ray1

1
Image courtesy of Svoboda, T. http://cmp.felk.cvut.cz
Muhammad Tufail Asian Institute of Technology 9 / 41
 Outline
Introduction
Methodology System Model
Experiments and Results
Conclusion and Future Work

Project the camera center to the second image2

2
Image courtesy of Svoboda, T. http://cmp.felk.cvut.cz
Muhammad Tufail Asian Institute of Technology 10 / 41
 Outline
Introduction
Methodology System Model
Experiments and Results
Conclusion and Future Work

The corresponding projection must lie on a specific line3

3
Image courtesy of Svoboda, T. http://cmp.felk.cvut.cz
Muhammad Tufail Asian Institute of Technology 11 / 41
 Outline
Introduction
Methodology System Model
Experiments and Results
Conclusion and Future Work

Derivation of Fundamental Matrix[Svoboda (2006)]

We already know: e0 = P0 C
Projection to the camera 2: x0 9 = P0 (λP+ x9 + C)
Line is a cross product of the points lying on it: e0 × x0 9 = l0 9
Putting together: e0 × (P0 λP+ x9 + P0 C) = l0 9
As e0 × P0 C = 0, then: e0 × λP0 P+ x9 = l0 9
Since the point x0 9 must lie on the line l0 9 , therefore l0 T 0
9 x 9 = 0.

Muhammad Tufail Asian Institute of Technology 12 / 41

 Outline
Introduction
Methodology System Model
Experiments and Results
Conclusion and Future Work

Derivation of Fundamental Matrix

 
0 −e3 e2
Let [e]x =  e3 0 −e1  The cross project is then written
−e2 e1 0
as a matrix multiplication l0 T 0 0 +
9 = ([e ]x λP P )x9

Inserting into l0 T 0
9 x 9 = 0 gives us:

0 0 + T 0
xT
9 ([e ]x λP P ) x 9 = 0 (2)
| {z }
F

Muhammad Tufail Asian Institute of Technology 13 / 41

 Outline
Introduction
Methodology System Model
Experiments and Results
Conclusion and Future Work

Epipolar geometry4

x0 i Fxi = 0 holds for any corresponding pair xi ←→ x0 i

All epipolar lines intersect in epipoles.
4
Image courtesy of Svoboda, T. http://cmp.felk.cvut.cz
Muhammad Tufail Asian Institute of Technology 14 / 41
 Outline
Introduction
Methodology System Model
Experiments and Results
Conclusion and Future Work

Estimation of fundamental matrix

For points matching xi ←→ x0 i

x0 i Fxi = 0 (3)

Each matching pair gives one linear equation

xx 0 y x 0 x 0 xy 0 y y 0 y 0 x y 1 f = 0
 
(4)

From n point matches, we can obtain a set of linear equation of

the form
Af = 0 (5)
In Equation 5, f is a nine-vector containing the entries of the
matrix F, and A is the equation matrix of size n × n. The system of
equation can be solved by singular value decomposition (SVD).
Muhammad Tufail Asian Institute of Technology 15 / 41
 Outline
Introduction
Methodology System Model
Experiments and Results
Conclusion and Future Work

Rectification

Figure: Image rectification. After rectification epipoles are at infinity and

epipolar lines become parallel. (Figure 8.4 from
[Hartley and Zisserman (2000)])

Muhammad Tufail Asian Institute of Technology 16 / 41

 Outline
Introduction
Methodology System Model
Experiments and Results
Conclusion and Future Work

Rectification

From estimated fundamental matrix F and epipoles e, e0 , select a

projective transformation H0 that maps the epipole e0 to the point
at infinity, (1, 0, 0)T . This transformation H0 , also called a
homography, is given by:

H0 = GRT (6)

Muhammad Tufail Asian Institute of Technology 17 / 41

 Outline
Introduction
Methodology System Model
Experiments and Results
Conclusion and Future Work

Homography for right image

 
1 0 −x0
T =  0 1 −y0  translates the origin to the center of the
0 0 1
image
 
cos θ − sin θ 0
R =  sin θ cos θ 0  is the rotation matrix where
0 0 1
θ = arctan(ey0 /ex0 ). R will rotate the epipole to a point
[f , 0, 1]t on the x-axis.
 
1 0 0
G =  0 1 0  where f = x-component of RTe0 .
− f1 0 1

Muhammad Tufail Asian Institute of Technology 18 / 41

 Outline
Introduction
Methodology System Model
Experiments and Results
Conclusion and Future Work

Homography for left image

The corresponding projective transformation for the left image H is

found through least-sqaures estimation. This means finding HA
minimizing the cost function
X  2
d HA x̂i , x̂0i
i

where F = [e0 ]x M, H0 = H0 M, x̂i = H0 xi , x̂0i = H0 x0i .

Muhammad Tufail Asian Institute of Technology 19 / 41

 Outline
Introduction
Methodology System Model
Experiments and Results
Conclusion and Future Work

Homography for left image

Equivalently we can minimize the function

P  2
f (a, b, c) = i axi0 + ayi0 + c − xˆi0 . The resulting HA will be of
the form  
a b c
HA =  0 1 0 
0 0 1
And finally
H = HA H0 (7)

Muhammad Tufail Asian Institute of Technology 20 / 41

 Outline
Introduction
Methodology System Model
Experiments and Results
Conclusion and Future Work

Dense correspondence

Figure: Disparity: d = x1 − x2 (the difference in retinal position between

the corresponding points in the two images.)

Muhammad Tufail Asian Institute of Technology 21 / 41

 Outline
Introduction
Methodology System Model
Experiments and Results
Conclusion and Future Work

Pseudocode for computing disparity map. (Pseudocode

from Forstmann et al. 2004)

Muhammad Tufail Asian Institute of Technology 22 / 41

 Outline
Introduction
Methodology System Model
Experiments and Results
Conclusion and Future Work

3D reconstruction - Linear method

[Hartley and Zisserman (2000)]
A scene point X can be reconstructed from it’s projections
xi ←→ x0 i and camera matrices P , P0 .
As x × (PX) = 0 and x0 × (P0 X) = 0, we can formulate a linear
equation for point X in the world coordinate frame as

xp3T − p1T
 
 y p3T − p2T 
AX =  x 0 p0 3T − p0 1T  = 0
 (8)
0 0
y 0 p 3T − p 2T

where piT are the rows of P. Then the scene point X can be
obtained by least squares solution of the homogenous linear system
using the SVD.
Muhammad Tufail Asian Institute of Technology 23 / 41
 Outline
Introduction Tools
Methodology Available data
Experiments and Results Results
Conclusion and Future Work

Tools

Structure and Motion Toolkit in Matlab[Torr (2002)]

Intel’s Open Source Computer Vision Library (OpenCV) 5

IrfanView: Image viewing software 6

5
http://www.intel.com/technology/computing/opencv/
6
http://www.irfanview.com/
Muhammad Tufail Asian Institute of Technology 24 / 41
 Outline
Introduction Tools
Methodology Available data
Experiments and Results Results
Conclusion and Future Work

Tools

Structure and Motion Toolkit in Matlab[Torr (2002)]

Intel’s Open Source Computer Vision Library (OpenCV) 5

IrfanView: Image viewing software 6

5
http://www.intel.com/technology/computing/opencv/
6
http://www.irfanview.com/
Muhammad Tufail Asian Institute of Technology 24 / 41
 Outline
Introduction Tools
Methodology Available data
Experiments and Results Results
Conclusion and Future Work

Tools

Structure and Motion Toolkit in Matlab[Torr (2002)]

Intel’s Open Source Computer Vision Library (OpenCV) 5

IrfanView: Image viewing software 6

5
http://www.intel.com/technology/computing/opencv/
6
http://www.irfanview.com/
Muhammad Tufail Asian Institute of Technology 24 / 41
 Outline
Introduction Tools
Methodology Available data
Experiments and Results Results
Conclusion and Future Work

Availabe Data
Camera Calibration
A camera calibration data is assumped to be provided by
[Nakarmi (2007)].

The camera parameters are:

Table: Camera parameters.

Num. Parameter Value

1 Horizontal field of view 70 deg
2 Focal length 475 pixels
3 View window size 640x480 pixels

Muhammad Tufail Asian Institute of Technology 25 / 41

 Outline
Introduction Tools
Methodology Available data
Experiments and Results Results
Conclusion and Future Work

Input images

(a) (b)

Figure: “Head and Lamp” image pair. (a) Left image. (b) Right image.

Muhammad Tufail Asian Institute of Technology 26 / 41

 Outline
Introduction Tools
Methodology Available data
Experiments and Results Results
Conclusion and Future Work

Input images

(a) (b)

Figure: “Bread and Oranges” image pair. (a) Left image. (b) Right
image.

Muhammad Tufail Asian Institute of Technology 27 / 41

 Outline
Introduction Tools
Methodology Available data
Experiments and Results Results
Conclusion and Future Work

Feature extraction and matching

Figure: Image corners. Crosses show the Harris corners.

Muhammad Tufail Asian Institute of Technology 28 / 41

 Outline
Introduction Tools
Methodology Available data
Experiments and Results Results
Conclusion and Future Work

Feature extraction and matching

Figure: Feature matching. Corners are matched by the correlation

matching technique.

Muhammad Tufail Asian Institute of Technology 29 / 41

 Outline
Introduction Tools
Methodology Available data
Experiments and Results Results
Conclusion and Future Work

Fundamental matrix and epipolar geometry

The calculated fundamental matrix is:

 
−0.206307 −12.407629 −6.458630
F = 13.766137
 −0.986742 −43.087103
7.831750 44.049470 1.000000
 
e = 3.088941 −0.571898 1.000000

e0 = 3.591156 −0.515095 1.000000

 

Muhammad Tufail Asian Institute of Technology 30 / 41

 Outline
Introduction Tools
Methodology Available data
Experiments and Results Results
Conclusion and Future Work

Rectification

Figure: Original left and right images before rectification.

Muhammad Tufail Asian Institute of Technology 31 / 41

 Outline
Introduction Tools
Methodology Available data
Experiments and Results Results
Conclusion and Future Work

Rectification

Figure: Rectified left and right images. Corresponding points lie on the
same horizontal line.

Muhammad Tufail Asian Institute of Technology 32 / 41

 Outline
Introduction Tools
Methodology Available data
Experiments and Results Results
Conclusion and Future Work

Dense correspondence

(a) (b)

Figure: Disparity map for “Head and Lamp” images. (a) Input left
image. (b) Estimated disparity map for “Head and Lamp” image pair.

Muhammad Tufail Asian Institute of Technology 33 / 41

 Outline
Introduction Tools
Methodology Available data
Experiments and Results Results
Conclusion and Future Work

Dense correspondence

(a) (b)

Figure: Disparity map for “Head and Lamp” images. (a) Estimated
disparity map. (b) Ground truth disparity map.

Muhammad Tufail Asian Institute of Technology 34 / 41

 Outline
Introduction Tools
Methodology Available data
Experiments and Results Results
Conclusion and Future Work

Dense correspondence

(a) (b)

Figure: Estimated disparity map for “Bread and Oranges” images. (a)
Input left image. (b) Estimated disparity map.

Muhammad Tufail Asian Institute of Technology 35 / 41

 Outline
Introduction Tools
Methodology Available data
Experiments and Results Results
Conclusion and Future Work

3D reconstruction

Figure: 3D reconstructed “Head and Lamp” image (View 1).

Muhammad Tufail Asian Institute of Technology 36 / 41

 Outline
Introduction Tools
Methodology Available data
Experiments and Results Results
Conclusion and Future Work

3D reconstruction

Figure: 3D reconstructed “Head and Lamp” image (View 2).

Muhammad Tufail Asian Institute of Technology 37 / 41

 Outline
Introduction Tools
Methodology Available data
Experiments and Results Results
Conclusion and Future Work

3D reconstruction

Figure: 3D reconstructed “Bread and Oranges” image.

Muhammad Tufail Asian Institute of Technology 38 / 41
 Outline
Introduction
Methodology
Experiments and Results
Conclusion and Future Work

Conclusion and Future Work

Following are some recommendations for futue work:
Instead of Harris corner detector, a more robust feature
detector such as SIFT can be used.
For rectification (Mellon and Whelan, 2005) presents very
good improvement on (Hartley, 1999) which is implemented in
this thesis. The improvement is in the way robust matching
homographies are computed which perfectly aligns the
epipolar lines in the two images.
The Birchfield algorithm implemented in OpenCV to compute
disparity map is open source and therefore can be modified to
bring improvements in it. Instead of matching individaul pixels
with each other, a correlation matching technique can be used
so that errors in image rectification up to one or two pixels
can be accounted for.
Muhammad Tufail Asian Institute of Technology 39 / 41
 Outline
Introduction
Methodology
Experiments and Results
Conclusion and Future Work

End

The presentation ends here...

Thank you!

Muhammad Tufail Asian Institute of Technology 40 / 41

 Outline
Introduction
Methodology
Experiments and Results
Conclusion and Future Work

References

Birchfield, S., and Tomasi, C. (1998). Depth discontinuities by

pixel-to-pixel stereo. International Conference on Computer
Vision. pp. 1073-1080. India.
Cox, I. J., Hingorani, S. L., Rao, S. B., Maggs, M. B. (1996).
A maximum likelihood stereo algorithm. Computer Vision and
Image Understanding, 63(3), pp. 542-567.
Forstmann, S., Kanou, Y., Ohya, J., Thuering, S., and
Schmitt, A. (2004). Real-time stereo by using dynamic
programming. In Proceedings of the IEEE Computer Society
Conference on Computer Vision and Pattern Recognition
Workshops (CVPRW’04), pp. 1063-69.

Muhammad Tufail Asian Institute of Technology 41 / 41

 Outline
Introduction
Methodology
Experiments and Results
Conclusion and Future Work

References

Harris, C. J., & Stephens, M. (1988). A combined corner and

edge detector. Proceedings of 4th Alvey Vision Conference, pp.
147151, Manchester.
Hartley, R. I. (1999). Theory and Practice of Projective
Rectification. International Journal of Computer Vision, 35(2),
pp. 1-16.
Hartley, R. & Zisserman, A. (2000). Multiple View Geometry
in Computer Vision. Cambridge University Press.

Muhammad Tufail Asian Institute of Technology 41 / 41

 Outline
Introduction
Methodology
Experiments and Results
Conclusion and Future Work

References

Mellon, J. & Whelan, P. (2005). Projective rectification from

the fundamental matrix. Image and Vision Computing. 23(7).
Montemerlo, M., Thrun, S., Koller, D., Wegbreit, B. (2002).
FastSLAM: A factored solution to the simultaneous
localization and mapping problem. American Association for
Artificial Intelligence.
Najjaran, H. & Goldberg, A. A. (2005). Landmine detection
using an autonomous terrain scanning robot. Industrial Robot
Journal, 32 (3), pp. 240-247.

Muhammad Tufail Asian Institute of Technology 41 / 41

 Outline
Introduction
Methodology
Experiments and Results
Conclusion and Future Work

References

Nakarmi, A. Dev. (2007). Outdoor mobile robot localization

and environment mapping using a single camera. (Master
thesis, Asian Institute of Technology, 2007). Bangkok: Asian
Institute of Technology.
Svoboda, T. (2006). Two-view geometry. Center for Machine
Vision, Czech Technical University in Prague. Available online:
http://www.cmp.felk.cvut.cz
Torr, H. S. Philip. (2002). A structure and motion toolkit in
MATLAB TM . Department of Computing, Oxford Brookes
University UK.

Muhammad Tufail Asian Institute of Technology 41 / 41