STEREOPSIS 1
Camera geometry and 3D-reconstruction
Norbert Krger and Dirk Kraft based on slides by Nicolas Pugeault
Outline
A. One camera
Lm
M1
perspective considerations projective geometry camera model
C
M'' M'
m
B. Two cameras
extrinsic parameters
M m1 m2
calibration epipolar geometry reconstruction
C1
C2
A. One camera
Lm
M1
M'' M'
m
Perspective
visual sensors form images by concentrating light on a sensitive surface using lenses Images are a projective mapping of the 3D space understanding this mapping is essential for inverting it
pinhole camera model projective geometry calibration
Perspective (cont'd)
Intuitively, we have a relation between apparent size and depth of the object
far objects appear small nearby objects appear big
Thales theorem yields:
f x= X d
The Projective plane
Point is designated by the three coordinates:
vectors are points at infinity
u m@ v A w
w=0
point at infinity
m1 m2
l1
line at infinity
l2
parallel lines meet at points at infinity
The Projective plane (cont'd)
There is a point-line duality Lines are represented by:
ax + by + cw = l m = 0
x y a +b +c =0 w w
Such that the following relation holds 8m 2 l
a l @ b A c
m
l3
l1 l2
m1 m2 m3
The Projective space
3D points are represented 0 1 by
Vectors are points at infinity They lie on the plane at infinity
plane at infinity
x B y C C M B @ z A w
M1
M L
The Projective space: 3D-planes
Planes in projective space are dual to points They are described by
h O
They satisfy the relation
nx B ny C C B @ nz A h
n is the normal vector h the distance to the origin O
M = 0; 8M 2
The Projective space: Pluecker lines
Pluecker coordinates represent 3D lines using
(normalized such that kk = 1) line moment , defined for all point m 2 L
line orientation vector
= m
L O m
L (; )
Conversion to and from homogeneous coordinates
we can easily convert from and to Homogeneous and Euclidian coordinates:
x x B B y C C homogeneous B @ y A B @ @ z A z 1 m m M
to
Euclidian
0 1 x x C 1 @ y C y A z A w z w M
from
Euclidian
Projective geometry
points at infinity (equivalent to Euclidian vectors): e.g. parallel lines meet at infinity allows for a linear representation of translations:
t = (tx ; ty ; tz )>
0 1 0 e.g., rotation 0 1 0 0 0 0 1 0
translation
x + tx 1 B y + ty C B 0 B C=B @ z + tz A @ 0 1 0
tx x ty C B y C CB C tz A @ z A 1 1
1 0
Projection matrix
Using homogeneous coordinates, the pinhole model can then be formalized as: where
m= AM
f A=@ 0 0
0 f 0
0 0 1
0 0 A 0
Projection matrix (cont'd)
We want to include the imaging parameters:
s is the skew (generally null) pixel arrangement u and v are the pixel resolution in u and v c = (u0 ; v0 )> is the projection of the optical center.
m= K AM 0 1 u s u 0 K = @ 0 v v0 A 0 0 1
Calibration
The result is the following matrix called:
The parameters (f; u ; v ; s; u0 ; v0 ) are called Intrinsic parameters
f u 0 KA=@ 0
f s f v 0
u0 v0 1
0 0 A 0
Calibration, is the process to estimate the following parameters:
s f u ; f v ; ; u0 ; v0 u
Distortion
Real lenses are only an approximation of a pinhole model => distortion e.g. radial distortion with wide-angle (small focal length) lenses. Lead to non-linear projection model
=> image undistortion
Calibration (2)
Tools exist for performing calibration
openCV
http://www.intel.com/technology/computing/opencv/
matlab calibration toolbox
http://www.vision.caltech.edu/bouguetj/calib_doc/
Based on regular grid patterns of known sizes, with manual or automatic corner extraction provides an estimation of all parameters + distortion provides tool for image rectification
Calibration(3)
B. Two cameras
M m1 m2
C1
C2
3D Pose
We need to model the relative position of the two cameras Combination of
translation between the two optical centers rotation between optical axes
m1
C1
m2
C2
This is a rigid transformation in 3D space (=> 6 DoF)
Extrinsic parameters
General projection matrix formulation:
m = (K A H) M
Where the Extrinsic parameters are
H=
R 013
t 1
R is the rotation that brings the optical axis to the z -axis t is the translation vector that brings C onto the origin
O = (0; 0; 0; 1)>
t = R C
Stereo Calibration
http://www.vision.caltech.edu/bouguetj/calib_doc/htmls/example5.html
Stereo calibration (cont'd)
Practical
Objectives:
Calibrate a stereo pair of cameras
compute the projection matrices P1 and P2 compute the fundamental matrix F12 display both left and right images when a point is clicked in the left image, displays the corresponding epipolar line in the right image reconstruct the corresponding 3D point when a stereo pair of points is selected.
Make a program that:
Outline
A. One camera
Lm
M1
perspective considerations projective geometry camera model
C m
M'' M'
B. Two cameras
extrinsic parameters
calibration epipolar geometry reconstruction
C1
M m1 m2
C2
Plucker line - prereq.
Classical line representation Point on line closest to origin:
L O
~ c ~ a
.
~ b
Every point on line can be decomposed: Cross product properties:
To Plucker lines and back (1)
Plucker line: To Plucker line:
To Plucker lines and back (2)
Back:
Epipolar geometry
Plausible 3D origins for an image point m form a halfline in space. This line passes through the optical center of the camera.
Lm
M'' M'
M1
Epipolar Geometry (cont'd)
If we consider the projection matrix
Here italic denotes to projective quantities, e.g., P and bold denotes Euclidean quantities, e.g., P P is a 3x3 matrix. p is a dimension 3 column vector.
Epipolar Geometry (cont'd)
Note that p contains the projective coordinates of the projection on the image plane of the world coordinate system's origin.
If there is not translation then we also have and
Epipolar Geometry (cont'd)
Then p is the projection of the optical center on the image plane
C = R 1 t = P 1 p
and P 1 gives the back-projection vector of a given image point
M1 =
P 1 m 0
Note that m denotes the projective coordinates of a point in the image.
Back-Projected ray
From this we can compute the coordinates of a line in space where lie all plausible 3D origins of the image point
Lm =
1 kM
1k
(M
1; C
1)
Effectively we can even say that the plausible origins lie on the half line (or ray) starting at the camera's optical center C This limits the possible locations of the correspondence in the second image to a line, a half line, or even a segment depending on the set-up.
Epipolar line
Xl X l
Lxl Il
Ir xl
xr
x r
Cl
er
Cr
Epipolar line (cont'd)
The plausible correspondences on the right image form a 2Dline that is the re-projection of the optical ray on the image It can be a line, a half-line or even a segment, depending if the endpoints lie at infinity or not. One endpoint is fixed and is e = P C 2 2 1 called the Epipole Note that if the optical center lie behind the camera, the epipole is not a boundary The second is the reprojection at infinity of the point considered:
e2
C2 e1
m2 m1
m2;1 = P2 M1;1
C1
Epipolar line (cont'd)
Then the line between those two points, where lie all plausible correspondences, is called the Epipolar line
(a)
(b)
l = e2 m2;1
(c)
(d)
e2
e2
e2
e2
C1 C1 C2 C1 C2 C2
C2 C1
Example (left image)
Example (right image)
Example (stereo)
C1
LEFT STEREO
C2
RIGHT
Fundamental matrix (1/2)
The fundamental matrix is defined as:
> 2
F12 m
=0
for m 1 and m 2 two corresponding points, respectively in the left and right image.
It can be computed from the two projection matrixes:
+ F12 = [e2 ] P2 P1
0 [v] = @ v3 v2
v3 0 v1
v2 v1 A 0
Fundamental matrix (1/2)
Where P + is the pseudo-inverse of P such that:
In this case P + can be decomposed into:
P P+ P = P
P + = H 1 A+ K 1
The fundamental matrix can be used to compute the epipolar line:
l = F12 m
Computation of the Fundamental matrix
1) find pairs of corresponding points in the two images (m1 ; m2 ) 2) Effectively 7 DOF, so at least 7 points are required.
3) Select a first set and generate an hypothesis
the distance from the correspondence to the epipolar line gives one constraint m> F12 m1 = 0 2 Solve a system of linear equations (RANSAC will be discussed in the Motion Estimation lecture)
4) outliers => use RANSAC
5) refine using non-linear optimization
Hartley and Zisserman, ch.11.6
Feature correspondences
left
right
correspondences
Computed Epipolar lines
left
right
Stereopsis: epipolar constraint
All potential correspondences of a primitive lie on its epipolar line:
left camera
right camera
Point reconstruction
Two corresponding stereo points back-project two optical rays in space. Theoretically, M lie at the intersection between the two lines. Practically, the two lines are unlikely to intersect. Often the closest point to both lines is chosen.
a)
m1
m2
C1
C2
b)
L2 M1
L1 M2
1 (2 2 ) (1 2 ) 1 (2 1 ) = 1 + 1 1 2 jj1 2 jj
Point reconstruction (cont'd)
Computing the point on L1 that is closest to L2
a)
m1
m2
= arg min d(x; L2 )
x2L1
C1
L1 = (1 ; 1 ) L2 = (2 ; 2 ) k1 k = k2 k = 1
C2
b)
L2 M1
L1 M2
1 (2 2 ) (1 2 ) 1 (2 1 ) = 1 + 1 1 2 jj1 2 jj
Practical
Objectives:
Calibrate a stereo pair of cameras
compute the projection matrices P1 and P2 compute the fundamental matrix F12 display both left and right images when a point is clicked in the left image, displays the corresponding epipolar line in the right image reconstruct the corresponding 3D point when a stereo pair of points is selected.
Make a program that:
Linear alternative
We can alternatively formulate the reconstruction as a linear optimization problem: Al bl M = Ar br Namely if we rewrite the projection matrices as: P = @
0 Q> 1 Q> 2 Q> 3
Then we can write the projection as the system of linear equations: 0 1 0 1 > Q1 q14 u M @ Q> q24 A = @ v A 2 1 > 1 Q3 q34
q14 q24 A q34
Linear formulation
Which translates into the system of equations 8 > < Q1 M + q14 = u Q> M + q24 = v : 2 Q> M + q34 = 3 By replacing we have:
Q> M uQ> M + q14 uq34 1 3 Q> M vQ> M + q24 vq34 2 3 > > Q1 uQ3 And when reordering: M = > > Q2 vQ3
= =
0 0
Ai
uq34 q14 vq34 q24
bi
Solving the system
We concatenate the equations provided by the corresponding points in both images into one system:
A M = b with
8 > > A < > > b :
= =
Al Ar bl br
The solution to this system is given by:
M = (A> A)1 A> b
(Linear least square)
Line reconstruction
The plausible origins of 2D lines form a plane in space The intersection of the two planes generated from the line in both images give a reconstructed 3D-line This process is ill-defined if the lines are parallel to the epipolar line -> then the two lines back-project the same plane in 3D !
L = (1 2 ; h2 1 h1 2 )
Image rectification
Make epipolar lines horizontal Note: there is still an horizontal shift of the images !
Points infinitely far do not have zero disparity
e2
rectification
e2
C1
C2
C1
C2
Image rectification example
before
after
left
right
