Photogrammetry & Robotics Lab Motivation

Direct Solutions for Computing

Fundamental and Essential
Matrix

Cyrill Stachniss

The slides have been created by Cyrill Stachniss.

1 Image courtesy: Collins 2

Topics of Today
Compute the
§ Fundamental matrix
given corresponding points Computing the
§ Essential matrix Fundamental Matrix
given corresponding points Given Corresponding Points
§ Rotation matrix and basis
given an essential matrix

3 4
Fundamental Matrix Fundamental Matrix
§ The fundamental matrix is The fundamental matrix can directly
be computed if we know the
§ calibration matrices
§ viewing direction of the cameras
§ It encodes the relative orientation
§ baseline
for two uncalibrated cameras
§ or the projection matrices
§ Coplanarity constraint through

How to compute F given ONLY

corresponding points in images?
5 6

Problem Formulation Fundamental Matrix

From Corresponding Points
§ Given: N corresponding points
§ For each point, we have the
coplanarity constraint
§ Wanted: fundamental matrix

7 8
Fundamental Matrix What is the Issue here?
From Corresponding Points
§ In standard least squares problems,
§ For each point, we have the we have a vector of unknowns
coplanarity constraint § Here, the matrix elements of F are
the unknowns

§ or
Question:
How to turn the unknown matrix
elements into a vector of unknowns?

unknowns! 9 10

Linear Dependency Linear Dependency

§ Linear function in the unknowns § Linear function in the unknowns

11 12
Linear Dependency Linear Dependency
§ Linear function in the unknowns § Linear function in the unknowns

13 14

Linear Dependency Using the Kronecker Product

§ Linear function in the unknowns § Linear function in the unknowns

(it holds in general: )

15 16
Linear System From All Points Solving the Linear System
§ We directly obtain a linear system if § Singular value decomposition solves
we consider all N points

§ and thus provides a solution for

§ SVD: f is the right-singular vector

corresponding to a singular value of A
that is zero
17 18

How Many Points Are Needed? How Many Points Are Needed?
§ The vector has 9 dimensions § The vector has 9 dimensions

§ Matrix has at most rank 8

§ We need 8 corresponding points

19 20
More Than 8 Points… Singular Vector
§ In reality: noisy measurements § Use the singular vector of A that
§ With more than 8 points, the matrix A corresponds to the smallest singular
will become regular (but should not!) value is the solution
§ Use the singular vector of A that
corresponds to the smallest singular
value is the solution
=

21 22

8-Point Algorithm 1st Try 8-Point Algorithm 1st Try

singular vector of the

smallest singular value
Not necessarily a matrix of rank 2
23 (but F should have: rank(F)=2) 24
Enforcing Rank 2 Enforcing Rank 2
§ We want to enforce a matrix with § We want to enforce a matrix with

§ should approximate our computed § should approximate our computed

matrix as close a possible matrix as close a possible
§ Use a second SVD (this time of F)
What to do?

with
and

25 26

8-Point Algorithm 8-Point Algorithm

27 28
Well-Conditioned Problem Conditioning/Normalization
to Obtain a Well-Conditioned
§ Example image 12MPixel camera
Problem
4000
§ Normalization of the point coordinates
substantially improves the stability
3000

§ Transform the points so that the

§ Ill-conditioned, numerically instable center of mass of all points is at (0,0)
§ Scale the image so that the x and y
new coordinated are within [-1,1]
coordinate
system

29 30

Conditioning/Normalization Singularity – Points on a Plane

§ Define so that coordinates § If all corresponding points lie on a
are zero-centered and in [-1,1] plane, then
§ Determine fundamental matrix from § Numerically instable if points are close
the transformed coordinates to a plane

§ Obtain essential matrix through

Images from the “Fundamental Matrix Song” Video by D. Wedge

31 32
Singularity – No Translation Summary so far
§ The projection centers of both § Estimating the fundamental matrix
cameras are identical: from N pairs of corresponding points
§ This happens if the translation of the § Direct solution of N>7 points based on
camera is zero between both images solving a homogenous linear system
(“8-Point Algorithm”)
scene

cam1
cam2
33 34

Reminder: Essential Matrix

§ Essential matrix = “fundamental
matrix for calibrated cameras”
Let’s Do the Same for the
Essential Matrix
§ Often parameterized through
(general parameterization of dependent images)

§ Coplanarity constraint for calibrated

cameras
35 36
Essential Matrix Essential Matrix
from 8+ Corresponding Points from 8+ Corresponding Points
§ For each point, we have the § For each point, we have the
coplanarity constraint coplanarity constraint

§ Note: Same equation as for the § or

fundamental matrix but for the
points in the camera c.s.
Remember:
37 38

As for the Fundamental Matrix… Constraints

§ For the fundamental matrix, we
enforced the rank(F) constraint

§ For the essential matrix, both non-

zero singular values are identical
build matrix A

solve Ae=0

build matrix E homogenous

Which constraints to consider? 39 More details: Förstner, Skript Photogrammetrie II, Sect 1.2.3 40
8-Point Algorithm for the Conditioning/Normalization
Essential Matrix to Obtain a Well-Conditioned
Problem (As Done Before)
§ As for the 8-Point algorithm for the
fundamental matrix, normalization of
build matrix A
the point coordinates is essential
solve Ae=0 § Transform the points so that the
build matrix Ea
center of mass of all points is at (0,0)
§ Scale the image so that the x and y
compute SVD of Ea
coordinated are within [-1,1]
build matrix E from Ea
by imposing constraints
41 42

Conditioning/Normalization Properties of the Essential Mat.

§ Define so that coordinates § Homogenous
are zero-centered and in [-1,1] § Singular: (determinant is zero)
§ Determine essential matrix from the § Two identical non-zero singular values
transformed coordinates

§ As a result of the skew-sym. matrix:

§ Obtain essential matrix through

43 44
 5-Point Algorithm
§ Proposed by Nistér in 2003/2004
§ Standard solution today to obtaining
the direct solution
§ Solving a polynomial of degree 10
5-Point Algorithm § 10 possible solutions
§ Often used together RANSAC
§ RANSAC proposes correspondences
§ Evaluate all 5-point solutions based on
the other corresponding points

45 46

5-Point Algorithm
§ More details in the script by Förstner
“Photogrammetrie II”, Ch 1.2
§ Stewenius, Engels, Nistér: “Recent Computing the
Developments on Direct Relative Orientation Parameters
Orientation”, ISPRS 2006
Given the Essential Matrix
§ Li and Hartley: “Five-Point Motion
Estimation Made Easy”

47 48
Compute Basis and Rotation The Solution We Want…
Given E
§ In short:

Question: Is there a unique solution?

49 50

Multiple Solutions from Math… Multiple Solutions from Math…

We only know b up to a scalar. We only know b up to a scalar.

So we can multiply it by -1… So we can multiply it by -1…

51 52
Multiple Solutions from Math… Multiple Solutions from Math…

We can also rotate the Or do both…

(second) camera by PI

53 54

Four Possible Solutions from One Solution from Physics…

Math…

The only physically plausible

solution that that the point are
in front of both cameras.
55 56
 Solution by Hartley & Zisserman
§ We know that

Algebraic Solution rotation

matrices
for Obtaining the Basis and Rotation
Matrix Given the Essential Matrix

57 58

Solution by Hartley & Zisserman Solution by Hartley & Zisserman

§ We know that

rotation
matrices

§ Define
skew-sym. mat rotation mat

§ So that
59 60
Solution by Hartley & Zisserman Solution by Hartley & Zisserman

61 62

Four Possibilities to Define Z, W Yields Four Solutions

2 solutions for SB 2 solutions for R

4 solutions

63 64
Solution by Hartley & Zisserman Summary (1)
§ Compute the SVD of E: § Algorithms to compute the relative
§ Normalize by orientation from image data
§ Compute the four solutions § Allow us to estimate the camera
motion (except of the scale)
§ Direct solutions
§ Test for which solutions all points are § F from N>7 points (“8-Point Algorithm”)
in front of both cameras § E from N>7 points (“8-Point Algorithm”)
§ Return the physically plausible § E from N=5 points (“Nister’s 5-Point
configuration Algorithm”)

65 66

Summary (2) Literature

§ Direct solutions § Förstner, Wrobel: Photogrammetric
§ Extracting from Computer Vision, Ch. 12.3.1-12.3.3
§ Not statistically optimal § Hartley: In Defence of the 8-point
§ Often used in combination with Algorithm
RANSAC for identifying in/outliers § Stewenius, Engels, Nistér: Recent
§ Direct solutions & RANSAC serves as Developments on Direct Relative
Orientation, ISPRS 2006
initial guess for iterative solutions
§ Subsequent refinement using least
squares only based on inlier points
67 68
Slide Information
§ The slides have been created by Cyrill Stachniss as part of the
photogrammetry and robotics courses.
§ I tried to acknowledge all people from whom I used images or
videos. In case I made a mistake or missed someone, please
let me know.
§ The photogrammetry material heavily relies on the very well
written lecture notes by Wolfgang Förstner and the
Photogrammetric Computer Vision book by Förstner & Wrobel.
§ Parts of the robotics material stems from the great
Probabilistic Robotics book by Thrun, Burgard and Fox.
§ If you are a university lecturer, feel free to use the course
material. If you adapt the course material, please make sure
that you keep the acknowledgements to others and please
acknowledge me as well. To satisfy my own curiosity, please
send me email notice if you use my slides.

Cyrill Stachniss, [email protected], 2014 69