• Stereo and lidar can fall victim to reflections?

• Yes, there’s no easy way around that

• https://youtu.be/pBzU8TD1iks
Last lecture: World vs Camera coordinates
Today’s Outline
• Epipolar Geometry
– Finding epipolar relationship between two images
– Using epipolar geometry to rule out outliers
– Finding dense correspondence along epipolar lines
Where do we need to search?
 Epipolar Geometry and
Stereo Vision
Chapter 11.3 in Szeliski

Many slides adapted from Derek Hoiem, Lana Lazebnik, Silvio Saverese, Steve Seitz, many figures from Hartley & Zisserman
• Epipolar geometry
– Relates cameras from two positions
Depth from Stereo
• Goal: recover depth by finding image coordinate x’ that corresponds to
x

X
X

x z
x x’

f f
x'
C Baseline C’
B
Depth from Stereo
• Goal: recover depth by finding image coordinate x’ that
corresponds to x
• Sub-Problems
1. Calibration: How do we recover the relation of the cameras (if
not already known)?
2. Correspondence: How do we search for the matching point x’?

x'
Correspondence Problem

x ?

• We have two images taken from cameras with different

intrinsic and extrinsic parameters

• How do we match a point in the first image to a point in the

second? How can we constrain our search?
Where do we need to search?
Key idea: Epipolar constraint
Key idea: Epipolar constraint
X

x x’

x’

x’

Potential matches for x have to lie on the corresponding line l’.

Potential matches for x’ have to lie on the corresponding line l.

Wouldn’t it be nice to know where
matches can live? To constrain our 2d
search to 1d.
VLFeat’s 800 most confident matches
among 10,000+ local features.
Epipolar geometry: notation
X

x x’

• Baseline – line connecting the two camera centers

• Epipoles
= intersections of baseline with image planes
= projections of the other camera center
• Epipolar Plane – plane containing baseline (1D family)
Epipolar geometry: notation
X

x x’

• Baseline – line connecting the two camera centers

• Epipoles
= intersections of baseline with image planes
= projections of the other camera center
• Epipolar Plane – plane containing baseline (1D family)
• Epipolar Lines - intersections of epipolar plane with image
planes (always come in corresponding pairs)
Example: Converging cameras
Example: Motion or displacement parallel
to image plane
Example: Forward motion
What would the epipolar lines look like if the camera moves
directly forward?

a b c d
Example: Forward motion

e’

Epipole has same coordinates in both

images.
Points move along lines radiating from e:
“Focus of expansion”
Epipolar constraint: Calibrated case
X

x x’

Given the intrinsic parameters of the cameras:

1. Convert to normalized coordinates by pre-multiplying all points with the
inverse of the calibration matrix; set first camera’s coordinate system to
world coordinates

xˆ  K 1 x  X xˆ   K 1 x   X 
3D scene point
Homogeneous 2d point
(3D ray towards X) 2D pixel coordinate 3D scene point in 2nd
(homogeneous) camera’s 3D coordinates
Epipolar constraint: Calibrated case
X

x x’

Given the intrinsic parameters of the cameras:

1. Convert to normalized coordinates by pre-multiplying all points with the
inverse of the calibration matrix; set first camera’s coordinate system to
world coordinates
2. Define some R and t that relate X to X’ as below
for some scale factor
ˆx  K 1 x  X xˆ   K 1 x   X 
xˆ  Rxˆ   t
Epipolar constraint: Calibrated case
X

x x’
𝑥ො
𝑥ො ′

xˆ  K 1 x  X xˆ   K 1 x   X 

xˆ  Rxˆ   t xˆ  [t  ( Rxˆ )]  0

ො 𝑅 𝑥ො ′ , and 𝑡 are co-planar)
(because 𝑥,
Essential matrix
X

x x’

xˆ  [t  ( R xˆ )]  0 xˆ T E xˆ   0 with E  t  R

Essential Matrix
(Longuet-Higgins, 1981)
Properties of the Essential matrix
X

x x’

xˆ  [t ( R xˆ )]  0 xˆ T E xˆ   0 with E  t  R
Drop ^ below to simplify notation

• E x’ is the epipolar line associated with x’ (l = E x’) Skew-

• ETx is the epipolar line associated with x (l’ = ETx) symmetric
• E e’ = 0 and ETe = 0 matrix
• E is singular (rank two)
• E has five degrees of freedom
– (3 for R, 2 for t because it’s up to a scale)
The Fundamental Matrix

Without knowing K and K’, we can define a similar

relation using unknown normalized coordinates

ˆxT E xˆ  0
xT F x  0 with F  K T E K 1
xˆ  K 1 x
xˆ   K 1 x 
Fundamental Matrix
(Faugeras and Luong, 1992)
Properties of the Fundamental matrix
X

x x’

xT F x  0 with F  K T E K 1
• F x’ = 0 is the epipolar line associated with x’
• FTx = 0 is the epipolar line associated with x
• F e’ = 0 and FTe = 0
• F is singular (rank two): det(F)=0
• F has seven degrees of freedom: 9 entries but defined up to scale, det(F)=0
Estimating the Fundamental Matrix

• 8-point algorithm
– Least squares solution using SVD on equations from 8 pairs of correspondences
– Enforce det(F)=0 constraint using SVD on F

• 7-point algorithm
– Use least squares to solve for null space (two vectors) using SVD and 7 pairs of
correspondences
– Solve for linear combination of null space vectors that satisfies det(F)=0

• Minimize reprojection error

– Non-linear least squares
Note: estimation of F (or E) is degenerate for a planar scene.
8-point algorithm
1. Solve a system of homogeneous linear equations
a. Write down the system of equations

xT F x  0
𝑢𝑢′ 𝑓11 + 𝑢𝑣 ′ 𝑓12 + 𝑢𝑓13 + 𝑣𝑢′ 𝑓21 + 𝑣𝑣 ′ 𝑓22 + 𝑣𝑓23 + 𝑢′ 𝑓31 + 𝑣 ′ 𝑓32 + 𝑓33 = 0

𝑓11
𝑓12
𝑢1 𝑢1 ′ 𝑢1 𝑣1 ′ 𝑢1 𝑣1 𝑢1 ′ 𝑣1 𝑣1 ′ 𝑣1 𝑢1 ′ 𝑣1 ′ 1
𝑓13
A𝒇 = ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ =0
𝑓
𝑢𝑛 𝑢𝑣′ 𝑢𝑛 𝑣𝑛 ′ 𝑢𝑛 𝑣𝑛 𝑢𝑛 ′ 𝑣𝑛 𝑣𝑛 ′ 𝑣𝑛 𝑢𝑛 ′ 𝑣𝑛 ′ 1 21
⋮
𝑓33
8-point algorithm
1. Solve a system of homogeneous linear equations
a. Write down the system of equations
b. Solve f from Af=0 using SVD

Matlab:
[U, S, V] = svd(A);
f = V(:, end);
F = reshape(f, [3 3])’;

For python, see

numpy.linalg.svd
Need to enforce singularity constraint
8-point algorithm
1. Solve a system of homogeneous linear equations
a. Write down the system of equations
b. Solve f from Af=0 using SVD
Matlab:
[U, S, V] = svd(A);
f = V(:, end);
F = reshape(f, [3 3])’;

2. Resolve det(F) = 0 constraint using SVD

Matlab:
[U, S, V] = svd(F);
S(3,3) = 0; For python, see
F = U*S*V’; numpy.linalg.svd
8-point algorithm
1. Solve a system of homogeneous linear equations
a. Write down the system of equations
b. Solve f from Af=0 using SVD
2. Resolve det(F) = 0 constraint by SVD

Notes:
• Use RANSAC to deal with outliers (sample 8 points)
– How to test for outliers?
How to test for outliers?
Problem with eight-point algorithm

 f11 
f 
 12 
 f13 
 
 f 21 
uu uv u vu vv v u v    1
f 22
 
 f 23 
f 
 31 
 f 32 
Problem with eight-point algorithm

 f11 
f 
 12 
 f13 
 
 f 21 
uu uv u vu vv v u v    1
f 22
 
 f 23 
f 
 31 
 f 32 
Poor numerical conditioning
Can be fixed by rescaling the data
The normalized eight-point algorithm
(Hartley, 1995)

• Center the image data at the origin, and scale it so

the mean squared distance between the origin and
the data points is 2 pixels
• Use the eight-point algorithm to compute F from the
normalized points
• Enforce the rank-2 constraint (for example, take SVD
of F and throw out the smallest singular value)
• Transform fundamental matrix back to original units:
if T and T’ are the normalizing transformations in the
two images, than the fundamental matrix in original
coordinates is T’T F T
VLFeat’s 800 most confident matches
among 10,000+ local features.
Epipolar lines
Keep only the matches at are “inliers” with
respect to the “best” fundamental matrix
The scale of algorithm name quality

better RANSAC
SIFT
Deep Learning

Optical Flow
Hough Transform
Neural Networks
Essential and Fundamental Matrix
worse Dynamic Programming
In class written Quiz format
• 15 to 20 short answer or multiple choice questions
• Typically can be done in half an hour
• No calculators needed
• Closed book
• Only covers material discussed in class, not book. But the book
is still a useful resource
• Covers all material through the quiz date