Lecture 5

Epipolar Geometry

Professor Silvio Savarese

Computational Vision and Geometry Lab

Silvio Savarese & Jeanette Bohg Lecture 5 - 26-Jan-21

Lecture 5
Epipolar Geometry
• Why is stereo useful?
• Epipolar constraints
• Essential and fundamental matrix
• Estimating F
• Examples
Reading: [AZ] Chapter: 4 “Estimation – 2D perspective transformations
Chapter: 9 “Epipolar Geometry and the Fundamental Matrix Transformation”
Chapter: 11 “Computation of the Fundamental Matrix F”
[FP] Chapter: 7 “Stereopsis”
Chapter: 8 “Structure from Motion”

Silvio Savarese & Jeanette Bohg Lecture 5 - 26-Jan-21

 Recovering structure from a single view
Pinhole perspective projection

P
p
Ow
Scene C
Calibration rig Camera K

From calibration rig ® location/pose of the rig, K

From points and lines at infinity

+ orthogonal lines and planes ® structure of the scene, K

Knowledge about scene (point correspondences, geometry of lines & planes, etc…
 Recovering structure from a single view
Pinhole perspective projection

P
p
Ow
Scene C
Calibration rig Camera K

Why is it so difficult?
Intrinsic ambiguity of the mapping from 3D to image (2D)
 Recovering structure from a single view

Intrinsic ambiguity of the mapping from 3D to image (2D)

Courtesy slide S. Lazebnik

Two eyes help!
 Two eyes help!

P = l × l"
[Eq. 1] P

l l'

p’
p
K =known K’ =known

R, T
O1 O2

This is called triangulation

 Triangulation
• Find P* that minimizes
d( p, M P*) + d( p', M ' P*) [Eq. 2]

P
P*

MP*
M’P*
p’
p
imag e2
e1 imag

O1 O2
 Multi (stereo)-view geometry
• Camera geometry: Given corresponding points
in two images, find camera matrices, position
and pose.

• Scene geometry: Find coordinates of 3D point

from its projection into 2 or multiple images.

• Correspondence: Given a point p in one image,

how can I find the corresponding point p’ in
another one?
 Epipolar geometry
P

p p’

e e’

O1 O2
• Epipolar Plane • Epipoles e, e’
• Baseline = intersections of baseline with image planes
= projections of the other camera center
• Epipolar Lines
Example of epipolar lines
 Example: Parallel image planes

v v

e p p’ e’

u u

O1 O2
• Baseline intersects the image plane at infinity
• Epipoles are at infinity
• Epipolar lines are parallel to u axis
Example: Parallel Image Planes
 Example: Forward translation

e’ P

O2

O1

• The epipoles have same position in both images

• Epipole called FOE (focus of expansion)
 Epipolar Constraint

Where is p’?
p

- Two views of the same object

- Given a point on left image, how can I find the corresponding point on right image?
 Epipolar geometry

P Epipolar line 2

p p’

O1 O2
 Epipolar Constraint

Epipolar line 2
p p’
 Epipolar Constraint
P
v

p’
p
u

O1
R, T O2

" %
M = K !" I 0 #$ M ' = K '$ RT
#
−RT T '
&
! u $ " u! %
# & $ '
M P = # v & = p [Eq. 3] M ! P = $ v! ' = p' [Eq. 4]
# 1 & $ 1 '
" % # &
 Epipolar Constraint ! 1 0 0 $
# &
K canonical =# 0 1 0 &
P # 0 0 1 &
" %

p’
p

O1
R, T O2

K = K’ are known " %

M = K !" I 0 #$ (canonical cameras) M ' = K '$ RT
#
−RT T '
&

M = [I 0]
" % [Eq. 6]
[Eq. 5] M ' = $ RT −RT T '
# &
 Epipolar Constraint
P

p’
p

O1
R, T O2

p’ in first camera reference system is = R p! + T

T × ((R p") + T ) = T × (R p") is perpendicular to epipolar plane

® p × [T ´ ( R p¢)] = 0
T
[Eq. 7]
 Cross product as matrix multiplication

é 0 - az a y ù ébx ù
ê úê ú
a ´ b = ê az 0 - ax ú êby ú = [a´ ]b
ê- a y ax 0 úû êëbz úû
ë
 Epipolar Constraint
P

p’
p

O1
R, T O2

p × [T ´ (R p¢)] = 0 ® p × [T´ ]× R p¢ = 0
T T

[Eq. 8] [Eq. 9]
E = Essential matrix
(Longuet-Higgins, 1981)
 Epipolar Constraint
T P
p ⋅ E p' = 0
[Eq. 10]

p p’
l l’
e e’

O1 O2

• l = E p’ is the epipolar line associated with p’

• l’ = ET p is the epipolar line associated with p
• E e’ = 0 and ET e = 0
• E is 3x3 matrix; 5 DOF
• E is singular (rank two)
 Epipolar Constraint
P

p’
p

R, T
O1
O2

M = K[I 0] " T
M ' = K '$ R
#
T %
−R T '
&
pc = K −1 p [Eq. 11] pc! = K '−1 p! [Eq. 12]
 −1 Epipolar Constraint
pc = K p
[Eq. 11] P
−1
pc! = K ' p!
[Eq. 12]

p’
p

O1
O2
[Eq.9]

p ⋅ [T× ] ⋅ R pc# = 0 ® (K p) × [T´ ]× R K¢ p¢ = 0

T
c
-1 T -1

× [T´ ]× R K¢ p¢ = 0 ® p F p¢ = 0
T -T -1 T
p K [Eq. 13]
 Epipolar Constraint
P

p’
p

O1
O2

p F p¢ = 0 × [T´ ]× R K ¢
T -T -1
[Eq. 13]
F=K
[Eq. 14]
F = Fundamental Matrix
(Faugeras and Luong, 1992)
 Epipolar Constraint
T P
p ⋅ F p' = 0

p p’
l l’
e e’

O1 O2

• l = F p’ is the epipolar line associated with p’

• l’= FT p is the epipolar line associated with p
• F e’ = 0 and FT e = 0
• F is 3x3 matrix; 7 DOF
• F is singular (rank two)
 Why F is useful?

l’ = FT p
p p’

- Suppose F is known
- No additional information about the scene and camera is given
- Given a point on left image, we can compute the corresponding epipolar line in the second imag
 Why F is useful?
• F captures information about the epipolar geometry of
2 views + camera parameters

• MORE IMPORTANTLY: F gives constraints on how the

scene changes under view point transformation
(without reconstructing the scene!)
• Powerful tool in:
• 3D reconstruction
• Multi-view object/scene matching
 Estimating F
The Eight-Point Algorithm
(Longuet-Higgins, 1981)
(Hartley, 1995) P

p’
p

O1
O2

p F p¢ = 0
T
 Estimating F
! u $ " u! %
[Eq. 13] p F p¢ = 0
T # &
p =# v &
$ '
p! = $ v! '
# 1 & $ 1 '
" % # &

æ F11 F12 F13 öæ u ' ö

ç ÷ç ÷
( u , v ,1) ç F21 F22 F23 ÷ç v ' ÷ = 0 æ F11 ö
ç ÷
çF F32 F33 ÷ç 1 ÷ ç F12 ÷
è 31 øè ø ç F13 ÷
ç ÷
ç F21 ÷
( uu ', uv ', u, vu ', vv ', v, u ', v ',1) çç F22 ÷÷ = 0
ç F23 ÷
ç ÷
ç F31 ÷
ç F32 ÷ [Eq. 14]
Let’s take 8 corresponding points çç ÷÷
è F33 ø
Estimating F
 Estimating F

!F $
# 11 &
# F12 &
# &
# F13 &
#F &
# 21 &
( )
ui u 'i ,ui v 'i ,ui ,vi u 'i ,vi v 'i ,vi ,u 'i ,vi ',1 # F22 & = 0 [Eq. 14]
# &
# F23 &
#F &
# 31 &
# F32 &
# &
" F33 %
 Estimating F
æ F11 ö
æ u1u '1 u1v '1 u1 v1u '1 v1v '1 v1 u '1 v '1 1ö ç ÷
ç ÷ ç F12 ÷
ç u2u '2 u2v '2 u2 v2u '2 v2v '2 v2 u '2 v '2 1÷
ç F13 ÷
ç u3u '3 u3v '3 u3 v3u '3 v3v '3 v3 u '3 v '3 1÷ ç ÷
ç ÷ ç F21 ÷
ç u4u '4 u4v '4 u4 v4u '4 v4v '4 v4 u '4 v '4 1÷ ç ÷
F22 = 0 [Eqs. 15]
W ç u5u '5
ç
u5v '5 u5 v5u '5 v5v '5 v5 u '5 v '5 1÷ ç ÷
÷ ç F23 ÷
ç u6u '6 u6 v '6 u6 v6u '6 v6 v '6 v6 u '6 v '6 1÷ ç ÷
çu u ' ÷ ç F31 ÷
u7 v '7 u7 v7u '7 v7 v '7 v7 u '7 v '7 1
ç 7 7 ÷ ç F32 ÷
çu u' u8v '8 u8 v8u '8 v8v '8 v8 u '8 v '8 1÷ø ç ÷
è 8 8 çF ÷ f
è 33 ø
• Homogeneous system Wf = 0
• Rank 8 A non-zero solution exists (unique)

• If N>8 Lsq. solution by SVD! F̂

f =1
 F̂ satisfies: p F̂ p¢ = 0
T

^ ^
and estimated F may have full rank (det(F) ≠0)

But remember: fundamental matrix is Rank2

Find F that minimizes F - F̂ = 0

Frobenius norm (*)

Subject to det(F)=0

SVD (again!) can be used to solve this problem

(*) Sq. root of the sum of squares of all entries
 Find F that minimizes F - F̂ = 0
Frobenius norm (*)

Subject to det(F)=0

! s 0 0 $
# 1 &
F =U# 0 s2 0 &V T Where:
# & ! s
#" 0 0 0 &% 0 0 $
# 1 &
U# 0 s2 0 &V T = SVD( F̂)
# &
#" 0 0 s3 &%
[HZ] pag 281, chapter 11, “Computation of F”
Data courtesy of R. Mohr and B. Boufama.
Mean errors:
10.0pixel
9.1pixel
Problems with the 8-Point Algorithm

Lsq solution
W f = 0, by SVD

f =1 F
- Recall the structure of W:
- do we see any potential
(numerical) issue?
Problems with the 8-Point Algorithm

Wf = 0
æ F11 ö
æ u1u '1 u1v '1 u1 v1u '1 v1v '1 v1 u '1 v '1 1ö ç ÷
ç ÷ ç F12 ÷
ç u2u '2 u2v '2 u2 v2u '2 v2v '2 v2 u '2 v '2 1÷
ç F13 ÷
ç u3u '3 u3v '3 u3 v3u '3 v3v '3 v3 u '3 v '3 1÷ ç ÷
ç ÷ ç F21 ÷
ç u4u '4 u4v '4 u4 v4u '4 v4v '4 v4 u '4 v '4 1÷ ç ÷
F22 = 0
ç u5u '5 u5v '5 u5 v5u '5 v5v '5 v5 u '5 v '5 ÷
1 ç ÷
ç ÷ç F ÷
ç u6u '6 u6 v '6 u6 v6u '6 v6 v '6 v6 u '6 v '6 1÷ ç 23 ÷
çu u ' ÷ ç F31 ÷
u7 v '7 u7 v7u '7 v7 v '7 v7 u '7 v '7 1
ç 7 7 ÷ ç F32 ÷
çu u' u8v '8 u8 v8u '8 v8v '8 v8 u '8 v '8 1÷ø ç ÷
è 8 8 çF ÷
è 33 ø
• Highly un-balanced (not well conditioned)

HZ pag 108
• Values of W must have similar magnitude
• This creates problems during the SVD decomposition
 Normalization
IDEA: Transform image coordinates such that the matrix
W becomes better conditioned (pre-conditioning)

For each image, apply a transformation T (translation

and scaling) acting on image coordinates such that:

• Origin = centroid of image points

• Mean square distance of the image
points from origin is ~2 pixels
 Example of normalization

2
pi
xe
ls
Coordinate system of the image
Coordinate system of the
after applying T
image before applying T

• Origin = centroid of image points

• Mean square distance of the image points from origin is ~2 pixels
 Normalization

2 pixels 2 pixels

qi = T pi qi¢ = T ¢ pi¢
The Normalized Eight-Point Algorithm

0. Compute T and T’ for image 1 and 2, respectively

1. Normalize coordinates in images 1 and 2:
qi = T pi qi¢ = T ¢ pi¢

2. Use the eight-point algorithm to compute F̂q from the

corresponding points q i and q’i .

1. Enforce the rank-2 constraint: ® Fq such that:

q T Fq q¢ = 0

2. De-normalize Fq: F = T Fq T
T ' det( Fq ) = 0
Without normalization
Mean errors:
10.0pixel
9.1pixel

Mean errors:
With normalization

1.0pixel
0.9pixel
The Fundamental Matrix Song

http://danielwedge.com/fmatrix/
Next lecture:
Stereo systems

Silvio Savarese & Jeanette Bohg Lecture 5 - 26-Jan-21

 Example: Parallel image planes

e p p’ e’
y u

O1 x O2
Hint :
K1=K2 = known E=? R=I T = (T, 0, 0)
x parallel to O1O2
 Essential matrix for parallel images

E = [T´ ]× R

" 0 −Tz Ty % " 0 0 0 %

$ ' $ '
E = $ Tz 0 −Tx ' R = $ 0 0 −T '
$ ' $ 0 T 0 '
$# −Ty Tx 0 ' # &
&
[Eq. 20]
T=[T 0 0]
R=I
 Example: Parallel image planes

v v’

e l p l’
p’ e’
y u u’

O1 x

" 0 0 0 % " u' % " 0 %

What are the $ '$ ' $ '
directions of l = E p' = $ 0 0 −T ' $ v' ' = $ −T ' horizontal!
epipolar lines? $ 0 T 0 ' $ 1 ' $ T v' '
# &# & # &
 Example: Parallel image planes

v v’

e p p’ e’
y u u’

O1 x

How are p T
and p’ p ⋅ E p' = 0
related?
 Example: Parallel image planes

v v’

e p p’ e’
y u u’

O1 x

How are p é0 0 0 ù æ u¢ ö æ 0 ö
ç ÷ ç ÷
andpT Ep’p¢ = 0 Þ ( u v 1) êê0 0 -T úú ç v¢ ÷ = 0 Þ ( u v 1) ç -T ÷ = 0 Þ Tv = Tv¢
related? êë0 T 0 ûú èç 1 ø÷ ç ÷
è Tv¢ ø ⇒ v = v"
 Example: Parallel image planes

v v’

e p p’ e’
y u u’

O1 x

Rectification: making two images “parallel”

Why it is useful? • Epipolar constraint ® v = v’
• New views can be synthesized by linear interpolation
Application: view morphing
S. M. Seitz and C. R. Dyer, Proc. SIGGRAPH 96, 1996, 21-30
 Rectification

H1 I1 Is

î1
u I2
v
îS
u
v

C1 H2
u
v
î2
Cs

C2
From its reflection!
 Deep view morphing
D. Ji, J. Kwon, M. McFarland, S. Savarese, CVPR 2017
 Deep view morphing
D. Ji, J. Kwon, M. McFarland, S. Savarese, CVPR 2017
 Deep view morphing
D. Ji, J. Kwon, M. McFarland, S. Savarese, CVPR 2017