Stereo:

Epipolar geometry

Prof. H. Sahli
VUB-ETRO
2022-2023

1
CV
 Why multiple views?
Structure and depth are inherently ambiguous from single views.
=> Many-to-one: any points along same ray map to same point in
image

Optical center

p’1=p’2

P2

P1

2
CV
 Epipolar geometry

3
CV
 Epipolar geometry

4
CV
 Epipolar constraint

Another way to construct the epipolar plane, this time given 𝒙𝒙

5
CV
 Epipolar constraint

Potential matches for 𝒙𝒙 lie on the epipolar line 𝒍𝒍′

6
CV
 Stereo geometry, with calibrated cameras

If the stereo rig is calibrated, we know :

The relative pose between the two cameras: Rotation: R;
translation: T. X' = RX + T
c c
7
CV
 From geometry to algebra

X X`

(1)
X' = RX + T T × X′ = T × RX + T × T
Normal to the plane

(2)
T × X′ = T× RX
CV
X′ ⋅ (T × X′) = X′ ⋅ (T × RX ) = 0 8
 Essential matrix

(2) X′ ⋅ (T × RX ) = 0
X′ ⋅ ([Tx ]RX ) = 0
Let E = [T x ]R
′
X EX = 0
T

E is called the essential matrix,

• “encodes” the epipolar geometry of two views,
• relates corresponding image points between both cameras,
given the rotation and translation.
• Maps a point from left image to corresponding epipolar line in
right image:
• has rank 2.
• E depends only on the EXTRINSIC Parameters
9
CV
 Longuet-Higgins equation

𝐗𝐗 ′𝑇𝑇 𝐄𝐄𝐄𝐄 = 0
these points are in camera coordinate systems => relates viewing rays

𝑓𝑓 𝑓𝑓
x= X x′ = X’
𝑍𝑍 𝑍𝑍′

𝑍𝑍 ′ ′ 𝑇𝑇 𝑍𝑍
( 𝑥𝑥 ) 𝐄𝐄( 𝑥𝑥) = 0
𝑓𝑓 𝑓𝑓

′𝑇𝑇
𝑥𝑥 𝐄𝐄𝑥𝑥 = 0
This relates 2D image points

10
 Stereo system - Parameters

− Intrinsic parameters
− Characterize the transformation from image plane coordinates to
pixel coordinates in each camera
− Parameters intrinsic to each camera
− Extrinsic parameters (R, T)
− Describe the relative position and orientation of the two cameras

X' = RX + T

− Can be determined from the extrinsic parameters of each camera:

11
CV
 Epipolar Lines

− Let l be a line in the image:

𝑙𝑙 = 𝑎𝑎𝑎𝑎 + 𝑏𝑏𝑏𝑏 + 𝑐𝑐 = 0
− Using homogenous coordinates:

 x
p =  y  l′ = Ep T
p= 
l l=T
p 0
 
 1 
- Re-wrire (change notation)

′𝑇𝑇 ′𝑇𝑇
𝒙𝒙 𝐄𝐄𝒙𝒙 = 0 𝒑𝒑 𝐄𝐄𝒑𝒑 = 0
12
CV
 Epipolar Lines

′𝑇𝑇
𝑝𝑝 𝐄𝐄𝑝𝑝 = 0
a 
l′T =  b  P’ belongs to the epipolar line in the right image defined by l′ = Ep
 
 c 

𝑝𝑝′𝑇𝑇 𝐄𝐄𝑝𝑝 = 0
a 
l T =  b  p belongs to the epipolar line in the left image defined by l = E T p′
 
 c 
13
CV
 Epipoles
Epipoles belong to the epipolar lines

′𝑇𝑇 ′𝑇𝑇
𝑒𝑒 𝐄𝐄𝑝𝑝 = 0 𝑝𝑝 𝐄𝐄𝑒𝑒 = 0
And they belong to all the epipolar lines

𝑒𝑒 ′𝑇𝑇 𝐄𝐄 = 0 𝐄𝐄𝑒𝑒 = 0

14
CV
 Essential Matrix vs Homography

What’s the difference between the essential matrix and a homography?

They are both 3 x 3 matrices but …

𝒍𝒍′ = 𝐸𝐸𝒙𝒙 𝒙𝒙′ = 𝐻𝐻𝒙𝒙

Essential Matrix maps a Homography Matrix maps a
Point into a Line Point into a Point

15
CV
 Summary epipolar geometry

− Longuet-Higgins equation
′𝑇𝑇
𝑝𝑝 𝐸𝐸𝑝𝑝 = 0
− Epipolar lines:

l′ = Ep l = E T p′
p′T l′ = 0 p T l = 0
− Epipolees:

′𝑇𝑇 𝐸𝐸𝑒𝑒 = 0
𝑒𝑒 𝐸𝐸 = 0
16
CV
 Fundamental matrix F
• When Cameras are not calibrated, we talk about
Fundamental Matrix

Image
correspondences

m0 = K l p 𝑝𝑝
m1 = K r p′
𝑝𝑝’

𝑓𝑓𝑥𝑥 𝑠𝑠 𝑐𝑐𝑥𝑥 1 0 0 0
𝐾𝐾 = 0 𝑓𝑓𝑦𝑦 𝑐𝑐𝑦𝑦 0 1 0 0
0 0 1 0 0 1 0

17
CV
 Fundamental matrix F
m0 = K l p m1 = K r p′
𝑝𝑝′𝑇𝑇 𝐸𝐸𝑝𝑝 = 0
(K r−1m1 )T E (K l−1m0 ) = 0

m1T (K r−1T EK l−1 )m0 = 0

T
m1 Fm0 = 0
E = [T x ]R F = K r−1T R [Tx ]K l−1
• has rank 2
• depends on the INTRINSIC and EXTRINSIC Parameters
• Analogous to essential matrix. The fundamental matrix also tells how pixels in
each image are related to epipolar lines in the other image.

18
CV
 Fundamental matrix F
• When Cameras are not calibrated, we talk about
Fundamental Matrix
• Essential matrix: particular form of fundamental
matrix when calibration is available

Image
correspondences

19
CV
 Properties of the Fundamental Matrix

• Epipolar lines

• Epipoles
Image
correspondences

CV
20
 How to Derive F?
T
m1 Fm 0 = 0
x' xf11 + x' yf12 + x' f13 + y ' xf 21 + y ' yf 22 + y ' f 23 + xf 31 + yf 32 + f 33 = 0
separate known from unknown

[x' x, x' y, x' , y' x, y' y, y' , x, y,1][ f11 , f12 , f13 , f 21 , f 22 , f 23 , f 31 , f 32 , f 33 ]T = 0
(data) (unknowns)

x’1

x1

 x'1 x1 x'1 y1 x'1 y '1 x1 y '1 y1 y '1 x1 y1 1

          f = 0
 x ' n xn x 'n y n x 'n y ' n xn y 'n y n y 'n xn yn 1

CV Af = 0 21
 Automatic computation of F

Step 1. Extract features

Step 2. Compute a set of potential matches
Step 3. do

RANSAC
Step 3.1 select minimal sample (i.e. 7 matches)
Step 3.2 compute solution(s) for F
} (generate
hypothesis)
Step 3.3 determine inliers (verify hypothesis)
until Γ(#inliers,#samples)<95%

Step 4. Compute F based on all inliers

Step 5. Look for additional matches
Step 6. Refine F based on all correct matches

22
CV
Stereo – How we reconstruct 3D points
Assume
 parallel optical axes,
 known camera parameters (calibrated cameras):

What is the expression of Z?

23
CV
 Geometry for a simple stereo system

What is the expression of Z?

Similar triangles (pl, P, pr) and

(Ol, P, Or):

T + xl − xr T
=
Z− f Z

T
Z= f
xr − xl
disparity
(wrt to camera origin of image plane) 24
CV
 Essential matrix example: parallel cameras
R=I xl = [ x, y, f ]
T = [−d ,0,0]Τ xr = [ x ' , y ' , f ]
0 0 0
E = [T x]R = 0 0 d
0 –d 0

Τ
xr Exl = 0

For the parallel cameras,

image of any point must lie
on same horizontal line in
each image plane.
25
CV
 Depth from disparity

image I(xl,yl) Disparity map D(x,y) image I´(xr,yr)

(xr,yr)=(xl+D(x,y), yl)

So if we could find the corresponding points in two images,

we could estimate the relative depth…
26
CV
 Triangulation

x x'

Having two points x (on left image) and x’

(right image) projections of a scene point X
-> How to estimate X?
From perspective projection we have:
𝐱𝐱 = 𝐏𝐏𝐗𝐗 ( )
x′ = P ' X

27
 Triangulation

28
CV
 General Triangulation

29
CV
Stereo image rectification
In practice, it is
convenient if image
scanlines (rows) are the
epipolar lines.

reproject image planes onto a common

plane parallel to the line between optical
centers
pixel motion is horizontal after this transformation
two homographies (3x3 transforms), one for each
input image reprojection
30
CV source: Li Zhang
 Stereo image rectification
 Trick: We can rewarp them through rotation, mapping both image
planes to a common plane parallel to the baseline, this is called rectification

 For this rotation around the camera center, the 3D structure must not be
known

EpipolarPlane
Epipolar Plane

Source: Prof. Dr.-Ing. Andreas Geiger

31
CV
 Stereo image rectification
Calculating the Rectifying Rotation Matrix

Principal Axis

Recti fi ed
Camera 1

Epipolar Plane

Camera 1 Camera 2

Source: Prof. Dr.-Ing. Andreas Geiger

32
CV
 Stereo image rectification
 To rectify the image, we calculate a rectification rotation 𝑹𝑹𝒓𝒓𝒓𝒓𝒓𝒓𝒓𝒓 = (𝒓𝒓𝟏𝟏 , 𝒓𝒓𝟐𝟐 , 𝒓𝒓𝟑𝟑 )𝑻𝑻
𝒕𝒕
 With 𝒓𝒓𝟏𝟏 = 𝒕𝒕 ; 𝒓𝒓𝟐𝟐 = 0,0,1 𝑇𝑇 × 𝒓𝒓𝟏𝟏 ;𝒓𝒓𝟑𝟑 = 𝒓𝒓𝟏𝟏 × 𝒓𝒓𝟐𝟐
𝟐𝟐
 As the epipole in the first image is in the direction of 𝒓𝒓𝟏𝟏 , it is easy to see that
the rotated epipole is ideal 𝑹𝑹𝒓𝒓𝒓𝒓𝒓𝒓𝒓𝒓 𝒓𝒓𝟏𝟏 = 1,0,0 𝑇𝑇
 Thus, applying 𝑹𝑹𝒓𝒓𝒓𝒓𝒓𝒓𝒓𝒓 to the first camera leads to parallel and horizontal
epipolar lines.

Rectification Algorithm:
� , decompose into 𝒕𝒕 and 𝑹𝑹, and construct 𝑹𝑹𝒓𝒓𝒓𝒓𝒓𝒓𝒓𝒓 as above
1. Estimate 𝑬𝑬
2. Warp pixels in the first image as follows: 𝑥𝑥�1′ = 𝑲𝑲𝑹𝑹𝒓𝒓𝒓𝒓𝒓𝒓𝒓𝒓 𝑲𝑲−𝟏𝟏 ′
𝟏𝟏 𝑥𝑥̅1
3. Warp pixels in the second image as follows: 𝑥𝑥�2′ = 𝑲𝑲𝑹𝑹𝑹𝑹𝒓𝒓𝒓𝒓𝒓𝒓𝒓𝒓 𝑲𝑲−𝟏𝟏 ′
𝟐𝟐 𝑥𝑥̅ 2

With 𝑲𝑲 a shared projection matrix that can be chosen arbitrarily (e.g 𝑲𝑲= 𝑲𝑲𝟐𝟐 )
In practice, the inverse transformation is used for warping

33
CV
 Stereo image rectification: example

34
CV Source: Alyosha Efros
 The Correspondence Problem
• Methods for establishing correspondences
− Two issues
− How to select candidate matches?
− How to determine the goodness of a match?
− Two main classes of correspondence (matching) algorithm:
− Correlation-based
− Attempt to establish a correspondence by matching image
intensities – usually over a window of pixels in each image
⇒Dense disparity maps
− Feature-based
− Attempt to establish a correspondence by matching a sparse
sets of image features – usually edges
− Disparity map is sparse
− Number of points is related to the number of image
features identified

38
CV
 Correlation-Based Methods
• Match image sub-windows in the two images using image
correlation
− oldest technique for finding correspondence between image pixels
• Scene points must have the same intensity in each image
−
Assumes
a) All objects are perfect Lambertian scatterers
ie the reflected intensity is not dependent on angle or
objects scatter light uniformly in all directions
• Informally: matte surfaces only
b) Fronto-planar surfaces
− (Visible) surfaces of all objects are perpendicular
to camera optical axes

39
CV
 Matching Windows:
?
=

f g

Sum of Squared
Minimize Differences

Cross correlation
Maximize

For each window, match to closest window on epipolar line in other image.

CV 40
 Matching Windows:
Minimize Sum of Squared
Differences

Maximize
Cross correlation

It is closely related to the SSD:

CV 41
 Solving for correspondences
Matching through correlation:

42
CV
 Correlation-Based Methods

43
CV
 Correlation-Based Methods
• Improvements
− Do not use image intensity values,
use intensity gradients instead!
− One scheme calculates thresholded signed gradient magnitudes at
each pixel
− Compute the gradient magnitude at each pixel in the two images
without smoothing
− Map the gradient magnitude values into three values: -1, 0, 1
(by thresholding the gradient magnitude)
− More sensitive correlations are produced this way
+ several dozen more

44
CV
 Feature-Based Methods

45
CV
Correlation vs. feature-based approaches
• Correlation methods
− Easier to implement
− Provide a dense disparity map (useful for reconstructing surfaces)
− Need textured images to work well (many false matches otherwise)
− Do not work well when viewpoints are very different, due to
− change in illumination direction
− Objects are not perfect (Lambertian) scatterers
− foreshortening
− perspective problem – surfaces are not fronto-planar

• Feature-based methods
− Suitable when good features can be extracted from the scene
− Faster than correlation-based methods
− Provide sparse disparity maps
− OK for applications like visual navigation
− Relatively insensitive to illumination changes
46
CV
How can we improve window-based matching?
• The similarity constraint is local (each reference
window is matched independently)
• Need to enforce non-local correspondence
constraints

47
CV
 Non-local constraints
• Uniqueness
• For any point in one image, there should be at most one
matching point in the other image

48
CV
 Non-local constraints
• Uniqueness
• For any point in one image, there should be at most one
matching point in the other image
• Ordering
• Corresponding points should be in the same order in both views

49
CV
 Non-local constraints
• Uniqueness
• For any point in one image, there should be at most one
matching point in the other image
• Ordering
• Corresponding points should be in the same order in both views

Ordering constraint doesn’t hold 50

CV
 Non-local constraints
• Uniqueness
• For any point in one image, there should be at most one
matching point in the other image
• Ordering
• Corresponding points should be in the same order in both views
• Smoothness
• We expect disparity values to change slowly (for the most part)

51
CV
 Stereo matching as energy minimization

I1 I2 D

W1(i) W2(i+D(i)) D(i)

E = α Edata ( I1 , I 2 , D) + β Esmooth ( D)

Edata = ∑ (W1 (i ) − W2 (i + D(i )) )

2
Esmooth = ∑ ρ (D(i) − D( j ))
neighbors i , j
i

• Energy functions of this form can be minimized using

several approaches (e.g gradient descent, graph cuts,
etc..)
Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI
2001 52
CV
 Summary stereo vision system

Source Stefano Mattoccia 53

CV
 Available Stereo Cameras

Source Stefano Mattoccia 54

CV
 Summary

Depth from stereo: main idea is to triangulate from

corresponding image points.

Epipolar geometry defined by two cameras

• We’ve assumed known extrinsic parameters relating their poses

Epipolar constraint limits where points from one view will be

imaged in the other
• Makes search for correspondences quicker

Terms: epipole, epipolar plane / lines, disparity,

intrinsic/extrinsic parameters, essential matrix,
rectification

55
CV
 Summary
Essential Matrix Longuet-Higgins equation
E = [T x ]R 𝑝𝑝′𝑇𝑇 𝐸𝐸𝑝𝑝 = 0
Epipoles belong to the epipolar lines
′𝑇𝑇
𝑒𝑒 𝐄𝐄𝑝𝑝 = 0 𝑝𝑝′𝑇𝑇 𝐄𝐄𝑒𝑒 = 0
And they belong to all the epipolar lines
𝑒𝑒 ′𝑇𝑇 𝐄𝐄 = 0 𝐄𝐄𝑒𝑒 = 0
Fundamental Matrix
F = K r−1T R [Tx ]K l−1
• has rank 2
• depends on the INTRINSIC and EXTRINSIC Parameters
• Analogous to essential matrix. The fundamental matrix also tells how pixels in
each image are related to epipolar lines in the other image.

56
CV