Image homographies

16-385 Computer Vision

http://16385.courses.cs.cmu.edu/ Spring 2021, Lecture 8
 Textbook for geometry part of class

• Amazing resource for everything related to

geometric methods in computer vision.

• Great introduction to projective geometry as

well.
 Overview of today’s lecture

• Motivation: panoramas.

• Back to warping: image homographies.

• Computing with homographies.

• The direct linear transform (DLT).

• Random Sample Consensus (RANSAC).

 Slide credits

Most of these slides were adapted from:

• Kris Kitani (15-463, Fall 2016).

• Noah Snavely (Cornell).
Motivation for image alignment: panoramas.
 How do you create a panorama?
Panorama: an image of (near) 360o field of view.
 How do you create a panorama?
Panorama: an image of (near) 360o field of view.

1. Use a very wide-angle lens.

 Wide-angle lenses
Fish-eye lens: can produce (near)
hemispherical field of view.

What are the pros and cons of this?

 How do you create a panorama?
Panorama: an image of (near) 360o field of view.

1. Use a very wide-angle lens.

• Pros: Everything is done optically, single capture.
• Cons: Lens is super expensive and bulky, lots of distortion (can be dealt-with in post).

Any alternative to this?

 How do you create a panorama?
Panorama: an image of (near) 360o field of view.

1. Use a very wide-angle lens.

• Pros: Everything is done optically, single capture.
• Cons: Lens is super expensive and bulky, lots of distortion (can be dealt-with in post).

2. Capture multiple images and combine them.

 Panoramas from image stitching

1. Capture multiple images

from different viewpoints.

2. Stitch them together into

a virtual wide-angle
image.
How do we stitch images from different viewpoints?

Will standard stitching work?

1. Translate one image relative to another.
2. (Optionally) find an optimal seam.
How do we stitch images from different viewpoints?

Will standard stitching work?

1. Translate one image relative to another.
2. (Optionally) find an optimal seam.

left on top right on top

Translation-only stitching is not enough to mosaic these images.

How do we stitch images from different viewpoints?

What else can we try?

How do we stitch images from different viewpoints?

Use image homographies.

Back to warping: image homographies
Classification of 2D transformations
 Classification of 2D transformations

PP3

PP1
Which kind of transformation is needed to warp
projective plane 1 into projective plane 2? PP2
 Classification of 2D transformations

PP3

PP1
Which kind of transformation is needed to warp
projective plane 1 into projective plane 2? PP2
• A projective transformation (a.k.a. a homography).
 Warping with different transformations
translation affine projective (homography)
 View warping
original view synthetic top view synthetic side view

What are these black areas near the

boundaries?
 Virtual camera rotations

synthetic
rotations

original view
 Image rectification

two
original
images

rectified and stitched

Street art
Carpet illusion
 Understanding geometric patterns
What is the pattern on the floor?

magnified view of floor

 Understanding geometric patterns
What is the pattern on the floor?

y
ph
ra
og
m
Ho

reconstruction from
rectified view
magnified view of floor rectified view
 Understanding geometric patterns
Very popular in renaissance drawings (when perspective was discovered)

rectified view
of floor

reconstruction
 A weird painting
Holbein, “The Ambassadors”
 A weird painting
Holbein, “The Ambassadors”

What’s this???
 A weird painting
Holbein, “The Ambassadors”

rectified view

skull under anamorphic perspective

 A weird painting
Holbein, “The Ambassadors”

DIY: use a polished spoon to see the

skull
 Panoramas from image stitching

1. Capture multiple images

from different viewpoints.

2. Stitch them together into

a virtual wide-angle
image.
When can we use homographies?
 We can use homographies when…

1. … the scene is planar; or

2. … the scene is very far or has

small (relative) depth variation
→ scene is approximately
planar
 We can use homographies when…

3. … the scene is captured under camera rotation only (no translation or pose change)

More on why this is the case in a later lecture.

Computing with homographies
 Classification of 2D transformations

PP3

PP1
Which kind of transformation is needed to warp
projective plane 1 into projective plane 2? PP2
• A projective transformation (a.k.a. a homography).
 Applying a homography
𝑥 𝑥
1. Convert to homogeneous coordinates: 𝑝= 𝑦 ⇒ 𝑃= 𝑦
1
What is the size of the homography matrix?

2. Multiply by the homography matrix: 𝑃( = 𝐻 ⋅ 𝑃

𝑥 ( 𝑥 (, (
𝑤
3. Convert back to heterogeneous coordinates: 𝑃( = 𝑦 ( (
⇒ 𝑝 = (
𝑦,
𝑤( 𝑤(
 Applying a homography
𝑥 𝑥
1. Convert to homogeneous coordinates: 𝑝= 𝑦 ⇒ 𝑃= 𝑦
1
What is the size of the homography matrix? Answer: 3 x 3

2. Multiply by the homography matrix: 𝑃( = 𝐻 ⋅ 𝑃

How many degrees of freedom does the homography matrix have?

𝑥 ( 𝑥 (, (
𝑤
3. Convert back to heterogeneous coordinates: 𝑃( = 𝑦 ( (
⇒ 𝑝 = (
𝑦,
𝑤( 𝑤(
 Applying a homography
𝑥 𝑥
1. Convert to homogeneous coordinates: 𝑝= 𝑦 ⇒ 𝑃= 𝑦
1
What is the size of the homography matrix? Answer: 3 x 3

2. Multiply by the homography matrix: 𝑃( = 𝐻 ⋅ 𝑃

How many degrees of freedom does the homography matrix have? Answer: 8

𝑥 ( 𝑥 (, (
𝑤
3. Convert back to heterogeneous coordinates: 𝑃( = 𝑦 ( (
⇒ 𝑝 = (
𝑦,
𝑤( 𝑤(
The direct linear transform (DLT)
 Create point correspondences
Given a set of matched feature points 𝑝1 , 𝑝1( find the best estimate of 𝐻 such that

𝑃( = 𝐻 ⋅ 𝑃

𝑝- 𝑝-( 𝑝.(
𝑝.

𝑝0 𝑝/ 𝑝0( 𝑝/(

original image target image

How many correspondences do we need?
 Determining the homography matrix
Write out linear equation for each correspondence:

𝑃( = 𝐻 ⋅ 𝑃 or
 Determining the homography matrix
Write out linear equation for each correspondence:

𝑃( = 𝐻 ⋅ 𝑃 or

Expand matrix multiplication:

 Determining the homography matrix
Write out linear equation for each correspondence:

𝑃( = 𝐻 ⋅ 𝑃 or

Expand matrix multiplication:

Divide out unknown scale factor:

How do you
rearrange terms
to make it a
linear system?
Just rearrange the terms
 Determining the homography matrix
Re-arrange terms:

Re-write in matrix form: How many equations

from one point
correspondence?
 Determining the homography matrix
Stack together constraints from multiple point correspondences:

Homogeneous linear least squares problem

 Reminder: Determining affine transformations
Affine transformation:

Vectorize transformation
parameters:

Stack equations from point

correspondences:

Notation in system form:

 Reminder: Determining affine transformations
Convert the system to a linear least-squares problem: In Python:
x = numpy.linalg.
solve(A, b)
Expand the error:

Minimize the error:

Set derivative to 0

Note: You almost never want to

Solve for x
compute the inverse of a matrix.
 Determining the homography matrix
Stack together constraints from multiple point correspondences:

Homogeneous linear least squares problem

• How do we solve this?
 Determining the homography matrix
Stack together constraints from multiple point correspondences:

Homogeneous linear least squares problem

• Solve with SVD
Singular value decomposition
orthonormal diagonal orthonormal

T
A = U⌃V
n⇥m n⇥n n⇥m m⇥m

9
X
T
= i ui v i
n⇥1 1⇥m

i=1
 General form of total least squares

(Warning: change of notation. x is a vector of parameters!)

(matrix form)

constraint

(Rayleigh quotient)
minimize
minimize
subject to

Solution is the eigenvector Solution is the column of V

corresponding to smallest corresponding to smallest singular
eigenvalue of (equivalent) value
 Solving for H using DLT
Given solve for H such that

1. For each correspondence, create 2x9 matrix

2. Concatenate into single 2n x 9 matrix

3. Compute SVD of

4. Store singular vector of the smallest singular value

5. Reshape to get
Linear least squares estimation only works when the transform function is linear! (duh)

Also doesn’t deal well with outliers.

 Create point correspondences

𝑝- 𝑝-( 𝑝.(
𝑝.

𝑝0 𝑝/ 𝑝0( 𝑝/(

original image target image

How do we automate this step?

 The image correspondence pipeline
1. Feature point detection
• Detect corners using the Harris corner detector.

2. Feature point description

• Describe features using the Multi-scale oriented patch descriptor.

3. Feature matching
 The image correspondence pipeline
1. Feature point detection
• Detect corners using the Harris corner detector.

2. Feature point description

• Describe features using the Multi-scale oriented patch descriptor.

3. Feature matching
bad
cor
res
pon
den
ce

good correspondence
Random Sample Consensus (RANSAC)
 Fitting lines
(with outliers)

Algorithm:
1. Sample (randomly) the number of points required to fit the model
2. Solve for model parameters using samples
3. Score by the fraction of inliers within a preset threshold of the model

Repeat 1-3 until the best model is found with high confidence
 Fitting lines
(with outliers)

Algorithm:
1. Sample (randomly) the number of points required to fit the model
2. Solve for model parameters using samples
3. Score by the fraction of inliers within a preset threshold of the model

Repeat 1-3 until the best model is found with high confidence
 Fitting lines
(with outliers)

Algorithm:
1. Sample (randomly) the number of points required to fit the model
2. Solve for model parameters using samples
3. Score by the fraction of inliers within a preset threshold of the model

Repeat 1-3 until the best model is found with high confidence
 Fitting lines
(with outliers)

Algorithm:
1. Sample (randomly) the number of points required to fit the model
2. Solve for model parameters using samples
3. Score by the fraction of inliers within a preset threshold of the
model

Repeat 1-3 until the best model is found with high confidence
 Fitting lines
(with outliers)

Algorithm:
1. Sample (randomly) the number of points required to fit the model
2. Solve for model parameters using samples
3. Score by the fraction of inliers within a preset threshold of the model

Repeat 1-3 until the best model is found with high confidence
 How to choose parameters?
• Number of samples N
– Choose N so that, with probability p, at least one random sample is free from
outliers (e.g. p=0.99) (outlier ratio: e )
• Number of sampled points s
– Minimum number needed to fit the model
• Distance threshold δ
–Choose δ so that a good point with noise is likely (e.g., prob=0.95) within threshold

Number of samples N required

 Given two images…

find matching features (e.g., SIFT) and a translation transform

Matched points will usually contain bad correspondences

bad
cor
res
pon
den
ce

good correspondence

how should we estimate the transform?

 LLS will find the "average" transform

‘average’
transform

solution is corrupted by bad correspondences

 Use RANSAC

How many correspondences to compute translation transform?

Need only one correspondence, to find translation model
Pick one correspondence, count inliers

1 correspondence
Pick one correspondence, count inliers

2 inliers
Pick one correspondence, count inliers

1 correspondence
Pick one correspondence, count inliers

5 inliers
 Pick one correspondence, count inliers

5 inliers

Pick the model with the highest number of inliers!

Estimating homography using RANSAC

• RANSAC loop
1. Get four point correspondences (randomly)
2. Compute H (DLT)
3. Count inliers
4. Keep if largest number of inliers
• Recompute H using all inliers
Estimating homography using RANSAC

• RANSAC loop
1. Get four point correspondences (randomly)
2. Compute H using DLT
3. Count inliers
4. Keep if largest number of inliers
• Recompute H using all inliers
Estimating homography using RANSAC

• RANSAC loop
1. Get four point correspondences (randomly)
2. Compute H using DLT
3. Count inliers
4. Keep if largest number of inliers
• Recompute H using all inliers
Estimating homography using RANSAC

• RANSAC loop
1. Get four point correspondences (randomly)
2. Compute H using DLT
3. Count inliers
4. Keep H if largest number of inliers
• Recompute H using all inliers
Estimating homography using RANSAC

• RANSAC loop
1. Get four point correspondences (randomly)
2. Compute H using DLT
3. Count inliers
4. Keep H if largest number of inliers
• Recompute H using all inliers
Useful for…
 The image correspondence pipeline
1. Feature point detection
• Detect corners using the Harris corner detector.

2. Feature point description

• Describe features using the Multi-scale oriented patch descriptor.

3. Feature matching and homography estimation

• Do both simultaneously using RANSAC.