CS 294-167 Spring 2022

Geometry and Learning for 3D Vision

Yi Ma

UC Berkeley

Invitation to 3D vision
 Uncalibrated Camera

Lecture 5
Uncalibrated Geometry & Stratification

MASKS © 2004 Invitation to 3D vision

 Uncalibrated Camera

calibrated
coordinates

Linear transformation

pixel
coordinates

MASKS © 2004 Invitation to 3D vision

 Overview

• Uncalibrated epipolar geometry

• Ambiguities in image formation

• Stratified reconstruction

• Autocalibration with partial/full scene knowledge

MASKS © 2004 Invitation to 3D vision

 Uncalibrated Camera

Calibrated camera
• Image plane coordinates

• Camera extrinsic parameters y

c x
• Perspective projection

Uncalibrated camera
• Pixel coordinates

• Projection matrix

MASKS © 2004 Invitation to 3D vision

 Taxonomy on Uncalibrated Reconstruction

• is known, back to calibrated case

• is unknown
n Calibration with complete scene knowledge (a rig) – estimate

n Uncalibrated reconstruction despite the lack of knowledge of

n Autocalibration (recover from uncalibrated images)

• Use partial knowledge

n Parallel lines, vanishing points, planar motion, constant intrinsic

• Ambiguities, stratification (multiple views)

MASKS © 2004 Invitation to 3D vision

 Uncalibrated Epipolar Geometry

• Epipolar constraint

• Fundamental matrix

• Equivalent forms of
MASKS © 2004 Invitation to 3D vision
 Properties of the Fundamental Matrix

• Epipolar lines

• Epipoles
Image
correspondences

MASKS © 2004 Invitation to 3D vision

 Properties of the Fundamental Matrix

• Epipolar lines

• Epipoles
Image
correspondences

MASKS © 2004 Invitation to 3D vision

 Properties of the Fundamental Matrix

A nonzero matrix is a fundamental matrix if has

a singular value decomposition (SVD) with

for some .

There is little structure in the matrix except that

MASKS © 2004 Invitation to 3D vision

 Estimating Fundamental Matrix

• Find such that the epipolar error is minimized

Pixel coordinates

• Fundamental matrix can be estimated up to scale

• Denote

• Rewrite
• Collect constraints from all points

MASKS © 2004 Invitation to 3D vision

 Two view linear algorithm – 8-point algorithm

• Solve the LLSE problem:

• Solution eigenvector associated with

smallest eigenvalue of

• Compute SVD of recovered from data

• Project onto the essential manifold:

• cannot be unambiguously decomposed into pose

and calibration

MASKS © 2004 Invitation to 3D vision

 Calibrated vs. Uncalibrated Space

MASKS © 2004 Invitation to 3D vision

 Calibrated vs. Uncalibrated Space

Distances and angles are modified by S

MASKS © 2004 Invitation to 3D vision

 Motion in the distorted space

Calibrated space Uncalibrated space

• Uncalibrated coordinates are related by

• Conjugate of the Euclidean group

MASKS © 2004 Invitation to 3D vision

 What Does Tell Us?

• can be inferred from point matches (eight-point algorithm)

• Cannot extract motion, structure and calibration from one

fundamental matrix (two views)

• allows reconstruction up to a projective transformation (as

we will see soon)

• encodes all the geometric information among two views

when no additional information is available

MASKS © 2004 Invitation to 3D vision

 Decomposing the Fundamental Matrix

• Decomposition of the fundamental matrix into a skew

symmetric matrix and a nonsingular matrix

• Decomposition of is not unique

• Unknown parameters - ambiguity

• Corresponding projection matrix

MASKS © 2004 Invitation to 3D vision

 Projective Reconstruction

• From points, extract , followed by computation

of projection matrices and structure
• Canonical decomposition

• Given projection matrices – recover structure

Projective ambiguity – non-singular 4x4 matrix H

<latexit sha1_base64="NgWE8hSOBo5S/Gqp8RpRWjMrVCg=">AAAB6HicbVBNS8NAEJ34WetX1aOXxSJ4KomIeix66bEF+wFtKJvtpF272YTdjVBCf4EXD4p49Sd589+4bXPQ1gcDj/dmmJkXJIJr47rfztr6xubWdmGnuLu3f3BYOjpu6ThVDJssFrHqBFSj4BKbhhuBnUQhjQKB7WB8P/PbT6g0j+WDmSToR3QoecgZNVZq1Pqlsltx5yCrxMtJGXLU+6Wv3iBmaYTSMEG17npuYvyMKsOZwGmxl2pMKBvTIXYtlTRC7WfzQ6fk3CoDEsbKljRkrv6eyGik9SQKbGdEzUgvezPxP6+bmvDWz7hMUoOSLRaFqSAmJrOvyYArZEZMLKFMcXsrYSOqKDM2m6INwVt+eZW0LivedcVtXJWrd3kcBTiFM7gAD26gCjWoQxMYIDzDK7w5j86L8+58LFrXnHzmBP7A+fwBnx+M0Q==</latexit>

0 1
<latexit sha1_base64="a3yhpb6Cp4IIeUOVR8xSwjlk3j4=">AAACInicbVDLSsNAFJ34rPUVdelmsIhuLImIj4VQdNNlBfuAJobJdNIOnUzCzEQsId/ixl9x40JRV4If46TNQlsvDHM451zuvcePGZXKsr6MufmFxaXl0kp5dW19Y9Pc2m7JKBGYNHHEItHxkSSMctJUVDHSiQVBoc9I2x9e53r7nghJI36rRjFxQ9TnNKAYKU155oXDtLmHPAqdEKmBH8AHjx7AS5g6DZp5Kc1g/S49svWnqcLSyTzimRWrao0LzgK7ABVQVMMzP5xehJOQcIUZkrJrW7FyUyQUxYxkZSeRJEZ4iPqkqyFHIZFuOj4xg/ua6cEgEvpxBcfs744UhVKOQl878x3ltJaT/2ndRAXnbkp5nCjC8WRQkDCoIpjnBXtUEKzYSAOEBdW7QjxAAmGlUy3rEOzpk2dB67hqn1atm5NK7aqIowR2wR44BDY4AzVQBw3QBBg8gmfwCt6MJ+PFeDc+J9Y5o+jZAX/K+P4BL8Oi1A==</latexit>

i i = ⇧i H
x HXe
0
<latexit sha1_base64="Xqwh/kg1CtTMb8HiFy+HLZAMWy0=">AAACGnicbVDLSsNAFJ34rPUVdelmsIiuSiKiboSiG5cV7AOaECbTSTt0MhlmJmIJ+Q43/oobF4q4Ezf+jZM2C209MHA451zm3hMKRpV2nG9rYXFpeWW1slZd39jc2rZ3dtsqSSUmLZywRHZDpAijnLQ01Yx0hSQoDhnphKPrwu/cE6lowu/0WBA/RgNOI4qRNlJgux4z4T4KKPRipIdhBB8CegQvYeY1aR5kVOSGllY3D0Rg15y6MwGcJ25JaqBEM7A/vX6C05hwjRlSquc6QvsZkppiRvKqlyoiEB6hAekZylFMlJ9NTsvhoVH6MEqkeVzDifp7IkOxUuM4NMliRzXrFeJ/Xi/V0YWfUS5STTiefhSlDOoEFj3BPpUEazY2BGFJza4QD5FEWJs2q6YEd/bkedI+qbtndef2tNa4KuuogH1wAI6BC85BA9yAJmgBDB7BM3gFb9aT9WK9Wx/T6IJVzuyBP7C+fgBlS6B7</latexit>

i i = ⇧ip Xp
x
Both ⇧i and are consistent with the epipolar geometry – give the
<latexit sha1_base64="jL9GNE772mJLKVbfouHuAm78qJA=">AAAB7HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE1GPRi8cKpi20oWy2m3bpZhN2J0IJ/Q1ePCji1R/kzX/jts1BWx8MPN6bYWZemEph0HW/ndLa+sbmVnm7srO7t39QPTxqmSTTjPsskYnuhNRwKRT3UaDknVRzGoeSt8Px3cxvP3FtRKIecZLyIKZDJSLBKFrJ7zVFX/SrNbfuzkFWiVeQGhRo9qtfvUHCspgrZJIa0/XcFIOcahRM8mmllxmeUjamQ961VNGYmyCfHzslZ1YZkCjRthSSufp7IqexMZM4tJ0xxZFZ9mbif143w+gmyIVKM+SKLRZFmSSYkNnnZCA0ZygnllCmhb2VsBHVlKHNp2JD8JZfXiWti7p3VXcfLmuN2yKOMpzAKZyDB9fQgHtogg8MBDzDK7w5ynlx3p2PRWvJKWaO4Q+czx+cTY6O</latexit>

same fundamental matrix

MASKS © 2004 Invitation to 3D vision

 Projective Reconstruction

• Given projection matrices recover projective structure

• This is a linear problem and can be solve using linear least-squares

• Projective reconstruction – projective camera matrices and

projective structure
Xp = HXe
<latexit sha1_base64="ww4DxjvsdXE7yK2enpyqakTh99U=">AAACBHicbVDLSsNAFL2pr1pfUZfdDBbBVUlE1I1QdNNlBfuANpTJdNIOnUzCzEQooQs3/oobF4q49SPc+TdO2oDaemDgzDn3cu89fsyZ0o7zZRVWVtfWN4qbpa3tnd09e/+gpaJEEtokEY9kx8eKciZoUzPNaSeWFIc+p21/fJP57XsqFYvEnZ7E1AvxULCAEayN1LfLvRDrkR+gTj9GV6iOfv60b1ecqjMDWiZuTiqQo9G3P3uDiCQhFZpwrFTXdWLtpVhqRjidlnqJojEmYzykXUMFDqny0tkRU3RslAEKImme0Gim/u5IcajUJPRNZbaiWvQy8T+vm+jg0kuZiBNNBZkPChKOdISyRNCASUo0nxiCiWRmV0RGWGKiTW4lE4K7ePIyaZ1W3fOqc3tWqV3ncRShDEdwAi5cQA3q0IAmEHiAJ3iBV+vRerberPd5acHKew7hD6yPb47ilsI=</latexit>

Euclidean Structure Projective Structure

MASKS © 2004 Invitation to 3D vision

 Summary of Two-View Geometry

MASKS © 2004 Invitation to 3D vision

 Euclidean vs Projective reconstruction

• Euclidean reconstruction – true metric properties of objects

lenghts (distances), angles, parallelism are preserved
• Unchanged under rigid body transformations
• => Euclidean Geometry – properties of rigid bodies under
rigid body transformations, similarity transformation

• Projective reconstruction – lengths, angles, parallelism are NOT

preserved – we get distorted images of objects – their distorted 3D
counterparts --> 3D projective reconstruction
• => Projective Geometry

MASKS © 2004 Invitation to 3D vision

 Homogeneous Coordinates (RBM)

3-D coordinates are related by:

Homogeneous coordinates:

Homogeneous coordinates are related by:

MASKS © 2004 Invitation to 3D vision

 Homogenous and Projective Coordinates

• Homogenous coordinates in 3D before – attach 1 as the last

coordinate – render the transformation as linear transformation
• Projective coordinates – all points are equivalent up to a scale

2D- projective plane 3D- projective space

• Each point on the plane is represented by a ray in projective space

• Ideal points – last coordinate is 0 – ray parallel to the image plane
• points at infinity – never intersects the image plane

MASKS © 2004 Invitation to 3D vision

 Vanishing points – points at infinity

Representation of a 3-D line – in homogeneous coordinates

When l -> 1 - vanishing points – last coordinate -> 0

Similarly in the image plane

MASKS © 2004 Invitation to 3D vision

 Ambiguities in Image Formation

• Potential Ambiguities

• Ambiguity in K (K can be recovered uniquely – Cholesky or QR)

• Structure of the motion parameters

• Just an arbitrary choice of reference frame

MASKS © 2004 Invitation to 3D vision

 Ambiguities in Image Formation

Structure of the (uncalibrated) projection matrix

• For any invertible 4 x 4 matrix

• In the uncalibrated case we cannot distinguish between camera

imaging world from camera imaging distorted world

• In general, is of the following form

• In order to preserve the choice of the first reference frame we can

restrict some DOF of
MASKS © 2004 Invitation to 3D vision
 Structure of the Projective Ambiguity

• 1st frame as reference

• Choose the projective reference frame

then ambiguity is

• can be further decomposed as

MASKS © 2004 Invitation to 3D vision

 Stratified (Euclidean) Reconstruction

• General ambiguity – while preserving choice of first reference

frame

• Decomposing the ambiguity into affine and projective one

• Note the different effect on the 4-th homogeneous coordinate

MASKS © 2004 Invitation to 3D vision

 Affine Upgrade

• Upgrade projective structure to an affine structure

• Exploit partial scene knowledge

n Vanishing points, no skew, known principal point

• Special motions
n Pure rotation, pure translation, planar motion, rectilinear motion

• Constant camera parameters (multi-view)

MASKS © 2004 Invitation to 3D vision

 Affine Upgrade using Vanishing Points

How to compute

Maps the points

To points with affine coordinates

Vanishing points – last homogeneous affine coordinate is 0

MASKS © 2004 Invitation to 3D vision

 Affine Upgrade

Need at least three vanishing points

3 equations, 4 unknowns (-1 scale)

Solve for

Set up and update the projective

structure

MASKS © 2004 Invitation to 3D vision

 Euclidean Upgrade
• We need to estimate remaining affine ambiguity

Alternatives:
• In the case of special motions (e.g. pure rotation) – no
projective ambiguity – cannot do projective reconstruction

• Estimate directly (special case of rotating camera –

follows)
• Multi-view case – estimate projective and affine ambiguity
together
• Use additional constraints of the scene structure (next)
• Autocalibration (Kruppa equations)

MASKS © 2004 Invitation to 3D vision

 Direct Stratification from Multiple Views

From the recovered projective projection matrix

we obtain the absolute quadric contraints

Partial knowledge in (e.g. zero skew, square pixel) renders

the above constraints linear and easier to solve.

The projection matrices can be recovered from the multiple-view

rank method to be introduced later.

MASKS © 2004 Invitation to 3D vision

 Geometric Stratification

MASKS © 2004 Invitation to 3D vision

 Overview of Calibration Methods

MASKS © 2004 Invitation to 3D vision

 Summary of (Auto)calibration Methods

MASKS © 2004 Invitation to 3D vision

 Calibration with a Rig

Use the fact that both 3-D and 2-D coordinates of feature
points on a pre-fabricated object (e.g., a cube) are known.

MASKS © 2004 Invitation to 3D vision

 Calibration with a Rig

• Given 3-D coordinates on known object

• Eliminate scale, two linear constraints per point:

• Recover projection matrix

• Factor the into and using QR decomposition

• Solve for translation

MASKS © 2004 Invitation to 3D vision

 Calibration with a Planar Rig

Use the fact that both 3-D and 2-D coordinates of feature
points on a pre-fabricated plane are known.

MASKS © 2004 Invitation to 3D vision

 Calibration with a Planar Rig

• Special world frame on the plane

• Homography from the plane to the image

• Two linear constraints on the calibration per image

MASKS © 2004 Invitation to 3D vision

Real-World Calibration (light-field/stage cameras)

A New Calibration Technique for Multi-Camera Systems of Limited Overlapping Field-of-Views,

Xing, Yu, and Ma, IROS, 2017
Real-World Calibration (with Lens Distortion)

Camera Calibration with Lens Distortion from Low-rank Textures,

Zhang, Matsushita, and Ma, in CVPR 2011
Real-World Calibration (with Lens Distortion)

Classic approach Low-rank method

Camera Calibration with Lens Distortion from Low-rank Textures,

Zhang, Matsushita, and Ma, in CVPR 2011
 Calibration with Scene Structure: vanishing points

• Vanishing points – intersections of the parallel lines

• Vanishing points of three orthogonal directions

• Orthogonal directions – inner product is zero

• Provide directly constraints on matrix

• S – has 5 degrees of freedom, 3 vanishing points gives

three linear constraints (need additional assumption on K)

• Assume zero skew and aspect ratio = 1

MASKS © 2004 Invitation to 3D vision

 Calibration with Motions – Pure Rotation

• Calibrated two views related by rotation only

• Mapping to a reference view – rotation can be estimated

• Mapping to a cylindrical surface

MASKS © 2004 Invitation to 3D vision

 Calibration with Motions: Pure Rotation

• Uncalibrated two views related by a pure rotation:

• Conjugate rotation can be estimated

• Given C, we have three linear constraints:

• Given two rotations around linearly independent

axes – S, K can be estimated using linear techniques
• Applications – image mosaics

MASKS © 2004 Invitation to 3D vision

 Calibration with Motions: General Motions

The fundamental matrix

satisfies the Kruppa’s equations

If the fundamental matrix is known up to scale

This give two nonlinear constraints on

Solution to Kruppa’s equations can be sensitive to noises.

MASKS © 2004 Invitation to 3D vision

 Calibration with Motions: Special Motions

Under special motions,

The scale can be determined, hence the

Kruppa’s equations become linear in .

Each Kruppa equation gives two linear constraints on

MASKS © 2004 Invitation to 3D vision

 Calibration with Motions: Special Motions

Section 6.6: Dinner with Kruppa of the book MASK

MASKS © 2004 Invitation to 3D vision