Vision for Multiple or Moving Cameras

Overview

VMMC – 2022 1 Unit I: The projected reality

 Vision for Multiple or Moving Cameras

UNIT I - THE PROJECTED REALITY

UNIT II - DETECTION, DESCRIPTION AND MATCHING OF REFERENCE POINTS
UNIT III - CALIBRATION OF CAMERA NETWORKS
http://people.inf.ethz.ch/moswald/publications/resources/Oswald-MR-PhD-Thesis.jpg

VMMC – 2022 2 Unit I: The projected reality

 Camera network for 3D reconstruction

https://software.intel.com/en-us/articles/photogrammetry-and-volumetric-capture

VMMC – 2022 3 Unit I: The projected reality

 Point of view interpolation

https://youtu.be/hnP7G7ahuus

VMMC – 2022 4 Unit I: The projected reality

 Structure from motion

https://youtu.be/bLtgTbzWmls

VMMC – 2022 5 Unit I: The projected reality

 Image stitching

http://matthewalunbrown.com/autostitch/autostitch.html

VMMC – 2022 6 Unit I: The projected reality

 Depth estimation with stereo cameras

https://youtu.be/bsA6RKUUA3M

VMMC – 2022 7 Unit I: The projected reality

 VMMC objective - 3D point cloud reconstruction

Mason 2017

VMMC – 2022 8 Unit I: The projected reality

 Vision for Multiple or Moving Cameras

UNIT I: The projected reality

VMMC – 2022 9 Unit I: The projected reality

 Introduction to projection Calibration based on 3D points
measurement
Historical review
.Calibration based on single view
Projection models
homographies
Projection relationships
Correction of the lens distortion
Projective Geometry: motivation
Two-view geometry
Fundamentals of Projective
Geometry The parallax effect

The Queen of the geometries Epipolar geometry – fundamental matrix

Rectification and disparity maps
Elements and definitions
Binocular reconstruction
Basic transforms

Calibration of a single camera

The projection matrix
Computation of the internal
parameters

VMMC – 2022 10 Unit I: The projected reality

 Introduction to projection
Historical review
Projection models

• Basic camera model – Perspective projection (Pinhole):

• Model parameters: focal length, F .

• Geometrical analysis :

• 3D coordinate origin ( O ) coincident with the camera center or focal

center ( C ).
• Z axis coincident with the camera’s optical axis .
• World units = image units.
• Reflected image plane.

• Ambiguity in the projection of 3D points located in a same line of sight.

• CAHV camera model:

• It adds the possibility of including the position of the camera ( C vector),

the direction in which it points ( A unit vector) and the orientation of the
image plane ( H, V unit vectors). Calibration.
• Model parameters: extrincsic ( A, C ), intrinsic ( H, V, F ).

VMMC – 2022 11 Unit I: The projected reality

 Projection relationships

Relationship between a change in the position of a 3D point and the

corresponding position change in the image plane: mapping functions

• General case: non-restricted 3D change + perspective projection:

x = F
( r1 x + r2 y + r3 F ) Z + Tx F
( r7 x + r8 y + r9 F ) Z + Tz F
( r4 x + r5 y + r6 F ) Z + Ty F
y = F
( r7 x + r8 y + r9 F ) Z + Tz F

• Projecive mapping: planar restriction + perspective projection

• 3D change is restricted to a generic planar

surface defined by: Z = aX + bY + c a0 + a1 x + a2 y
x =
1 + c1 x + c2 y
b0 + b1 x + b2 y
y =
1 + c1 x + c2 y

VMMC – 2022 12 Unit I: The projected reality

 Projective geometry: motivation

• Euclidean geometry adequately models the 3D world, as distances and

angles (parallelism, orthogonality) are preserved by the typical 3D
transformations (rotation+translation=Euclidean transformation).

• Projective transformations are not adequately modelled by Euclidean

geometry, as distances and angles are not preserved (vanishing points): the
model leads to non-linear relationships.

• Projective geometry: linear model (i.e., simple, computationally

affordable,tracktable, etc.) of the image formation process in a conventional
camera. This has boosted the development of many practical applications:

• Recognition of planar object/images.

• Image stitching: panoramic images.
• View interpolation in the absence of occlusion.
• Reconstruction from motion in the absence of occlusion.

VMMC – 2022 13 Unit I: The projected reality

 Fundamentals of Projective Geometry

The Queen of the geometries

• Classical or synthetic geometry vs analytic geometry

• Hierarchy of geometries (Klein, 1872)

Elements and definitions

• Homogeneous coordinates
• Duality
• Ideal points or points at infinity
• Conics and points at infinity
• The projective space
• Homographies
Basic transforms
• 2D Euclidean transforms.
• 3D Euclidean transforms.
• Projection.
• Projection + 3D Euclidean transform

VMMC – 2022 14 Unit I: The projected reality

 Calibration of a single camera
The projection matrix

• Defines the relationship between a point in the 3D space, M = ( X , Y , Z ,1) , and a

pixel in the image plane, m = ( u, v,1). It results from the formalization of three
effects or transformations:

1) 3D Euclidean transformation between the world coordinate system ( O )

and the camera's coordinate system ( C ):

Xc   c   r11 r12 r13 tx   X 

Y    r t y   Y 
r22 r23  R t
MC =  c 
 Zc 


c 


 21
 r31
 0T 1  M or MC R t M
c r32 r33 tz   Z   
       
1  c  0 0 0 1 1 

2) Projection of point M C at point mC of the image plane: based on a pin-

hole model and described by projective geometry:

 
 x x  f 0 0  c
0 
mC =  y  y 
 
0
 f 0 0   c 
=  MC =  R t   M
  0 0

 1   t   0 0 
1 0  c

 c 

VMMC – 2022 15 Unit I: The projected reality

 3) 2D affine transformation between the projected image plane, ( x, y ) ,
and the final image plane ( u, v ) : change of coordinates and
coordinates measured in pixels that may not be square nor orthogonal.

u  u   ku −ku cos  u0   x 
m =  v  v 
 
0
 kv sin  v0    y  = A 0  mC
1   t   0 0 1   1 

• In conclusion, the objective relationship in homogeneous coordinates results:

m 
A 0  mC

mC 0  MC m A0  0  R t   M = A  R t   M = M  m M
MC  R t   M 


, where A groups the internal parameters and  R t  the external parameters:

 f  ku − f  ku cos  u0 0
A =  0 f  kv sin  v0 0 
 0  p11 p12 p13 p14 
0 1 0 
A   R t  =  p21 p24 
5 p22 p23
 p31 p32 p33 p34 
 R t 11
R t =  T 
0 1
6

VMMC – 2022 16 Unit I: The projected reality

 Computation of the internal parameters
• The choice of the matrix A has been presented derived from an analysis
linked to practical issues which derives in 5 parameters:
 a c u0 0  a c u0 
A =  0 b v0 0  , although it is usually written A =  0 b v0 
 
 0 0 1 0   0 0 1 

• Depending on the type of problem, the matrix can present the 5 parameters
(general case), 4 (+ =  2 ), 3 (+square pixels) or 1 (+known principal point).

• Matrix A is enough to verify that the projection matrix can be decomposed

according to: A   R t  . This is verified by any superior triangular matrix (the
given one or any other).

• For the most general case of matrix A , taking into account that the projection
matrix can be decomposed into =  P p , where, if the previous point is
fulfilled, P = A  R , the following relation results:

P  PT = A  R  ( A  R ) = A  R  RT  AT = A  AT
T

, where, in addition, matrix A  AT = K is known as Kruppa's matrix, from which

the A matrix can be directly obtained.

VMMC – 2022 17 Unit I: The projected reality

 Calibration based on 3D points measurement

• The calibration methods require knowing some type of 3D information of the

scene captured.
• They can be classified as: linear / non-linear, explicit / implicit.

• Method based on 3D coordinates - Faugeras (1986):

• It is linear and implicit.  0 u0 0

• It assumes a matrix of 4 intrinsic parameters : A =  0  v0 0 
 0 0 1 0 
• It requires the knowledge of the 3D coordinates (referring to the 3D
coordinate system that you want) of at least 6 M i points projected in the
corresponding 6 m i points of a same image.
• First solves a linear system to obtain the 12 elements of the projection
matrix , then solving the problem of proportionality by imposing unity
norm on the rotation vector around the Z axis.
• Finally, estimates the parameters of the camera obtaining the explicit
expression of each one according to the values of .
• Quick method that in turn requires measurements in the 3D scene in a
two-plane pattern. Imprecise due to 3D measurements.

VMMC – 2022 18 Unit I: The projected reality

 Calibration based on single view homographies

• Projection between a plane of the world and the image plane: homography.
• Affine interpretation of a projection: real points and points at infinity.
• The absolute conic and the matrix of intrinsic parameters.

• Zhang method (1999)

• It is implicit and non-linear.

• It assumes the general matrix of intrinsic parameters A .
• First of all, it obtains the matrix of the IAC, B , from the existing
relationship between this matrix and several homographies between a
same pattern with different orientations and the image plane.
• From the matrix B the five internal parameters in A are obtained, as
has been seen previously.
• Once known A , from one of the homographies obtained it is possible,
although not always very useful, to estimate the external parameters.

VMMC – 2022 19 Unit I: The projected reality

 Correction of the lens distortion

• The techniques seen are based on the pin-hole projection model, which
establishes collinearity between a point, its image and the optical center. The
distortion effect of the lenses, mainly of radial type, introduces non-linearities.

• In order to calibrate (and, in general to apply any of the techniques based on

projective geometry) it is necessary to correct this distortion previously, or
consider it within the same calibration process or, in case it is not appreciable,
correct it in a postprocess.

• The distortion occurs prior to the affine transformation in the image plane (given
by the internal parameter matrix A ).

• The correction of radial distortion is usually based on:

• An approximate polynomial model of the variation of the distortion with the

distance from the point to the center of distortion.
• Locating the model parameters by adjusting real points to points without
distortion.

VMMC – 2022 20 Unit I: The projected reality

 Two-view geometry

Study of the relationship between two projections of the same scene. The main
objective is the location of points of the 3D scene from their 2D projections.

The parallax effect

• Displacement effect derived from the projection of a scene from two points of
view with a different optical center: allows to estimate 3D info, makes it difficult to
establish correspondences between single points of both projections, and does
not result in a biunivocal global transform between both image planes.

• The correspondence between two images of the same 3D plane is given by a

homography (planar homography) that depends on the projection matrices and
on the observed plane. Particular case of the plane at infinity.

• In the absence of parallax (rotating camera) it is not possible to recover the

depth of a projected point in two views. However, in this case the
correspondence between image planes is given by a single homography
(independently of the 3D structure of the scene).

VMMC – 2022 21 Unit I: The projected reality

 Epipolar geometry – fundamental matrix

• Epipolar geometry is the basic restriction that appears when relating the
projections of a scene captured from two points of view. This happens in two
situations:

• Two cameras that simultaneously capture a scene: stereovision.

• A mobile camera that captures the scene in two different instants and
positions: structure from motion.

• Search for homologous points in both projections: geometric deduction of the

fundamental matrix and deduction of the expression of the epipolar constraint
between projections of the same scene.

• The relationship between planar homographies and the fundamental matrix

allows obtaining a closed algebraic deduction of the fundamental matrix if the
cameras are calibrated, which facilitates the location of corresponding points in
both image planes.

VMMC – 2022 22 Unit I: The projected reality

 Rectification and disparity maps

• The relations between homologous points are simplified remarkably if the

cameras are parallel: the image planes being part of a same plane and with one
of their respective axes aligned.

• If the previous situation is not verified, it is possible to apply to each image

plane a homography so that the transformed planes do verify it. This technique is
called rectification.

• The disparity is the horizontal difference of coordinates between two

homologous pixels, which in the described situation of horizontal epipolar lines is
precisely the projective effect of the parallax.

• Applications:

• Detection of obstacles with respect to a reference plane.

• Obtaining different points of view of the same scene.

VMMC – 2022 23 Unit I: The projected reality

 Bonicular reconstruction

• Study of the information of the world ( M i ) and of the cameras ( ,  ) that it is

possible to obtain according to which is the information from which we start :

• If only pairs of homologous points are known ( mi , mi ) one can aspire to
a projective reconstruction, that is, information of the world that is invariant
to projective transformations (what is in front of or behind a plane, and little
more ...).

• If H  is also known, it is possible to aspire to an affine reconstruction,

that is, information invariant to this type of transformation (e.g., order of
depths, parallelism, ratio of distances between 3 aligned points).

• If the IAC or something equivalent (internal parameters) is also known,

we can aspire to a Euclidean reconstruction, that is, information invariant to
this type of transformation (e.g., angles, relative distances). And vice versa,
with 3D information of the world it is possible to obtain the internal
parameters (autocalibration).

VMMC – 2022 24 Unit I: The projected reality

 A

A x

−x X X
= x = −F P = X =X Z
T
Y
F Z  Z  p = x =  x  = −F  X 
−y Y  y Z Y 
Y     C =  0 0 0
T
= y = −F
F Z Z

VMMC – 2022 25 Unit I: The projected reality

 A
y

A x

x X X
= x=F
F Z  Z  p = x =  x = F  X 
 y Z Y 
y Y
= y=F
Y    
F Z Z

VMMC – 2022 26 Unit I: The projected reality

 T
C = Cx Cz  P = X =X Z
T
Cy Y

 x F  X − Cx 
COp=   = Y − C 
 y  Z − Cz  y 

VMMC – 2022 27 Unit I: The projected reality

 T
C = Cx Cz  P = X =X Z
T
Cy Y

 x F  X − Cx 
A   0 0 1  p =   = T
T
 
 y  A  ( P − C)  Y − Cy 

VMMC – 2022 28 Unit I: The projected reality

 T
C = Cx Cz  P = X =X Z
T
Cy Y

A   0 0 1
T

 x F  HT  ( P − C ) 
H  1 0 0  p =   = T
T
 T 
 y  A  ( P − C)  V  ( P − C) 
V   0 1 0
T

VMMC – 2022 29 Unit I: The projected reality

 Mapping function:

Z  Z Z 
 =
( r1 x + r2 y + r3 F ) Z + Tx F
3D scene-point change:  x F
 F x = r1 F x + r2 F y + r3 Z + Tx  ( r7 x + r8 y + r9 F ) Z + Tz F
 X   r1 r2 r3   X  Tx    
 Y   = r r5 r6   Y  + Ty 
Z Z Z
  y = r4 x + r5 y + r6 Z + Ty  y = F ( r4 x + r5 y + r6 F ) Z + Ty F
  4 
 Z    r7 r8 r9   Z  Tz  F F F
 ( r7 x + r8 y + r9 F ) Z + Tz F
 Z Z
Perspective projection:  Z  = r7 F x + r8 F y + r9 Z + Tz

 x F  X   x  F  X  
 y = Z  Y  ,  y = Z   Y  
       

Observations:
• Invariance against change in a T  ⎯⎯
→ k T 

x ⎯⎯
→ x
same line of sight. Z ⎯⎯
→ kZ y ⎯⎯
→ y

k
Tz ⎯⎯
→ kTz  Z ⎯⎯
→ Z + kTz = kZ  Z = Tz
k −1

• The 2D mapping function depends on Z, i.e., the estimation of this 2D transformation does not
allow to obtain an unrestricted 3D transformation.

VMMC – 2022 30 Unit I: The projected reality

 Projective mapping:
3D camera-point change:
 X   r1
 ( r1 x + r2 y + r3 F ) Z + Tx F  a0 + a1 x + a2 y
 x = 1 + c x + c y
r3   X  Tx 
 x = F
r2
 Y   = r r5 r6   Y  + Ty    ( r7 x + r8 y + r9 F ) Z + Tz F 

1 2
  4 
 Z    r7r8 r9   Z  Tz   y = F ( r4 x + r5 y + r6 F ) Z + Ty F  y = b0 + b1 x + b2 y
  1 + c1 x + c2 y
Perspective projection:  ( r7 x + r8 y + r9 F ) Z + Tz F 

 x  F  X   x  F  X   
 y  = Z  Y  ,  y = Z   Y   
         Fc
 Z =
Planar Surface:  F − ax − by
Z = aX + bY + c 


  r3c + Tx   r1c − Tx a   r2c − Txb 

  F + x+ y
 x = 1( r Fc − T x Fa ) x + ( r2 Fc − Tx Fb ) y + ( r3 c + Tx ) F 2

= 9
r c + Tz   9r c + Tz   9r c + Tz 
 ( r7 c − Tz a ) x + ( r8c − Tzb ) y + ( r9c + Tz ) F  r c − Tz a   r8c − Tz b 
 1+  7 x+ y
  F ( r9 c + T )
z   F ( r9 c + T )
z 

  r6 c + Ty   r4c − Ty a   r5c − Tyb 
+ +

 y =
( r4 Fc − Ty Fa ) x + ( r5 Fc − Ty Fb ) y + ( r6c + Ty ) F 2

=



r9 c + Tz 
F 
 r9c + Tz 
 x 
 r9c + Tz 
y

 ( r7 c − Tz a ) x + ( r8c − Tzb ) y + ( r9c + Tz ) F  r c − Tz a   r8c − Tz b 

 1+  7 x+ y
  F ( r9 c + T )
z   F ( r9 c + T )
z 

VMMC – 2022 31 Unit I: The projected reality

 The target objects are planar:

Vivino Shot & Find Smartify iTrack Wildlife

The target objects are not planar:

PlantNet iNaturalist Google Lens

VMMC – 2022 32 Unit I: The projected reality

 Classical geometry:

In charge of studying constructions in a real affine plane using ruler and compass, that is, through the synthetic
method, introducing the five postulates of Euclid collected in 'The Elements' (300-BC). Subsequently, and given
that any construction is a repetition of these five basic operations on the same elements (lines and points), they
began to be treated as operations with algebraic symbols (analytic method) and these two approaches began
to diverge.

Algebraic derivation of the geometries:

Felix Klein (1872) gave a formal definition of what is meant by Geometry : he proposed to classify geometric
properties according to their invariance with respect to different groups of transformations.

Klein introduces in the Geometry the algebraic concept of group: associative with respect to an operation or
transformation, with neutral element and with symmetrical element. Each geometry is the study of certain
properties that do not change (invariant) when a type of transformations are applied on a set of elements, which
must have a group structure. Observe that instead of defining a structure (elements + transformation) and to
see what is invariant to that transformation (translation, rotation, similarity, etc.), what is done is to define the
transformation and see over which elements (real points, points at infinity, absolute conic) has to be defined to
make certain properties invariant.

This contribution allows us to classify the geometries, understanding which is a "subgeometry" of which (without
more than identifying a subgroup of a given group of transformations), and confirms that the synthetic and
algebraic methods do not generate different geometries, but actually study the same geometry in each case.

VMMC – 2022 33 Unit I: The projected reality

 Hierarchy of geometries

Properties that are preserved (invariants)

incidence, cross
center of mass

Straightness,
Parallelism,

Connection
Convexity
Geometry Considered transform

Direction

Distance

Angles

ratio
Translation Y Y Y Y Y Y Y

Euclídean Isometry Rotation Y Y Y Y Y Y Y

Similarity
Homography
Affinity

Reflections NO Y Y Y Y Y Y
Homeomorphism
Included by

Homothetic
Conformal Y* NO Y Y Y Y Y
transform

Affine Parallel projection Y* NO NO Y Y Y Y

Projective Perspective projection Y* NO NO NO Y Y Y

Topology Topological transform Y* NO NO NO NO NO Y

VMMC – 2022 34 Unit I: The projected reality

 Homogeneous vectors:

Given the equation of a line in the plane, ax + by + c = 0 , the ( a, b, c ) vector represents it (not its
direction). Any other proportional vector, (  a, b,  c ) , also represents it. This class of vectors is known
to be homogeneous. The homogeneous coordinates of a line, r , in the plane are those of any vector of
this class, i.e., are proportional to ( a, b, c ) : r ( a, b, c )

Homogeneous coordinates of a point in the projective plane (Möbius, 1837 ):

A point ( x, y ) belongs to a line if it verifies: ( x, y,1) ( a, b, c ) = 0   x y 1  a b c  = 0

T

Any tuple ( X , Y , T ) proportional to ( x, y,1) represents a point ( x, y ) of that line in 2 :

This family of tuples, which can be

seen in 3 as 3D points aligned in
the same direction or ray including the
origin, are declared to be
homogeneous.

The projective point (point of the

projective plane) ( x, y,1)  P2 has
homogeneous coordinates:
( X , Y , T ) ( x, y,1)

X Y
( X , Y , T ) ( X , Y , T ) ( x, y,1)  P2 = 3
− 0 , ( x, y )  2
, x= ,y=
T T

VMMC – 2022 35 Unit I: The projected reality

 In conclusion, every point x = ( x, y )  2
has an homogeneous representation:

X ( X , Y , T )  P2
, so that to every point X  P2 corresponds a single point x = ( X T , Y T )  , and to this point x
2

corresponds a point X  P2 which represents a ray with infinite number of points X = (  x,  y,  )  . 3

Example: equation of a 2
line in homogeneous coordinates, instead of affine coordinates:
X Y
ax + by + c = 0  a + b + c = 0  aX + bY + cT = 0   X Y T   a b c  = 0
T

T T

T =1 2

T =1

Image obtained from “en.wikipedia.org”

VMMC – 2022 36 Unit I: The projected reality

 The equaton of a line in homogemeous coordinates can be described by the dot product of two vectors, one
representing points (in homogeneous coordinates) and the other the lines which contain them (idem):

X Y T   a b c = 0
T

If we fix the  a b c  vector (a line in 2

) and vary the  X Y T  vector, we obtain points
belonging to the line.

If we fix the  X Y T  vector (a point in P 2 ) and vary the a b c  vector, we obtain lines that
contain the point.

In conclusión, everything you can demonstrate for the X , Y , T coordinates (i.e., starting from points), is
equally aplicable to the a, b, c coordinates (i.e., starting from lines).

Example: Two points X1 , X2 determine a line:

 X Y T
aX 1 + bY1 + cT1 = 0 
  a b c X1  X2 = det  X 1 Y1 T1  =  X1  XT2
aX 2 + bY2 + cT2 = 0
 X 2 Y2 T2 
By duality, two lines r1 , r2 determine a point:
 b1c2 − c1b2   0 −c1 b1   a2 
a1 X + b1Y + c1T = 0 
  X Y T r1  r2 = c1a2 − a1c2  =  c1 0 −a1    b2 
a2 X + b2Y + c2T = 0 
 a1b2 − b1a2   −b1 a1 0   c2 
r1  r2T

VMMC – 2022 37 Unit I: The projected reality

 What does the  X 0 tuple represent? And the x = ( X T , Y T ) = ( 0,0)  point ?
2
Y

If two parallel lines are expressed in affine coordinates, there is no point that verifies both equations.
However, if they are expressed in homogeneous coordinates, there is a solution:

X Y T
aX + bY + cT = 0 
  X Y T r1  r2 = det  a b c  b −a 0
aX + bY + cT = 0
 a b c

, which is the direction vector of both lines, or of any line parallel to them. This point (in homogeneous
coordinates), which represents a ray or direction in T = 0 is the point where all these parallel lines cut,
so it is called the infinity point of the line.

The set of infinity points define in

the projective plane the line at
infinity ( T = 0 ), which in 3 is
represented as a plane..

The set of real points that are at

an infinite distance from the
projection plane are all projected
onto x = ( 0,0 ) .

VMMC – 2022 38 Unit I: The projected reality

 Conics and points at infinity:

A conic (hyperbola, ellipse and parabola) is a curve described by a second-degree equation:

• In affine coordinates: ax2 + bxy + cy 2 + dx + ey + f = 0

• In homogeneous coordinates: aX 2 + bXY + cY 2 + dXT + eYT + fT 2 = 0

, which can be written in matrix form as: XT CX = 0 , for X being a column vector and with the conic
coefficient matrix C given by:
 a b 2 d 2
C =  b 2 c e 2 
 d 2 e 2 f 

Note that C is symmetric and, as it is an homogeneous representation of the conic (any proportional
matrix fulfils the conic equation), it only presents five degrees of freedom.

In 2 , two conics that cut, may cut in two or four points, except for circles. In P 2 , all conics cut in four
points (circles are not special conics as parallel ar not especial lines): circles might cut in two real points,
but always also cut in two complex points located at infinity: the circular points.

In 3 , two quadric surfaces that intersect, intersect in a fourth-degree curve, except for spheres. In P 3
, spheres always intersect the plane at infinity in a complex conic: the absolute conic.

This is to remark that, the same as parallels, circles are not special elements in Projective Geometry.

VMMC – 2022 39 Unit I: The projected reality

 Projective space:

It is the space of the homogeneous coordinates.

Projective space P n : space of dimension n formed by all the elements of the Euclidean space n+1
except the null element Pn = n +1 − 0 , in which given two points X, Y  Pn with components
X = ( X1 ,..., X n +1 ) , Y = (Y1 ,..., Yn +1 ) , it is verified that X = Y    0 / X = Y

Affine coordinates vs homogeneous coordinates: ( x1 ,..., xn ) → ( x1 ,..., xn ,1)

 X1 X 
 ,..., n   ( X1 ,..., X n , X n +1 )
 X n +1 X n +1 

Ideal points or points at infinity: X n +1 = 0 ( a point in P , a line in P , a plane in P , etc.)

1 2 3

The projective line P1 The projective plane P 2

VMMC – 2022 40 Unit I: The projected reality

 Homographies:

A homography is a bijective transformation of the projective space, of the form X = H  X :

 X   X   a11 a12 a13   X 

 X   X   a11 a12   X  Y  → Y   a
T  → T   a  a23    Y 
a22   T       21 a22
     21  T   T    a31 a32 a33   T 

Homography between projective lines = Projectivity Homography between projective planes

P1 P2

H Y
X
X

H
Y
X
X

VMMC – 2022 41 Unit I: The projected reality

 Two proportional matrices define the same homography since they yield homogeneous vectors.

In an affine space this transformation would not be bijective, due to the special treatment of the infinity.
Taking as an example a homography in P 2 :

• The points that verify the relation  a31 a32 a33    X Y T  = 0 yield tuples of form:
T

 X  Y  0 , that is, they go to infinity.

• Analogously, the points of the line at infinity  X Y 0 , are transformed into a line with
coefficients ( a11 , a21 , a31 )  ( a12 , a22 , a32 )

VMMC – 2022 42 Unit I: The projected reality

 2D Euclidean transform:
y'

y x 
ty

tx x'

 x '  cos ( ) − sin ( )  x  t x   x

 y ' =  sin      + t  = R    + t
   ( ) cos ( )   y   y 
In affine coordinates:
 y

    cos ( ) − sin ( ) t x   
  R t   =  sin   
In homogeneous coordinates:  0T 1    ( ) cos ( ) t 

   y 
     0
    0 1   

 x '  
, or, making the proportionality factor    R t  
  y ' =  T    
explicit:
 1  
0 1
 

VMMC – 2022 43 Unit I: The projected reality

 Z
3D Euclidean transform : Z' z

y
t

Y'

x

X' X

 x '  x
 y ' = R   y  + t
In affine coordinates:     R  RT = I
 z '   z 

 x '  x    
 y ' t   y    t   
In homogeneous coordinates :   R , or   R 
 0T  
 z '  1  z     0T
 1  
       
1 1     

VMMC – 2022 44 Unit I: The projected reality

 Projections:

In affine coordinates:
X Y
x= f ,y= f
Z Z
In homogeneous coordinates :

    X 
x  f 0 0 0     Y  x   fX 
y  0 0 0       =   yields y  =  fY   x = f X , y = f Y
   f , which for    
    Z  Z Z
 t   0 0 1 0         t   Z 
    1

VMMC – 2022 45 Unit I: The projected reality

 Projection + 3D Euclidean transform

In affine coordinates, see introduction…

In homogeneous coordinates :

 
x   p11 p12 p13 p14   
y  p p22 p23 p24    
   21  
 t   p31 p32 p33 p34   
 

VMMC – 2022 46 Unit I: The projected reality

 Scene projection onto the camera plane: the
projection matrix

m
c

m M   m = M

r r12 r13 tx   
u  u   f  ku − f  ku cos  0  11
t y   
u0
m =  v  v   0 0   21
r r22 r23 
   f  kv sin  v0 
r r32 r33 tz   
 1   t   0 0 1 0  31   
0 0 0 1  

VMMC – 2022 47 Unit I: The projected reality

 Estimation of the projection matrix (Faugueras 1986):

For a pair of corresponding points, m = ( u, v ) and M = ( X , Y , Z ,1) , the following linear relationships
are established from the projection matrix: :
X 
u   p11 p12 p13 p14     UT  M 
Y  T  uW  M − U  M = 0
T T
   v  =  p21 p22 p23 
p24    = V M    T
Z   T  vW  M − VT  M = 0
 1   p31 p32 p33 p34     W  M 
1
, where UT , VT , WT are the three rows of . If n  6 pairs of points mi = ( ui , vi ) , Mi are used, a
linear system of 2n equations is obtained, which can be put into form L  x = 0 , where L is a 2n12
matrix and xT = UT VT WT  is the row vector of unknowns formed by the 12 coefficients of the
projection matrix. The solution, generally not exact, can be calculated as the vector corresponding to the
smallest singular value of the SVD (Single Value Decomposition) of L .

Another alternative is to apply the least squares method directly to the expression:

(u WT  Mi − UT  Mi ) + ( vni WT  Mi − VT  Mi ) 
2 2
ni
i

The result, in both cases, are the 12 parameters of a matrix proportional to the projection matrix. To
resolve this ambiguity the result is normalized taking into account that:

 p31 p32 p33  =  r31 r32 r33   p31 p32 p33 = p31
2
+ p32
2
+ p33
2
=1

VMMC – 2022 48 Unit I: The projected reality

 Estimation of the projection matrix (Faugueras 1986) :

Finally, the internal and external parameters are calculated directly from those obtained:

Parameters obtained Parameters desired

p1T p14   r1T + u0r3T  t x + u0 t z 
   
=  P p  = pT2 p24  = A   R t  =   r2T + v0r3T  t y + v0t z 
pT3 p34   
  r3T tz 

Example of templates used to apply this method :

Image obtained from “The Geometry of Multiple Images”, Faugeras & Luong, 2001

VMMC – 2022 49 Unit I: The projected reality

 Projection between a plane of the world and the image plane:

Given a plane  , if the axes of the world coordinate system are taken as shown in the figure, it is
verified :
 
   p11 p12 p13 p14     p11 p12 p14   
m  M  m    p21 p22 p23 p24     =  p21 p22 p24     = H  M Π
0
   p31 p32 p33 p34     p31 p32 p34   
 
, which in coordinates of the image and the
plane results in :

u   p11 p12 p14   X 

  v  =  p21 p22 p24    Y 
 1   p31 p32 p34   1 

With 4 pairs of homologous points in

"general position", it is possible to obtain the
8 independent parameters in H .

Once found, it allows obtaining any point of

the  plane from a single image without
speaking in terms of rotations, translations,
or camera parameters.

VMMC – 2022 50 Unit I: The projected reality

 Affine interpretation of a projection :
T
According to the projection matrix, the points at infinity, M = dT 0 , are projected on the points of
the image plane m = P  d , being =  P p . Therefore, it can be considered that P defines a
homography between the plane at infinity   and the image plane.

The projections of a set of parallel lines of the world that join in a point of the plane at infinity have to join
in the corresponding point of the image plane. Vice versa, given a point m of the image plane, d = P−1  m
is the direction of the parallels of the world whose projections vanish to it.

Since P depends on A, R but not

on t , it is concluded that the
projection of   is invariant to
translations.

The effect of the translations is

included in p , which is the
projection of the origin of world
coordinates :

O = 0 0 0 1
T

VMMC – 2022 51 Unit I: The projected reality

 The absolute conic:

In the projective plane, , the intersection of the line at infinity with any circumference results in two
2

complex points (located on the line at infinity) :

( x − a) + ( y − b) = c 2  x 2 + y 2 − 2ax − 2by + ( a 2 + b2 − c 2 ) = 0
2 2
Affine:

Homogeneous: 2
+ 2
− 2a − 2b + ( a 2 + b2 − c 2 ) 2
=0


 I ( i 0) (1 i 0)
Intersection with =0:
2
+ 2
=0 = i 

J ( −i 0 ) (1 −i 0 )

In the projective space, , the intersection of the plane at infinity with any sphere results in a complex
3

conic (located in the plane at infinity) that is called the absolute conic c :

=  P p 1 0 0   
= 0: 2
+ 2
+ 2
=  0 1 0   = 0
0 0 1   
ω

The similarities are precisely the homographies that preserve this conic.

VMMC – 2022 52 Unit I: The projected reality

 The absolute conic and the matrix of internal parameters :

The points at infinity, ( , , ,0 )  , are projected on m = P    , from which it resultsT

T

that a point m of the image plane corresponds to a point at infinity such that P −1  m =  .
The infinite points ( = 0 ) that belong to the absolute conic will verify its equation :

(P  m )  ω  ( P−1  m ) = 0  ( P−1  m )  ( P −1  m ) = mT  P −T  P −1  m = 0
−1 T T

Taking into account that P = A  R  P−T  P−1 = ( A  R )  ( A  R ) = A−T  A−1 , it finally results:
−T −1

mT  A−T  A−1  m = mT  B  m = 0, B = A−T  A−1

, that is to say, the points of the image plane that are projection of the absolute conic (called Image of the
Absolute Conic, IAC) form at the same time a conic that is given by the matrix: B = A−T  A−1

Note that B does not depend on the position or orientation of the camera, and that once known B , its
inverse is the Krupa matrix K from which A can be directly obtained :

 a c u0   a 2 + c 2 + u02 bc + u0 v0 u0 
 
A =  0 b v0   K = B −1 = ( A −T  A −1 ) = A  AT =  bc + u0 v0
−1
b 2 + v02 v0 
 0 0 1   u0 v0 1 


VMMC – 2022 53 Unit I: The projected reality

 Homography of a flat pattern and the image of the absolute conic :

Be a plane  = ( 0, 0,1, 0 ) , that is, aligned with the axes X , Y . As it has been seen, for the points of
this plane it is verified :

m H  M , where, for A  r1 r2 r3 t  results: H = h1 h2 h3  A  r1 r2 t

, from where clearing r1 , r2 and taking into account that they are orthonormal, we obtain the relations :
r1T  r2 = 0  ( A−1  h1 )  ( A−1  h2 ) = h1T  A−T  A−1  h2 = h1T  B  h2 = 0
T

r1T  r1 = r2T  r2  ...  h1T  B  h1 = hT2  B  h2

, which only depend on the orientation of the plane, not on its position. Thus, an orientation of the plane
defines a homography that establishes two linear restrictions in matrix B of the IAC. Since this matrix
only depends on five parameters, it is enough to obtain three homographies (resulting from three
orientations of the plane) to find all their values.

VMMC – 2022 54 Unit I: The projected reality

 Obtaining a homography :

A possible method (the one proposed by Zhang) consists of two steps.

First we obtain an initial approximation to the parameters of the matrix H following the linear method
based on coordinates, but simplified by the fact that Z = 0 , which requires a minimum of n = 4.5 points,
because the "3D" coordinates of the M i points of the board (that is, their 2D positions on the board) it is
possible to calculate them a priori automatically and precisely, and because the marking of the m i points
of the image is possible to assist it with automatic tools.

ˆ i of the points M i
Secondly, the homography obtained is used to calculate the estimated projections m
on the image plane, and iterated to minimize the geometric error :
n

m
i =1
i −m
ˆi

For numerical reasons, in practice n = 9

points are necessary for the results to
converge.

The points must be as distributed as

possible in the image.

VMMC – 2022 55 Unit I: The projected reality

 Obtaining the external parameters :

Starting from one of the calculated homographies:

H = h1 h2 h3  A  r1 r2 t     h1 h2 h3  = A  r1 r2 t

, it is possible to establish the following relationships:

  A−1  h1 = r1   A−1  h2 = r2   A−1  h3 = t

Taking into account that A is known, from the first relation it is possible to obtain the proportionality
factor by imposing unity norm on the rotation vector:

1
  A −1  h1 = r1 = 1   =
A −1  h1

The rest of the external parameters is immediate to obtain them:

A −1  h1 A −1  h 2 A −1  h3
r1 = −1 r2 = −1 t = −1 r3 = r1 ×r2
A  h1 A  h1 A  h1

Note that they are obtained with respect to the origin of world coordinates, that is, with respect to the
origin of coordinates ( X = 0, Y = 0 ) taken in the pattern.

VMMC – 2022 56 Unit I: The projected reality

 Distortion produced by the lens:

Distortion by which a point of the scene that, in the absence of non-linearities, should be projected at
point m of the image plane, in practice it does so at point m , so that :

m = m + δ ( r , ) = m + δr + δ , being:

δ r : radial distortion, due to radial differences of magnification factors of the lens.

δ : tangential distortion (decentering), due to errors in the centering of the elements of the lens system.

VMMC – 2022 57 Unit I: The projected reality

 Ortographic Barrel Pincushion

Top images obtained from http://www.vanwalree.com

Bottom images obtained from http://www.panotools.org

VMMC – 2022 58 Unit I: The projected reality

 Correction of radial distortion :

By working directly on the image (that is, in pixel coordinates), the following relationship can be
established :
m = cd + L ( r )  ( m − cd )

, m being the corrected point, m the projected point (uncorrected), c d the center of radial distortion
(which does not have to coincide with the principal point), r = m − cd the distance between the
projected point and the center of distortion, and L ( r ) the function that models the variation of distortion
as a function of the aforementioned distance. L ( r ) only takes values for positive r , and in general it can
be approximated in Taylor series by :

L ( r ) = 1 + k1r + k2 r 2 + k3r 3 + ...

The most general approach to the calculation of the L ( r ) function and the distortion center c d is to
impose that the straight line images are effectively straight.

VMMC – 2022 59 Unit I: The projected reality

 The parallax effect:

With parallax: Without parallax:

• No biunivocal relationship between projections. • One-to-one relationship between projections

• The relative positions of the objects vary. • The relative positions of the objects do not vary.
• 3D reconstruction is possible: once known m • 3D reconstruction is not possible: the knowledge
and m it is possible to find M by triangulation. of m and m does not allow locating point M .

VMMC – 2022 60 Unit I: The projected reality

 Correspondence between two images or projections of the same 3D plane:

As we have seen, the relationship between the points

of the plane  and their projection in the image
plane is given by a homography. The same is
true for the projection of  over  . Thus:

m HM 
  m H  H −1  m = H   m
m H  M 

It is possible to obtain a closed expression for the

general case of homography H  as a function of the
projection matrices ,  and the  plane.

In conclusion, well-known the projection of a point M

on , is known directly its projection on  , that
was the main objective.

For the particular case in which  =   : m PM 

  m P  P −1  m = H   m
m P  M 

VMMC – 2022 61 Unit I: The projected reality

 Correspondence in the particular case of absence of parallax:

A stationary camera, that is to say, that only rotates

around its own optical center, according to a matrix of
rotation R , can be seen as a particular case in
which C = C .

The situation describes a projective transformation

between the image planes and  that is
independent of the plane on which the points of the
scene are situated (i.e., of their 3D position):

m HR  m, HR = Ha = Hb = H

The last equality allows to relate the homography

with the projection matrices ,  :

HR = H = P  P −1

This situation does not allow recovering the depth of a point M from its projections, but it does allow, for
example, to obtain the matrix of internal parameters of a rotating camera.

VMMC – 2022 62 Unit I: The projected reality

 Epipolar geometry concepts:

The line joining the optical centers C, C intersects

the image planes ,  respectively in two specific
points, the epipoles e, e .

The lines of the image plane passing through the

epipole (case of lm and l m ) are called epipolar
lines.

The plane  defined by the two rays Lm and Lm

that project the point M in the image planes, is
called the epipolar plane. Intersects with the image
planes in two epipolar lines.

The epipolar constraint:

The projection of point M in the image plane  (or ) must fall on the straight line projection of the ray Lm
(or Lm ) over  (or ), which must be an epipolar line, that is, it must include the epipole e (or e ).

This conclusion restricts the search from m in  (or from m in ) to a 1D problem. It only remains to
know the direction of the image plane in which to look for...

VMMC – 2022 63 Unit I: The projected reality

 The fundamental matrix:

The relationship between a projection m (in ) and its epipolar line lm (in  ) is the composition of
two projective transformations: that of the Lm ray at point m (of ) and that of the same ray on the
epipolar line lm (of  ), both dependent on the position of the cameras (not on the projected point and
ray). Therefore, this relation is also a projective transformation, defined by a 3x3 matrix F that is called
the fundamental matrix and that is unique for a pair of points of view:
lm F m
For calibrated cameras, as we will see later, it is possible to deduce a closed expression for the matrix F
from the projection matrices ,  . For uncalibrated cameras, F can be obtained from several pairs of
matching projections:

Expression of the epipolar constraint: m  lm  mT  lm = 0  mT  F  m = 0 or mT  FT  m = 0


Since epipole e belongs to the epipolar line lm , it is verified: l m F T  m

eT  F  m = 0, m  eT  F = 0  FT  e = 0 and, analogously, F  e. = 0

In conclusion, the rank of F must be 2 ( det ( F ) = 0 ) . So, taking into account also the scale
factor, the matrix F just depends on 7 (9-2) parameters.

Calculation of F : each pair of points m, m offers a linear constraint or equation in the parameters of F
. Therefore, 7 points would be enough to calculate it. In practice, due to noise, the calculation is complex.

VMMC – 2022 64 Unit I: The projected reality

 Planar homographies vs fundamental matrix:

If the points e and m belong to the same epipolar

line lm it is verified:

e  m = lm  e  H  m = lm

Taking into account that lm F  m , it results :

F  m e  H  m  F e  H
Now applying this expression to the epipole e :

F e e  H  e  e  H  e = 0
0

, so it is further concluded that H  e e .

 0 −1 ev1 
In conclusion, if you know F , you only need 3 points
to find H  . The epipoles, which can be obtained e =  1 0 −eu 
from F , supplement the fourth point.  −ev1 eu 0 
Additionally, being any H  related to F and being this matrix unique, all
planar homographs are related (knowing one is possible to generate all the
others).

VMMC – 2022 65 Unit I: The projected reality

 Computation of the fundamental matrix from the projection matrices:

T
The optical center, expressed in homogeneous coordinates C = CT c  , has to verify :

 −P −1p 
 C = 0  P  C + pc = 0  C  
 1 

Additionally, the epipole of the image plane  has to verify :

 −P −1p 
e C  P  p   p − PP −1p
 1 

By applying a planar homography we would obtain the fundamental matrix. Choosing the homography H  :

F = e  H = p − PP −1p   P  P −1



This relationship allows you to obtain F directly in calibrated camera systems and, therefore, obtain the
search direction for homologous points.

VMMC – 2022 66 Unit I: The projected reality

 Coplanar image planes and an aligned axis :

In this situation the epipolar plane intersects the

image planes in a same line :

lm = lm (0 1 −mv )

, where m = ( mu , mv ) and m = ( mu , mv ) , so

the search for homologous points is limited to the
horizontal of the point m .

The expression of the fundamental matrix is in

this case very simplified:
 mu   0  0 0 0
F0  m lm  F0   mv   1  , mu , mv  F0 0 0 1 
 
 1   −mv  0 −1 0

, expression that also verifies the epipolar restriction (which in this case forces points m and m to have the
same vertical affine coordinate):
0 0 0   

mT  F0  m = 0    0 0 1     =  −  = 0   =  v = v
0 −1 0    

VMMC – 2022 67 Unit I: The projected reality

 Image rectification:

Two image planes are related by the fundamental

matrix F , so the epipolar restriction is verified:

mT  F  m = 0

If on each plane a homography is applied to place

them parallel, the fundamental matrix that relates the
transformed planes becomes F0 :

mˆ = H m 
  ( H  m )  F0  ( H  m ) =
T

ˆ  = H   m 
m
= mT  HT  F0  H  m = 0
F

Given F , the objective is to look for two

homographies H, H , which satisfy the previous
relationship.

VMMC – 2022 68 Unit I: The projected reality

 Disparity maps:

Left view Right view

Disparity map

Databases from “vision.middlebury.edu/stereo/”

VMMC – 2022 69 Unit I: The projected reality

 Invariants in binocular reconstruction :

Absolute conic
Straightness
Mundo
3D World
Reconstrucción
Projective
Plane at infinity proyectiva
reconstruction

‘Real’ points

Reconstrucción
Affine
Reconstrucción ‘euclidea’
Euclidean reconstruction reconstruction
afín

+ Absolute conic + Plane at infinity

+ Angles + Parallelism
+ Distances ratio + Distances ratio for
aligned points

VMMC – 2022 70 Unit I: The projected reality