Lecture 3: Image processing

basics:Image Formation and Overview

of projective geometry
Deepak Mishra
[email protected]
CCD Camera
https://sites.google.com/a/globalsystemsscience.org/digital-earth-watch/tools/digital-cameras-overview/what-happens-to-the-near-infared-entering-the-camera
Full frame sensor gathers 50x more light than phone sensor
CMOS sensor

photodiode

https://micro.magnet.fsu.edu/primer/digitalimaging/cmosimagesensors.html http://www.ysctech.com/digital-microscope-CCD-camera-info.html
CMOS fabrication
CMOS vs. CCD

CCD (Charged Coupled Device) CMOS

+ Larger fill factor(the ratio of a pixel’s light + Random access to pixels
sensitive area to its total area) + Cheaper to manufacture
- Harder to read out image - Lower fill factor
- More expensive https://meroli.web.cern.ch/lecture_cmos_vs_ccd_pixel_sensor.html
Shutters

CCD requires a shutter CMOS

- Stop exposing while data read off + Can expose all the time (rolling shutter)
- Causes some new artifacts

https://meroli.web.cern.ch/lecture_cmos_vs_ccd_pixel_sensor.html
Rolling shutter artefact (write the difference between
temporal aliasing (waggon wheel effect) and rolling
shutter artefacts )
Vanishing points and lines
Recap : Vanishing point
Vertical vanishing
point
(at infinity)

Vanishing
Vanishing
point
point

Slide from Efros, Photo from Criminisi

 Projection: world coordinates image coordinates
Projection: world coordinates!image coordinates
. .
x
X= y
z
. .
3 3
f
2
f
2
5
z
. y 5
z
y

v’ ? v ?

. .
Camera Camera
Center
Center
u (0, 0, 0)
u’ (0, 0, 0)
u'
x=
v'
f 2
If X = 2, Y = 3, u' = x * u' = 2 *
z 5
If X = 2, Y = 3, Z = 5, and f = 2 v' y f 2
What are U and V? = v' = y * v' = 3 *
Z = 5, and f = 2 f z z 5
What are U and V?
Projection: world coordinates!image coordinates

Optical
Center
.
. .
(u0, v0)
f Z Y

.u
Camera
Center
(tx, ty, tz)

How do we handle the general case?

Homogenous coordinate system
Homogeneous coordinates
Conversion
Converting to homogeneous coordinates

homogeneous image homogeneous scene

coordinates coordinates

Converting from homogeneous coordinates

Properties of Homogenous
Homogeneous coordinates coordinate

Invariant to scaling
x kx kx x
k y = ky kw
ky = w
y
kw w
w kw
Homogeneous Cartesian
Coordinates Coordinates

Point in Cartesian is ray in Homogeneous

Camera Projection matrix Slide Credit: Savarese

Projection matrix
R,t
jw

X
kw
Ow

x iw

x: Image Coordinates: (u,v,1)

x=K R t X K: Intrinsic Matrix (3x3)
R: Rotation (3x3)
t: Translation (3x1)
X: World Coordinates: (X,Y,Z,1)
 Projection
Projection matrix
matrix
X

Intrinsic Assumptions Extrinsic Assumptions

Unit aspect ratio No rotation
Optical center at (0,0) Camera at (0,0,0)
No skew K

x
u f 0 0 0
y
x=K I 0 X w v = 0 f 0 0
z
1 0 0 1 0
1
Slide Credit: Savarese
 Projection Matrix
Remove assumption: known optical center

Intrinsic Assumptions Extrinsic Assumptions

Unit aspect ratio No rotation
No skew Camera at (0,0,0)

x
u f 0 u0 0
y
x=K I 0 X w v = 0 f v0 0
z
1 0 0 1 0
1
 Projection matrix
Remove assumption: square pixels

Intrinsic Assumptions Extrinsic Assumptions

No skew No rotation
Camera at (0,0,0)

x
u 0 u0 0
y
x=K I 0 X w v = 0 v0 0
z
1 0 0 1 0
1
Camera Projection matrix
Remove assumption: non-skewed pixels

Intrinsic Assumptions Extrinsic Assumptions

No rotation
Camera at (0,0,0)

x
u s u0 0
y
x=K I 0 X w v = 0 v0 0
z
1 0 0 1 0
1
 Camera Projection matrix
Oriented and Translated Camera

R
jw

X
t kw
Ow
x iw
 Camera Projection matrix
Allow camera translation

Intrinsic Assumptions Extrinsic Assumptions

No rotation

x
u 0 u0 1 0 0 t x
y
x=K I t X w v = 0 v0 0 1 0 t y
z
1 0 0 1 0 0 1 tz
1
Camera Projection matrix
Slide Credit: Saverese

3D Rotation of Points

Rotation around the coordinate axes, counter-clockwise:

1 0 0
Rx ( ) = 0 cos sin
p’
0 sin cos
cos 0 sin
Ry ( ) = 0 1 0
p
y
sin 0 cos
cos sin 0
Rz ( ) = sin cos 0
z
0 0 1
Camera Projection matrix
Allow camera rotation

x=K R t X

x
u s u0 r11 r12 r13 t x
y
w v = 0 v0 r21 r22 r23 t y
z
1 0 0 1 r31 r32 r33 t z
1
Camera Projection matrix
Degrees of freedom

x=K R t X

5 6
x
u s u0 r11 r12 r13 t x
y
w v = 0 v0 r21 r22 r23 t y
z
1 0 0 1 r31 r32 r33 t z
1
Field of View (Zoom, focal length)
 Lens distortion in camera
Beyond Pinholes: Radial Distortion

Corrected Barrel Distortion

 Primitives and transformation
Geometric primitives are the basic building blocks used to describe 3D shapes

• ︎ In this unit, we introduce points, lines and planes

• ︎ Furthermore, the most basic transformations are discussed

• ︎ This unit covers the topics of the Szeliski book, chapter 2.1

• ︎ A more exhaustive introduction can be found in the book: Hartley and

Zisserman: Multiple View Geometry in Computer Vision
 Homogenous point
2D points can be written in inhomogeneous coordinates as

(y)
x 2
X= ∈ℝ

x
(1)
X̃ = y ∈ℙ 2

Where ℙ2 = ℝ3 is called projective space. Except the element (0,0,0)

Remark: Homogeneous vectors that differ only by scale are considered equivalent and define an
equivalence class. ⇒ Homogeneous vectors are defined only up to scale.
 Two Dimensional point
An inhomogeneous vector x is converted to a homogeneous vector x as follows

x̃ x
(1)
= ( ) = X̄
y X
X̃ = ỹ =
1
w̃
with augmented vector X̄.To convert in the opposite direction we divide by w̃:
x̃
x x̃ w̃

(1)
1 1
= ( ) = X̃ =
y X ỹ =
X̄ = x̃
1 w̃ w̃ w̃
w̃
1
Homogeneous points whose last element is w̃ = 0 are called ideal points or points at infinity. These
points can’t be represented with inhomogeneous coordinates!
�D Points
Homogeneous Vector

2D-
points
Augmented Vector

Homogeneous Inhomogeneous
Coordinates Coordinates
 2D-Line
˜
2D lines can also be expressed using homogeneous coordinates l = [a, b, c] T

˜T
{X̄ | l X̄ = 0} ⇔ {x, y | ax + by + c = 0}
˜ ˜ T
We can normalize l so that l = (nx, ny, d) = (n, d) with | | n | | = 1. In this case, n is the normal
vector perpendicular to the line and d is its distance to the origin.

An exception is the line at infinity l∞ = (0,0,1) which passes through all ideal points.
 Cross Product
Cross product expressed as the product of a skew-symmetric matrix and a vector:

0 −a3 a2 b1 a2b3 − a3b2

a × b = [a]×b = a3 0 −a1 b2 = a3b1 − a1b3
−a2 a1 0 b3 a1b2 − a2b1

we use squared brackets to distinguish matrices from vectors.

 2D Line Arithmetic
In homogeneous coordinates, the intersection of two lines is given by:

˜ ˜
X̃ = l1 × l2
Similarly, the line joining two points can be compactly written as:
˜l = X̃1 × X̃2

The symbol × denotes the cross product. Proof as exercise.

�D Line Arithmetic 2D line Arithmetic

2 30 1 0 1 0 1
0 1 1 1 2 2
6 7B C B C B C
l̃1 ⇥ l̃2 = [l̃1 ]⇥ l̃2 = 4 1 0 05 @ 0 A = @ 1A = @1A
1 0 0 2 1 1
�D Line Arithmetic

2 30 1 0 1 0 1
0 1 0 1 0 ⇣ ⌘> 0
6 7B C B C B C
l̃1 ⇥ l̃2 = [l̃1 ]⇥ l̃2 = 4 1 0 15 @ 0 A = @1A 0 0 1 @1A = 0
0 1 0 2 0 | {z } 0
l̃1
��
 2.2 The 2D projective plane 29
x2

ideal
point

O x3
x

x1

Fig. 2.1. A model of the projective plane. Points and lines of IP2 are represented by rays and planes,
respectively, through the origin in IR3 . Lines lying in the x1 x2 -plane represent ideal points, and the
x1 x2 -plane represents l∞ .

2
points lie on a single line. This is not true in the standard Euclidean geometry of IR ,
 Vanishing point
Vanishing points
image plane

vanishing point v
COP

line on ground plane

Vanishing point
• projection of a point at infinity
Vanishing points
image plane

vanishing point v

COP

line on ground plane

http://stevewebel.com/
line on ground plane

Properties
• Any two parallel lines have the same vanishing point v
• The ray from COP through v is parallel to the lines
From: http://www.atpm.com/9.09/design.shtml
 2
Conics in P
Conics in P2

A conic in affine space (inhomogeneous coordinates) is

2 2
ax + bxy + cy + dx + ey + f = 0

Homogenizing this by replacements x = X1 /X3 and y = Y1 /Y3 , we

obtain
2 2 2
aX1 + bX1 X2 + cX2 + dX1 X3 + eX2 X3 + fX3 =0

which can be written in matrix notation as T

X CX
= 0 where C is
symmetric and is the homogeneous representation of a conic.
�D Conics
Conics
More complex algebraic objects can be represented using polynomial homogeneous
equations.
More complexForalgebraic
example,objects
conic sections (arising as the
can be represented intersection
using polynomialof ahomogeneous
plane and a
�D cone) conic
example, can besections
written using quadric
(arising as the equations:
intersection of a plane and a written using
quadric equations:
T
| X̄ Q
{x̄ X̄| x̄ > Qx̄
X̄ =
= 0}
0

Circle Ellipse Parabola Hyperbola

Useful for multi-view

Usefulgeometry and camera
for multi-view calibration,
geometry see Hartley
and camera and Zisserman.
calibration,
 2
Conics in P

I The line l tangent to a conic C at any point x is given by

l = Cx.
I t
x Cx = 0 =) (C 1 l)t C((C 1 l)= t 1
lC l =0
t
(because C = C 1 ). This is the equation of the dual conic.
 2
Conics in P

I The degenerate conic of rank 2 is defined by two line l and m

as
t t
C = lm + ml
t
Points on line l satisfy
lx = 0 and are hence on the conic
t t t t
because (x l)(m x) + (x m)(l x) = 0. (Similarly for m).
t t
The dual conic xy + yx represents lines passing through x
and y.
 Three Dimensional Points
3D points can be written in inhomogeneous coordinates as
x
(z)
X= y ∈ℝ 3

or in homogeneous coordinates as

x̃
ỹ 3
X̃ = ∈ℙ
z̃
w̃
3 4
with projective space ℙ = ℝ excluding{(0,0,0,0)}
�D Planes
3D-Planes
�D planes can also be represented as homogeneous coordinates m̃ = (a, b, c, d) :
>

>
{x̄ | m̃ x̄ = 0} , {x, y, z | ax + by + cz + d = 0}

Again, we can normalize m̃ so that m̃ = (nx , ny , nz , d)

= > with knk2 = 1. In
(n, d) >

this case, n is the normal perpendicular to the plane and d is its distance to the origin.

An exception is the plane at in�nity m̃ = (0, 0, 0, 1) > which passes through all ideal
points (= points at in�nity) for which w̃ = 0.
�D Planes

3D
Planes
�D Lines

3D lines
�D lines are less elegant than either �D lines or �D planes. One possible representation
is to express points on a line as a linear combination of two points p and q on the line:

{x | x = (1 )p + q ^ 2 R}

However, this representation uses 6 parameters for � degrees of freedom.

Alternative minimal representations are the two-plane parameterization or Plücker
coordinates. See Szeliski, Chapter �.� and Hartley/Zisserman, Chapter � for details.

��
�D Quadrics 3D quadrics
The �D analog of �D conics is a quadric surface:

>
{x̄ | x̄ Q x̄ = 0}

Useful in the study of multi-view geometry. Also serves as useful modeling primitives
(spheres, ellipsoids, cylinders), see Hartley and Zisserman, Chapter � for details.

�8
Projective transformation
 tions. From
possible, basedthis
on point of view, result.
the following 2D projective geometry is the study of properties of
2
the projective plane IP that are invariant transformations
2.3 Projective under a group of transformations known 33 as
projectivities. 2 2
Theorem 2.10. A mapping h : IP → IP is a projectivity if and only if there exists a
A projectivity
non-singular
mous.
isvectors)
true
3
toh(x)
that
×Projective transformation
(a helpful name), a projective transformation or a homography:
points
=
is matrix
3 an invertible
inHx. 2
H such mapping
that for from
any points
point in IP 2
in IP
IP that maps lines to lines. More precisely,
2
the
represented
terms
by
are
a
synony-
(that is homogeneous
vector x 3-it
In definition 2.9, a projectivity is defined in terms of a coordinate-free geometric
2
concept
Definition of 2.9.
pointAline incidence.is An
projectivity an equivalent
invertible2
algebraic
mapping hdefinition
from IP of
to a projectivity
itself such is
that
To interpret
possible, based
this theorem, any point in IP is represented as a homogeneous 3-vector,
three points x1 ,onx2the
andfollowing
x3 lie on result.
the same line if and only if h(x1 ), h(x2 ) and h(x3 ) do.
x, and Hx is a linear mapping of homogeneous coordinates. The theorem asserts that
any Projectivities
projectivity form aasgroup
arises such since
a 2 thetransformation
linear inverse
2 of a projectivity
in is also a coordinates,
homogeneous projectivity, and
Theorem 2.10. A mapping h : IP → IP is a projectivity if and only if there exists a
so is
that the composition
conversely any suchof mapping
two projectivities.
is a A projectivity
projectivity. The 2 is alsowill
theorem called
not a be
collineation
proved in
non-singular 3 × 3 matrix H such that for any point in IP represented by a vector x it
full here. It will
is true that h(x) = Hx. only be shown that any invertible linear transformation of homoge-
neous coordinates is a projectivity.
2
To interpret this theorem, any point in IP is represented as a homogeneous 3-vector,
T
Proof.
x, and Hx Letisxa1 ,linear
x2 and x3 lie on
mapping of ahomogeneous
line l. Thus coordinates.
l xi = 0 for The i = theorem
1, . . . , 3.asserts
Let H that
be a
T −1
non-singular
any projectivity 3 ×arises
3 matrix. One
as such verifies
a linear that l H Hxin
transformation i = 0. Thus, the coordinates,
homogeneous points Hxi alland lie
−T
on
thatthe line H l,any
conversely andsuch
collinearity
mappingisispreserved
a projectivity.by theThetransformation.
theorem will not be proved in
full here.
The It will
converse only be shown
is considerably that to
harder anyprove,
invertible
namelylinear
thattransformation
each projectivity of homoge-
arises in
neous
this way.coordinates is a projectivity.
The converse is considerably harder to prove, namely that each projectivity arises in
this way. Definition of projective
transformation
As a result of this theorem, one may give an alternative definition of a projective
transformation (or collineation) as follows.

Definition 2.11. Projective transformation. A planar projective transformation is a

linear transformation on homogeneous 3-vectors represented by a non-singular 3 × 3
matrix:
 "
   
x1 h11 h12 h13 x1
 "    
 x2  =  h21 h22 h23   x2  , (2.5)
"
x3 h31 h32 h33 x3
"
or more briefly, x = Hx.

Note that the matrix H occurring in this equation may be changed by multiplication
by an arbitrary non-zero scale factor without altering the projective transformation.
Consequently we say that H is a homogeneous matrix, since as in the homogeneous
34 2 Projective Geometry and Transformations of 2D

/
x
O
/

y/
x/ y
x

Fig. 2.3. Central projection maps points on one plane to points on another plane. The projection
also maps lines to lines as may be seen by considering a plane through the projection centre which inter-
sects with the two planes π and π! . Since lines are mapped to lines, central projection is a projectivity
and may be represented by a linear mapping of homogeneous coordinates x = Hx.!

Mappings between planes. As an example of how theorem 2.10 may be applied,

 Change in coordinate convention
Projective transformations are important mappings representing many more situations
than36the perspective imaging of a world
2 Projective plane.
Geometry and Transformations of 2D
image 1 image 2
X

R,t x
x image 1 x x

x
x/

image 2
X
planar surface

a b c

Fig. 2.5. Examples of a projective transformation, x! = Hx, arising in perspective images. (a)
The projective transformation between two images induced by a world plane (the concatenation of two
projective transformations is a projective transformation); (b) The projective transformation between
two images with the same camera centre (e.g. a camera rotating about its centre or a camera varying its
focal length); (c) The projective transformation between the image of a plane (the end of the building)
and the image of its shadow onto another plane (the ground plane). Figure (c) courtesy of Luc Van Gool.
CS 4495 Computer Vision – A. Bobick
Isometrics
Two Views Part 1: 2D transforms and
Projective Geometry

Special Projective Transformations

• Translation Isometries are transformations of the
plane IR that preserve Euclidean
2

x' 1 0 tx x distance (from iso = same, metric =

measure)
y' 0 1 ty y
1 0 0 1 1

• Preserves:
• Lengths/Areas R
• Angles
R
• Orientation
• Lines
 the matrix diag(−1, 1, 1),
2.4.1 Class with a Euclidean transformation.
I: Isometries Two Views Part 1: 2D transforms and

Isometrics
CS 4495 Computer Vision – A. Bobick Projective Geometry
2
Isometries are transformations of the plane IR that preserve Euclidean
Euclidean transformations modelAnthe motion of aasrigid o
Special Projective Transformations
iso = same, metric = measure). isometry is represented
most important isometries in practice,  ! 
x

and
! cos θ − wesin θwill
tx concentr

x

 !    
• Euclidean (Rigid body)  y  =  ! sin θ
orientation reversing isometries1 often arise cos θ ty   y 
as ambiguities in
0 0 1 1
x' A planar
cos( ) Euclidean ) !t x= transformation
sin(where x can be written more c
±1. If ! = 1 then the isometry is orientation-preserving and
transformation (a composition of a translation-and rotation)..If ! = −1
y' sin( ) cos( ) t y y R t
try reverses orientation. An! example is the composition of a reflection
x =H x=
the matrix diag(−1, 1, 1), with a Euclidean transformation. x
1 0 0 1 1 E T
0 1
Euclidean transformations model the motion of a rigid object. The
most important isometries in practice, and we will concentrate on thes
where R is a 2 × 2 rotation matrix (an orthogonal matrix s
orientation reversing isometries often arise as ambiguities in structure r
A planar Euclidean transformation can be written more concisely in
t a translation 2-vector, and 0 a null 2-vector.
• Preserves: - Special
. case
! R t
• Lengths/Areas
t = 0) and a pure x = x =
R translation (when R = I). 0A 1Euclidean tra
H T x E

• Angles
• Lines
as a displacement.
where R is a 2 × 2 rotation matrix (an orthogonal matrix such that R T

t a translation 2-vector, and 0 a null 2-vector. Special cases are a pure

A planar Euclidean transformation has three degrees of
t = 0) and a pure translation (when R = I). A Euclidean transformatio
fr
and two for the astranslation.
a displacement. Thus three parameters must be
CS 4495 Computer Vision – A. Bobick
2.4.2 Class II: Similarity
Two Viewstransformations
Part 1: 2D transforms and
Projective Geometry
A similarity transformation (or more simply a similarity) is an
Special Projective Transformations
Similarity transformation an isotropic scaling. In the case of a Euclidean transformation
(i.e. no reflection) the similarity has matrix representation
A similarity transformation (or more
• Similarity (trans, rot, scale) transform  simply!

a

similarity) is an isometry
 
x s cos θ −s sin θ tx x
 composed
!  with an isotropic scaling. In
 y  =  s sin θ s cos θ ty   y
x' a cos( ) a sin( ) t x x
the
1
case of a Euclidean
0
transformation
0 1 1
composed with a scaling (i.e. no
y' a sin( ) a cos( ) t y
This can be written
y
reflection) the similarity has matrix
more concisely in block form as
representation
1 0 0 1 1 !
- .
sR t
x = HS x = T x
0 1

where the scalar s represents the isotropic scaling. A similar

• Preserves: known as an equi-form transformation, because it preserves
• Ratios of Areas similarity transformation has four degrees of freedom, the sc
R
• Angles more degree of freedom than a Euclidean transformation. A si
from two point correspondences.
• Lines
 tion (chapter 10) is metric. The description metric structure im
is defined up to a similarity.
Two Views Part 1: 2D transforms and

Affine Transform
CS 4495 Computer Vision – A. Bobick Projective Geometry

Special Projective Transformations

2.4.3 Class III: Affine transformations
An affine transformation (or more simply an affinity)
• Affine transform An affine transformation
is a non-singular (or
linear
rotation more simply
transformation an affinity)
followed by is
a a non-
defor
mation followed It
translation. byhas
a translation. It has the matrix representati
the matrix representation
x' a b c x  a    
!
x a11 a12 tx x
y' d e f y Fig. 2.7. Distortions arisingfrom ! a planar
 affine transformation.
  (a)R
mation R(−φ) D R(φ). Note,  the yscaling  a21 inathe
 =directions ty   y are
22 deformation o
1 0 0 1 1 1 0 0 1 1
or in block form
! "
• Preserves: ! A t
x = HA x = x
• Parallel lines R
0T 1
• Ratio of Areas
• Lines
with A a 2 × 2 non-singular matrix. A planar affine transform
with A a 2 × 2 non-singular matrix
freedom corresponding to the six matrix elements. The trans
puted from three point correspondences.
A helpful way to understand the geometric effects of the
an affine transformation is as the composition of two funda
freedom corresponding to the six matrix elements. The transformation can be com-
puted from three point correspondences.
A helpful way to understand the geometric effects of the linear component A of
an affine transformation is as the composition of two fundamental transformations,
namely rotations and non-isotropic scalings. The affine matrix A can always be decom-
posed as

A = R(θ) R(−φ) D R(φ) Affine Transform

(2.12)

where R(θ) and R(φ) are rotations by θ and φ respectively, and D is a diagonal matrix:
! "
λ1 0
D= .
0 λ2
40 2 Projective Geometry and Transformations of 2D
This decomposition follows directly from the SVD (section A4.4(p585)): writing A =
UDVT = (UVT )(VDVT ) = R(θ) (R(−φ) D R(φ)), since U and V are orthogonal matrices.
The affine matrix A is hence seen to be the concatenation of a rotation (by φ); a
scaling by λ1 and λ2 respectively in the (rotated) x and y directions; a rotation back
(by −φ); and finally another rotation (by θ). The only “new” geometry, compared to
a similarity, is the non-isotropic scaling. This accounts for the two extra degrees of
freedom possessed by an affinity over a similarity. They are the angle φ specifying the
scaling direction, and the ratio of the scaling parameters λ1 : λ2 . The essence of an
affinity is this scaling in orthogonal directions, oriented at a particular angle. Schematic
examples are given in figure 2.7.
rotation deformation
a b
 Two Views Part 1: 2D tran
CS 4495 Computer Vision – A. Bobick Projectiv

Projective Transformations
• Remember, these are homogeneous coordinates

x' sx ' a b c x
y' sy ' d e f y
1 s 0 0 s 1
 A projective transformation was defined in (2.5). It is a ge

General Projective transform

CS 4495 Computer Vision – A. Bobick
transformation of homogeneous coordinates. This generalizes
which is
Two the
Views composition
Part 1: 2D transforms and of a general non-singular linear tran
Projective Geometry
neous coordinates and a translation. We have earlier seen t
(Homography)
Projective Transformationstransformation (in section 2.3). Here we examine its block fo
" #
• General projective transform (or Homography) ! A t
x = HP x = T x
v v
x' x' a b c x
T
y' y' d e f where
y the vector v = (v1 , vThe
2 ) . The matrix has nine eleme
matrix has nine elements with
significant, so the transformation is specified
only their by eight
ratio significant, paramet
so the
1 w' g h i w
possible to scale the matrix such that v is isunity
transformation since
specified by v mig
eight parameters. Note,
transformation between two planes can be computed from fou it is not
always possible to scale the matrix
with no three collinear on either
suchplane.
that v isSee
unityfigure
since v2.4.
might
• Preserves: Unlike the case of affinities, itbeiszero.
notApossible
projective to distingu
transformation between
preserving and orientation reversing projectivities in IP . We two 2
• Lines R planes can be computed from four
in section 2.6. point correspondences, with no
• Also cross ratios three collinear on either plane.
(maybe later)
44 Summary
2 Projective Geometry and Transformations of 2D

Group Matrix Distortion Invariant properties

Concurrency, collinearity, order of contact:
! "
h11 h12 h13 intersection (1 pt contact); tangency (2 pt con-
Projective h21 h22 h23 tact); inflections
8 dof h31 h32 h33 (3 pt contact with line); tangent discontinuities
and cusps. cross ratio (ratio of ratio of lengths).
! " Parallelism, ratio of areas, ratio of lengths on
a11 a12 tx
Affine collinear or parallel lines (e.g. midpoints), lin-
a21 a22 ty
6 dof ear combinations of vectors (e.g. centroids).
0 0 1
The line at infinity, l∞ .

! "
sr11 sr12 tx
Similarity Ratio of lengths, angle. The circular points, I, J
sr21 sr22 ty
4 dof (see section 2.7.3).
0 0 1
! "
r11 r12 tx
Euclidean r21 r22 ty Length, area
3 dof 0 0 1
 2D Transformation
�D Transformations

Translation: (�D Translation of the Input, � DoF)

" #
0 0 I t
x =x+t , x̄ = > x̄
0 1
I Using homogeneous representations allows to chain/invert transformations
I Augmented vectors x̄ can always be replaced by general homogeneous ones x̃
��
 INFORMATIK

2D
�D Transformations Transformation
Transformation hierarchy: isometries
(iso=same, metric=measure)
& x' # ,ε cosθ − sin θ t x )& x #
$ ! * ' $ !
$ y ' ! = * ε sin θ cosθ t y '$ y ! ε = ±1
$1! * 0 0 1 '($% 1 !"
% " +
orientation preserving: ε =1
orientation reversing: ε = −1
& R t#
x' = H E x = $ T ! x T
R R=I
%0 1"
3DOF (1 rotation, 2 translation)
special cases: pure rotation, pure translation
Euclidean: (�D Translation + �D Rotation, � DoF)
" area
Invariants: length, angle, #
0 0R t
x = Rx + t , x̄ = > x̄ 15

0 1
I R 2 SO(2) is an orthonormal rotation matrix with RR> = I and det(R) = 1
I Euclidean transformations preserve Euclidean distances
��
�D Transformations
INFORMATIK2D transformation
Transformation hierarchy: similarities
& x' # , s cosθ − s sin θ t x )& x # (isometry + scale)
$ ! * ' $ !
$ y ' ! = * s sin θ s cosθ t y '$ y !
$1! * 0 0 $
1 (% 1 "
' !
% " +

& sR t #
x' = H S x = $ T ! x R T
R I
Similarity: (�D Translation + Scaled �D
% 0 1" Rotation, � DoF) =
" #
0 0 sR t
4DOF (1x scale,
= sRx +t ,
1 rotation, x̄ = >
2 translation) x̄
0 1
also know as equi-form (shape preserving)
I R2 SO(2) is metric
a rotation matrix
structure and s up
= structure is an arbitrary(inscale
to similarity factor
literature)
I The similarity transform preserves
Invariants: ratios angles
of length, angle,between lines
ratios of areas,
�D Transformations 2D Transformation

INFORMATIK

Transformation hierarchy: affinities

Af�ne: (�D Linear Transformation,
& x' # , a611DoF)
a12 t x )& x #
$ ! * ' $ !" #
$ y ' ! = *a21 a22 t y '$ y ! A t
0$ ! 0 !
$
x =1 Ax*+0t 0, 1x̄' 1= > x̄
% " + (% " 0 1
& A t#
I A 2 R2⇥2 is an arbitrary 2 ⇥x'2 =matrix
HA x = $ T ! x
% 0 1"
I Parallel lines remain parallel under af�ne transformations
&λ1 0# ��
A = R(θ )R(− φ )DR(φ ) D=
 INFORMATIK

2D transformation
�D Transformations Transformation hierarchy: homographies
& h11 h12 h13 # !
$ ! &A t#
H P = $ h21 h22 h23 ! = $$ ! T !!
$h ! % v v"
h
% 31 32 h33 "
Changes
homogeneous
coordinate!
&A t# T
x' = H
Perspective: (Homography, 8 DoF) P x = $vT ! x v = (v1 , v2 )
% v"
✓ ◆
0 1
x̃ = H̃ x̃ x̄ = x̃
8DOF (2 scale, 2 rotation, w̃
2 translation, 2 line at infinity)

I H̃ 2 Acts non-homogeneous over the plane

R is an arbitrary homogeneous 3 ⇥ 3 matrix (speci�ed up to scale)
3⇥3
Allows to observe
I Perspective transformations preserve straight lines vanishing points,
Invariants: cross-ratio of four points on a line ��
 2D transformation hierarchy
& h11 h12 h13 #
Projective $h h22 h23 !!
A square transforms to: 8dof $ 21
$%h31 h32 h33 !"
& a11 a12 tx #
Affine $a !
6dof $ 21 a22 ty !
$% 0 0 1 !"
& sr11 sr12 t x #
Similarity $ sr sr t !
4dof $ 21 22 y!

$% 0 0 1 !"
& r11 r12 t x #
Euclidean $r !
3dof $ 21 r22 t y!

$% 0 0 1 !"
19
 2D transformation
�D Transformations on Co-Vectors of co-vectors
Considering any perspective �D transformation

0
x̃ = H̃x̃

the transformed �D line equation is given by:

0> 0 0> > 0 > >

l̃ x̃ = l̃ H̃x̃ = (H̃ l̃ ) x̃ = l̃ x̃ = 0

Therefore, we have:
0 >
l̃ = H̃ l̃

Thus, the action of a projective transformation on a co-vector such as a �D line

or �D normal can be represented by the transposed inverse of the matrix.
 Overview of 2D transformation
Overview of �D Transformations

I Transformations form nested set of groups (closed under composition, inverse)

I Interpret as restricted 3 ⇥ 3 matrices operating on �D homogeneous coordinates
I Transformations preserve properties below (similarity: parallelism, straight lines)
��
 Overview of 3D
Overview of �D Transformations transformation

I �D transformations are de�ned analogously to �D transformations

I 3 ⇥ 4 matrices are extended with a fourth [0> 1] row for homogeneous transforms
I Transformations preserve properties below (similarity: parallelism, straight lines)
��
Estimation of Homography
Lets define the correspondence
Computed homography
 Direct Linear Transformation Method (DLT):
More is the merrier —> use more correspondence points
Dimension of Ai is 2x9
Solving homogenous eq
Cross ratio
Image rectification
Overview of the Camera calibration
“Type a quote here.”

–Johnny Appleseed
Next class homography estimation
Direct Linear Transform for Homography Estimation
Direct linear transformation
How can we estimate a homography from a set of �D correspondences?
Let X = {x̃i , x̃i }i=1 denote a set of N �D-to-�D correspondences related by x̃i = H̃x̃i .
0 N 0

As the correspondence vectors are homogeneous, they have the same direction but
differ in magnitude. Thus, the equation above can be expressed as x̃i ⇥ H̃x̃i = 0.
0

Using h̃k to denote the k’th row of H̃, this can be rewritten as a linear equation in h̃:
>

2 32 3
0 > 0 >
w̃i x̃i 0 >
ỹi x̃i h̃1
6 0 > > 0 > 76 7
4 w̃i x̃i 0 x̃i x̃i 5 4h̃2 5 =0
0
ỹi x̃i> 0 >
x̃i x̃i 0 > h̃3
| {z } | {z }
Ai h̃

The last row is linearly dependent (up to scale) on the �rst two and can be dropped.
 Direct Linear Transformation
Direct Linear Transform for Homography Estimation
Each point correspondence yields two equations. Stacking all equations into a 2N ⇥ 9
dimensional matrix A leads to the following constrained least squares problem

⇤ 2 2
h̃ = argmin kAh̃k2 + (kh̃k2 1)
h̃
> > >
= argmin h̃ A Ah̃ + (h̃ h̃ 1)
h̃

2
where we have �xed = 1 as H̃ is homogeneous (i.e., de�ned only up to scale)
kh̃k2
and the trivial solution to h̃ = 0 is not of interest. The solution to the above
optimization problem is the singular vector corresponding to the smallest singular
value of A (i.e., the last column of V when decomposing A = UDV , see > also Deep
http://citeseerx.ist.psu.edu/viewdoc/
summary?doi=10.1.1.186.4411

Learning lecture ��.�). The resulting algorithm is called Direct Linear Transformation.
��
 Panorama stitching
Application: Panorama Stitching
 What is an image?

An image is a collection of pixels

Projective geometry