Lecture 9:

2D Transformation & Alignment

Instructor: Soumyadip (Roni) Sengupta

ULA: Andrea Dunn, William Li, Liujie Zheng
Course Website:
Scan Me!
Steps of creating a Panorama

This is your next homework assignment!

Why extract features? (last week)
• Motivation: panorama stitching
• We have two images – how do we combine them?
Why extract features? (last week)
• Motivation: panorama stitching
• We have two images – how do we combine them?

Step 1: extract features

Step 2: match features
How to align and combine two images?
• Motivation: panorama stitching
• We have two images – how do we combine them?

Step 1: extract features Last Week

Step 2: match features
Step 3: align images
Step 4: blending images This Week
What is the geometric relationship between these
two images?

?
What is the geometric relationship between these
two images?

Very important for creating mosaics!

First, we need to know what this transformation is.
Second, we need to figure out how to compute it using feature matches.
Image Warping
• image filtering: change range of image
• g(x) = h(f(x))
f g
h

x x

• image warping: change domain of image

• g(x) = f(h(x))
f g
h

x x
Richard Szeliski Image Stitching 8
Image Warping
• image filtering: change range of image
• g(x) = h(f(x))
f g
h

• image warping: change domain of image

• g(x) = f(h(x))
f g
h

Richard Szeliski Image Stitching 9

Parametric (global) warping
• Examples of parametric warps:

translation rotation aspect

Richard Szeliski Image Stitching 10

Parametric (global) warping
T

p = (x,y) p’ = (x’,y’)

• Transformation T is a coordinate-changing machine:

p’ = T(p)
• What does it mean that T is global?
• Is the same for any point p
• can be described by just a few numbers (parameters)
• Let’s consider linear transforms (can be represented by a 2x2 matrix):
Today’s class
• Types of 2D Transformations
• Linear
• Affine
• Perspective (Homography)

• Computing 2D Transformations
• Linear Least Squares
• Affine
• Perspective (Homography)
Today’s class
• Types of 2D Transformations
• Linear
• Affine
• Perspective (Homography)

• Computing 2D Transformations
• Linear Least Squares
• Affine
• Perspective (Homography)
Common linear transformations
• Rotation by angle θ (about the origin)

θ
(0,0) (0,0)

What is the inverse?

For rotations:
2-D Rotation

x = r cos (f)
y = r sin (f)
x’ = r cos (f + q)
y’ = r sin (f + q)
(x’, y’)
(x, y)

q f
2-D Rotation

x = r cos (f)
y = r sin (f)
x’ = r cos (f + q)
y’ = r sin (f + q)
(x’, y’)
Trig Identity…
(x, y) x’ = r cos(f) cos(q) – r sin(f) sin(q)
y’ = r sin(f) cos(q) + r cos(f) sin(q)

q f
2-D Rotation

x = r cos (f)
y = r sin (f)
x’ = r cos (f + q)
y’ = r sin (f + q)
(x’, y’)
Trig Identity…
(x, y) x’ = r cos(f) cos(q) – r sin(f) sin(q)
y’ = r sin(f) cos(q) + r cos(f) sin(q)

q f
Substitute…
x’ = x cos(q) - y sin(q)
y’ = x sin(q) + y cos(q)
2-D Rotation
•This is easy to capture in matrix form:

é x ' ù écos(q ) - sin (q )ù é x ù

ê y 'ú = ê sin (q ) cos(q ) ú ê y ú
ë û ë ûë û

θ
(0,0) (0,0)
 2x2 Matrices

• What types of transformations can be

represented with a 2x2 matrix?
2D Identity?
x' = x é x'ù = é1 0ù é x ù
y' = y êë y 'úû êë0 1úû êë y úû

2D Scale around (0,0)?

x' = s x * x é x 'ù é s x 0 ùé xù
y' = s y * y ê y 'ú = ê 0 s y úû êë y úû
ë û ë
 2x2 Matrices

• What types of transformations can be

represented with a 2x2 matrix?
2D Rotate around (0,0)?
x' = cos Q * x - sin Q * y é x 'ù écos Q - sin Qù é x ù
y ' = sin Q * x + cos Q * y ê y 'ú = ê sin Q cos Q ú ê y ú
ë û ë ûë û

2D Shear?
x ' = x + shx * y é x 'ù é 1 shx ù é x ù
y ' = shy * x + y ê y 'ú = ê sh 1 úû êë y úû
ë û ë y
 2x2 Matrices

• What types of transformations can be

represented with a 2x2 matrix?
2D Mirror about Y axis?
x' = - x é x ' ù = é - 1 0ù é x ù
y' = y êë y 'úû êë 0 1úû êë y úû

2D Mirror over (0,0)?

x' = - x é x'ù = é- 1 0 ù é x ù
y' = - y êë y 'úû êë 0 - 1úû êë y úû
2x2 Matrices
• What types of transformations can be
represented with a 2x2 matrix?
2D Translation?
NO!

Translation is not a linear operation on 2D coordinates

Only linear 2D transformations

can be represented with a 2x2 matrix
All 2D Linear Transformations
• Linear transformations are combinations of …
• Scale,
• Rotation, é x'ù éa b ù é x ù
• Shear, and ê y 'ú = ê c d ú ê y ú
ë û ë ûë û
• Mirror
• Properties of linear transformations:
• Origin maps to origin
• Lines map to lines
• Parallel lines remain parallel
• Ratios are preserved
• Closed under composition
é x'ù = éa b ù é e f ùé i jùé xù
êë y 'úû êë c d úû êë g h úû êëk l úû êë y úû
Today’s class
• Types of 2D Transformations
• Linear
• Affine
• Perspective (Homography)

• Computing 2D Transformations
• Linear Least Squares
• Affine
• Perspective (Homography)
 Homogeneous Coordinates

• Q: How can we represent translation as a 3x3 matrix?

x' = x + t x
y' = y + t y

•Homogeneous coordinates

é xù
• represent coordinates in 2
dimensions with a 3-vector

é x ù homogeneous coords ê ú
ê y ú ¾¾ ¾ ¾ ¾¾®ê y ú
ë û êë 1 úû
Homogeneous coordinates (x, y, w)
w

Trick: add one more coordinate: homogeneous plane

(x/w, y/w, 1)
w=1
x
homogeneous image
coordinates
y

Converting from homogeneous coordinates

 Homogeneous Coordinates
• Add a 3rd coordinate to every 2D point
• (x, y, w) represents a point at location (x/w, y/w)
• (x, y, 0) represents a point at infinity
• (0, 0, 0) is not allowed

y
2
(2,1,1) or (4,2,2) or (6,3,3)
1

Convenient 1 2 x
coordinate system to
represent many
useful
transformations
 Homogeneous Coordinates

• Q: How can we represent translation as a 3x3 matrix?

x' = x + t x
y' = y + t y
é1 0 t x ù
ê ú
• A: Using the rightmost column: Translation = ê0 1 t y ú
ê0 0 1 ú
ë û
Affine transformations

any transformation represented by

a 3x3 matrix with last row [ 0 0 1 ]
we call an affine transformation
Basic affine transformations

é x ' ù é1 0 t x ù é x ù é x 'ù é s x 0 0ù é x ù
ê y ' ú = ê0 1 t ú ê y ú ê y 'ú = ê 0 sy 0úú êê y úú
ê ú ê y úê ú ê ú ê
êë 1 úû êë0 0 1 úû êë 1 úû êë 1 úû êë 0 0 1úû êë 1 úû
Translate Scale

é x'ù écosq - sin q 0ù é x ù é x 'ù é 1 shx 0ù é x ù

ê y 'ú = ê sin q cosq 0ú ê y ú ê y 'ú = ê sh
ê ú ê úê ú ê ú ê y 1 0úú êê y úú
êë 1 úû êë 0 0 1úû êë 1 úû êë 1 úû êë 0 0 1úû êë 1 úû
2D in-plane rotation Shear
 Matrix Composition

• Transformations can be combined by

matrix multiplication

é x' ù æ é1 0 tx ù écos Q - sin Q 0ù é sx 0 0ù ö é x ù

ê y ' ú = ç ê0 1 ty ú ê sin Q cos Q 0ú ê 0 sy 0ú ÷ ê y ú
êë w'úû çç êë0 0 1 úû êë 0 0 1 ú ê ú ÷÷ ê ú
û ë 0 0 1 û ø ë wû
è
p’ = T(tx,ty) R(Q) S(sx,sy) p

Does the order of multiplication matter?

Affine transformations
• Affine transformations are combinations of …
• Linear transformations, and
é x'ù é a b c ùé x ù
• Translations ê y 'ú = êd e f úê y ú
ê wú ê 0 0 1 úû êë wúû
ë û ë
• Properties of affine transformations:
• Origin does not necessarily map to origin
• Lines map to lines
• Parallel lines remain parallel
• Ratios are preserved
• Closed under composition
Today’s class
• Types of 2D Transformations
• Linear
• Affine
• Perspective (Homography)

• Computing 2D Transformations
• Linear Least Squares
• Affine
• Perspective (Homography)
Where do we go from here?

what happens when we

mess with this row?
affine transformation
 Projective Transformations aka Homographies aka Planar
Perspective Maps

Called a homography
(or planar perspective map)
Any two images of the same planar surface in 3D space are
related by a homography (assuming a pinhole camera model).
Homographies

What happens when the

denominator is 0?
Points at infinity
(x,y,0) (x’,y’,1)
Image warping with homographies

image plane in front image plane below

black area
where no pixel
maps to
Homographies (Projective Transformation)

• Homographies …
• Affine transformations, and
• Projective warps

where the length of the

• Properties of projective transformations: vector [h00 h01 … h22] is 1.
• Origin does not necessarily map to origin
• Lines map to lines
• Parallel lines do not necessarily remain parallel
• Ratios are not preserved
• Closed under composition
2D image transformations
Today’s class
• Types of 2D Transformations
• Linear
• Affine
• Perspective (Homography)

• Computing 2D Transformations
• Linear Least Squares
• Affine
• Perspective (Homography)
 Computing transformations
• Given a set of matches between images A and B
• How can we compute the transform T from A to B?

• Find transform T that best “agrees” with the matches

Today’s class
• Types of 2D Transformations
• Linear
• Affine
• Perspective (Homography)

• Computing 2D Transformations
• Linear Least Squares
• Affine
• Perspective (Homography)
Simple case: translations

How do we solve for

?
Simple case: translations

Displacement of match i =

Mean displacement =
Another view

• System of linear equations

• What are the knowns? Unknowns?
• How many unknowns? How many matches do we need?
Another view

• Problem: more equations than unknowns

• “Overdetermined” system of equations
• We will find the least squares solution
Least squares: linear regression
12

10

8
(yi, xi)

Mileage
6

4 y = mx + b

0
0 1 2 3 4 5 6
Time
Linear regression
12

10

residual error

Mileage
6

0
0 1 2 3 4 5 6
Time

Calculate partial derivatives

w.r.t. m and b and set them to 0.
Linear regression
Least squares

• Find t that minimizes

• To solve, form the normal equations

Check proof here

Another view

• Problem: more equations than unknowns

• “Overdetermined” system of equations
• We will find the least squares solution
Least squares formulation
• For each point

• we define the residuals as

Least squares formulation
• Goal: minimize sum of squared residuals

• “Least squares” solution

• For translations, is equal to mean (average)
displacement
Least squares formulation
• Can also write as a matrix equation

2n x 2 2x1 2n x 1
Today’s class
• Types of 2D Transformations
• Linear
• Affine
• Perspective (Homography)

• Computing 2D Transformations
• Linear
• Affine
• Perspective (Homography)
Affine transformations

• How many unknowns?

• How many equations per match?
• How many matches do we need?
Affine transformations
• Residuals:

• Cost function:

Calculate partial derivatives w.r.t. (a,b,c,d,e,f) and set to 0.

Affine transformations
• Matrix form

2n x 6 6x1 2n x 1
Today’s class
• Types of 2D Transformations
• Linear
• Affine
• Perspective (Homography)

• Computing 2D Transformations
• Linear
• Affine
• Perspective (Homography)
Homographies

p’
p

To unwarp (rectify) an image

• solve for homography H given p and p’
• solve equations of the form: wp’ = Hp
– linear in unknowns: w and coefficients of H
– H is defined up to an arbitrary scale factor
– how many matches are necessary to solve for H?
Solving for homographies

Not linear!
Solving for homographies
Solving for homographies

2n × 9 9 2n

Defines a least squares problem:

• Since h is only defined up to scale, solve for unit vector
• Rank(A) = ?
 Solving for homographies
Rank(A) = 8
2n × 9 9 2n

• Calculate Singular Value decomposition of A -> A= UDVT

• U is 2n x r ; D is r x r (diagonal matrix with singular values) ; V is 9 x r, where r=rank(A).
• In ideal case r=rank(A)=8, h is in null-space of A.
• In practice rank(A)=9, thus the goal is to find the smallest singular value of A.
• Smallest singular value of A also indicates how well the homography can be estimated.

• Calculate ATA = VDUTUDVT = V D2 V -1 (Since, UTU = VTV =I)

• This is eigen-decomposition of ATA
• Smallest singular value of A -> Smallest eigen value of ATA.
• Solution: optimal h = eigenvector of ATA with smallest eigenvalue.
Computing transformations
Image alignment algorithm
Given images A and B

1. Compute image features for A and B

2. Match features between A and B
3. Compute homography between A and B using least squares on set
of matches

What could go wrong?

Outliers outliers

inliers

Lead to next class on RANSAC

Slide Credits
• CS5670, Introduction to Computer Vision, Cornell Tech, by Noah
Snavely.
• CS 194-26/294-26: Intro to Computer Vision and Computational
Photography, UC Berkeley, by Alyosha Efros.
• Fall 2022 CS 543/ECE 549: Computer Vision, UIUC, by Svetlana
Lazebnik.