Image Transformation

Instructor: Mohsen Ali

Full Disclosure

Szeliski Sec 2.1.2

Alexei Efros, UC Berkeley
Dr. Sohaib / Aamer Zaheer (LUMS)
Some slides from Steve Seitz

Whats on the menu today?

Spatial and Geometric Transformation
What Can we do with it?
Registration anyone?

Parametric (global) transformations

Translation, Scaling, Rotation, Shear

Warping
inverse of Transformation
Interpolation
Linear interpolation
Bilinear interpolation

Some more in depth look

Image Transformations
image filtering: change range of image
g(x) = T(f(x))
f

T
x

image warping: change domain of image

g(x) = f(T(x))
f

T
x

x
Source: Efros

Image Transformations
image filtering: change range of image
g(x) = T(f(x))
f

image warping: change domain of image

g(x) = f(T(x))
f

Source: Efros

Parametric (global) warping

Examples of parametric warps:

translation

affine

rotation

perspective

aspect

cylindrical
Source: Efros

2D Transformation
Definition: A mapping from one 2D coordinate
system to another
Also called
spatial transformation,
geometric transformation,
warp

Image Registration: Process of transforming

two images so that same features overlap

Source: Dr. Sohaib / Aamer Zaheer

Example Application: Image Registration

Reference
Image

Source: Dr. Sohaib / Aamer Zaheer

Mission
Images

Source: Dr. Sohaib / Aamer Zaheer

Registration = Computing Transformation

Source: Dr. Sohaib / Aamer Zaheer

Source: Dr. Sohaib / Aamer Zaheer

Source: Dr. Sohaib / Aamer Zaheer

Source: Dr. Sohaib / Aamer Zaheer

Panoramas
Multiple Images Stitched Together

Applications
of 2D Image
Registration

Source: Dr. Sohaib / Aamer Zaheer

Panoramas
Multiple Images Stitched Together

Applications
of 2D Image
Registration

Image by Sergey Semenov (http://www.sergesemenov.com/) - Winner of Epson International Photographic Pano Award 2013
http://www.dailymail.co.uk/sciencetech/article-2260276/New-York-youve-seen-Incredible-interactive-panorama-lets-zoom.html

TRANSFORMATIONS

Parametric (global) warping

T
p = (x,y)

p = (x,y)

Transformation T is a coordinate-changing machine:

p = T(p)
What does it mean that T is global?
Is the same for any point p
can be described by just a few numbers (parameters)

Lets represent a linear T as a matrix:

p = Mp

x'
x
y ' = M y

Source: Efros

Scaling
Scaling a coordinate means multiplying each of its components by
a scalar
Uniform scaling means this scalar is the same for all components:

Source: Efros

Scaling
Non-uniform scaling: different scalars per component:

X 2,
Y 0.5

Source: Efros

Scaling

Scaling operation:

x ' = ax
y ' = by

Or, in matrix form:

x ' a 0 x
y ' = 0 b y


scaling matrix S
Whats inverse of S?

Source: Efros

2-D Rotation

(x, y)
(x, y)

x = x cos(q) - y sin(q)
y = x sin(q) + y cos(q)

Source: Efros

2-D Rotation

(x, y)
(x, y)

x = r cos (f)
y = r sin (f)
x = r cos (f + q)
y = r sin (f + q)
Trig Identity
x = r cos(f) cos(q) r sin(f) sin(q)
y = r sin(f) cos(q) + r cos(f) sin(q)
Substitute
x = x cos(q) - y sin(q)
y = x sin(q) + y cos(q)
Source: Efros

2-D Rotation
This is easy to capture in matrix form:

x ' cosq sin q x

y ' = sin q cosq y


R
Even though sin(q) and cos(q) are nonlinear functions of q,
x is a linear combination of x and y
y is a linear combination of x and y

What is the inverse transformation?

Rotation by q
For rotation matrices

R 1 = R T

Source: Efros

2x2 Matrices
What types of transformations can be
represented with a 2x2 matrix?

2D Identity?

x' = x
y' = y

x' = 1 0 x
y ' 0 1 y

2D Scale around (0,0)?

x' = s x * x
y' = s y * y

x ' s x
y ' = 0

0 x
s y y
Source: Efros

2x2 Matrices
What types of transformations can be
represented with a 2x2 matrix?

2D Rotate around (0,0)?

x' = cos Q * x sin Q * y
y ' = sin Q * x cos Q * y

x ' cos Q sin Q x

y ' = sin Q cos Q y

2D Shear?
x ' = x shx * y
y ' = shy * x y

x ' 1
y ' = sh
y

shx x
1 y
Source: Efros

2x2 Matrices
What types of transformations can be
represented with a 2x2 matrix?

2D Mirror about Y axis?

x' = x
y' = y

x ' = 1 0 x
y ' 0 1 y

2D Mirror over (0,0)?

x' = x
y' = y

x' = 1 0 x
y ' 0 1 y
Source: Efros

2x2 Matrices
What types of transformations can be
represented with a 2x2 matrix?

2D Translation?
x' = x t x
y' = y t y

NO!

Only linear 2D transformations

can be represented with a 2x2 matrix

Source: Efros

All 2D Linear Transformations

Linear transformations are combinations of

Scale,
Rotation,
Shear, and
Mirror

x ' a b x
y ' = c d y

Properties of linear transformations:

Origin maps to origin

Lines map to lines
Parallel lines remain parallel
Ratios are preserved
Closed under composition

x' = a b e
y ' c d g

f i
h k

j x
l y
Source: Efros

Consider a different Basis

j =(0,1)

v =(vx,vy)
p

u=(ux,uy)
i =(1,0)
q=4i+3j = (4,3)

p=4u+3v

Source: Efros

Linear Transformations as Change of Basis

v =(vx,vy)

j =(0,1)

puv

pij

u=(ux,uy)
i =(1,0)
puv = (4,3)
px=4ux+3vx
py=4uy+3vy

pij = 4u+3v

ux
pij =
u y

v x 4 u x
=
v y 3 u y

vx
puv
vy

Any linear transformation is a basis!!!

Source: Efros

Whats the inverse transform?

j =(0,1)

v =(vx,vy)
pij

puv

u=(ux,uy)
i =(1,0)
pij = (5,4) = pxu + pyv

puv =

ux

u y

-1

vx

vy

5 u x
= u
4 y

puv = (px,py) = ?
-1

vx
pij
vy

How can we change from any basis to any basis?

What if the basis are orthogonal?
Source: Efros

Projection onto orthogonal basis

j =(0,1)

v =(vx,vy)
pij

puv

u=(ux,uy)
i =(1,0)
pij = (5,4)

ux
uv
p =
v y

puv = (upij, vpij)

ux

vy

5 u x
= v
4 x

uy
ij
p

vy
Source: Efros

Homogeneous Coordinates
Q: How can we represent translation as a 3x3
matrix?

x' = x t x
y' = y t y

Source: Efros

Homogeneous Coordinates
Homogeneous coordinates
represent coordinates in 2
dimensions with a 3-vector

x
x homogeneou
s coords
y y

Source: Efros

Homogeneous Coordinates
Add a 3rd coordinate to every 2D point
(x, y, w) represents a point at location (x/w, y/w)
(x, y, 0) represents a point at infinity
(0, 0, 0) is not allowed
y
2

(2,1,1) or (4,2,2) or (6,3,3)

Convenient
coordinate system to
represent many
useful
transformations

Source: Efros

Homogeneous Coordinates
Q: How can we represent translation as a 3x3
matrix?

x' = x t x
y' = y t y
A: Using the rightmost column:

1 0 t x

Translation = 0 1 t y
0 0 1

Source: Efros

Translation
Example of translation

Homogeneous Coordinates
x ' 1 0 t x x x t x
y ' = 0 1 t y = y t
y
y

1 0 0 1 1 1

tx = 2
ty = 1
Source: Efros

Basic 2D Transformations
Basic 2D transformations as 3x3 matrices

x ' s x
y ' = 0

1 0

x ' 1 0 t x x
y ' = 0 1 t y
y

1 0 0 1 1

Translate

x' cos Q
y ' = sin Q

1 0

sin Q
cos Q
0

Rotate

0
sy
0

0 x
0 y
1 1

Scale
0 x
0 y
1 1

x ' 1
y ' = sh
y
1 0

shx
1
0

0 x
0 y
1 1

Shear
Source: Efros

Matrix Composition
Transformations can be combined by
matrix multiplication

x' 1 0 tx cos Q sin Q 0 sx 0 0 x

y ' = 0 1 ty sin Q cos Q 0 0 sy 0 y

w' 0 0 1 0

0
1
0 0 1 w

p =

T(tx,ty)

R(Q)

S(sx,sy)

Source: Efros

Affine Transformations
x' a b
Affine transformations are combinations of
y' = d e
Linear transformations, and
w 0 0

Translations

c x
f y
1 w

Properties of affine transformations:

Origin does not necessarily map to origin

Lines map to lines
Parallel lines remain parallel
Ratios are preserved
Closed under composition
Models change of basis

Will the last coordinate w always be 1?

Source: Efros

Projective Transformations
Projective transformations
Affine transformations, and
Projective warps

x' a
y' = d
w' g

b
e
h

c x
f y
i w

Properties of projective transformations:

Origin does not necessarily map to origin

Lines map to lines
Parallel lines do not necessarily remain parallel
Ratios are not preserved
Closed under composition
Models change of basis

Source: Efros

2D image transformations

These transformations are a nested set of groups

Closed under composition and inverse is a member
Source: Efros

Hierarchy of 2D Transformations

Length Ratios
Length Ratios
Length Cross-Ratios

2D Affine Warping

Source: Dr. Sohaib/ Aamer Zaheer

Warping
Inputs:
Image X
Affine Transformation A = [a1 a2 b1 a3 a4 b2]T

Output:
Generate X such that X = AX

Obvious Process:
For each pixel in X
Apply transformation
At that location in X, put the same color as at the original
location in X

Problems?

Source: Dr. Sohaib/ Aamer Zaheer

Warping
This will leave holes
Because every pixel does not map to an integer
location!

Reverse Transformation
For each integer location in X
Apply inverse mapping
Problem?

Will not result in answers at integer locations, in

general
Bilinearly interpolate from 4 neighbors

Source: Dr. Sohaib/ Aamer Zaheer

Interpolation
In 1D

3
Use y=mx+c

m = 1, c = -2

Substitute
x=4.3, => y = 2.3

4
4.3 ?

Source: Dr. Sohaib/ Aamer Zaheer

2D Bilinear Interpolation
Four nearest points of
(x, y)

x, y

x, y

x, y , x, y , x, y , x, y

x, y

where

x = int( x)
y = int( y )

x, y

x, y

x = x 1
y = y 1
Source: Dr. Sohaib/ Aamer Zaheer

Bilinear Interpolation
f ' ( x , y ) = x y f ( x, y ) x y f ( x , y ) x y f ( x , y ) x y f ( x, y )

x = x x
y = y y

x, y

x, y

x = x x

y = y y
y

x, y

x, y
Source: Dr. Sohaib/ Aamer Zaheer

A LITTLE MORE IN DEPTH

COVERAGE

Groups
A group is a set G together with an operation that
combines two elements of a and b to form another
element a b. To form a group, (G, ) must satisfy
the following axioms
Closure
For all a, b in G, a b is also in G

Associativity
For all a, b, c in G, (a b) c = a (b c)

Identity
There exists an element e in G s.t. a e = e a = a

Inverse
For each a in G, there exists an element b in G s.t. a b = e

Hierarchy of Transformation Groups

Translation

Rigid Body Transformation

Similarity

Each higher group completely contains the lower group

Affine Group
General 2 x 3 linear transform

Contains rotation, scaling, shear,

translation and any
combination thereof
Preserves Parallel lines [Proof?]

Ref: Steve Mann & Rosalind W. Picard, Video Orbits of the

Projective Group: A simple approach to featureless estimation
of parameters, IEEE Trans. on Image Processing, Vol. 6, No.
9, September 1997

Order of Transformations
Rotation/Scaling/Shear, followed by Translation

1 0 b1 a1
0 1 b a
2 3

0 0 1 0

a2
a4
0

0 a1
0 = a3
1 0

a2
a4
0

b1
b2
1

Translation, followed by Rotation/Scaling/Shear

a1
a
3
0

a2
a4
0

0 1 0 b1 a1
0 0 1 b2 = a3
1 0 0 1 0

a2
a4
0

a1b1 a2b2
a3b1 a4b2

Projective Group (Homography)

3 x 3 transform defined in
Homogeneous coordinates

Simulates out of plane rotations

Preserves straight lines
Physical Interpretation:
Plane + Camera

Examples of Projective Transformations

http://wearcam.org/orbits/gallery.html

Hierarchy of 2D Transformations

Length Ratios
Length Ratios
Length Cross-Ratios

Pseudo Perspective Model

Linear approximation of projective model

Small motion approximation of full homography

Does not preserve straight lines
Obtained through Taylor Expansion

Bilinear Model
Linear approximation of projective
model

8 parameters, so perfectly maps four

points
Does not preserve straight lines
Preserves relative distances (nonchirping model
Obtained through Taylor Expansion

Displacement Models

Ref: Steve Mann & Rosalind W. Picard, Video Orbits of

the Projective Group: A simple approach to featureless
estimation of parameters, IEEE Trans. on Image
Processing, Vol. 6, No. 9, September 1997