Computer Vision (CS708)

Lecture02 - Image formation and OpenCV

Dr. Abdul Basit

Assistant Professor - CSIT

University of Balochistan

September 29, 2016

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 1 / 54
 Contents

The contents of this lecture are taken form

Computer Vision: Algorithms and Applications (Richard Szeliski )
Introduction to computer vision CS
Learning OpenCV Computer Vision in C++ with OpenCV library Adrian
Kaebler & Gary Bradski

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 2 / 54
 Contents

1 Computer vision techniques

2 Image formation
Pinhole camera

3 Projective geometry
2D geometry
Homogeneous coordinates
3D projection
Vanishing points and lines

4 OpenCV Installation

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 3 / 54
 Computer vision overview
Overview of vision, examples of the state of the art, preview of
projects

Figure: Computer vision overview.

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 4 / 54
 Computer vision algorithms

Figure: A pictorial summary of the computer vision algorithms. Reprinted form

Szeliski (2011).
Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 5 / 54
 Computer vision algorithms (Cond)

Figure: A pictorial summary of the computer vision algorithms. Reprinted form

Szeliski (2011).
Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 6 / 54
 Contents overview

Figure: Relationship between images, geometry, and photometry, as well as a

taxonomy of the topics covered in this book. Szeliski (2011).
Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 7 / 54
 Contents

1 Computer vision techniques

2 Image formation
Pinhole camera

3 Projective geometry
2D geometry
Homogeneous coordinates
3D projection
Vanishing points and lines

4 OpenCV Installation

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 8 / 54
 A camera

What do we need to make a camera from scratch?

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 9 / 54
 Image formation
This lecture

Mapping between image and world coordinates

Pinhole camera model
Projective geometry
Vanishing points and lines
Projection matrix

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 10 / 54
 Image formation
Camera and world geometry

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 11 / 54
 Camera and world geometry

How tall is the woman?

How high the camera is?
What is the camera rotation?
What is the focal length of the camera?
Which ball is closer?

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 12 / 54
 Image formation
Let’s design a camera
Idea 1: put a piece of film in front of an object
Do we get a reasonable image?

Figure: Image formation understanding.

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 13 / 54
 Image formation
Idea 2: add a barrier to block off most of the rays
This reduce blurring
The opening known as the aperture

Figure: Image formation understanding.

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 14 / 54
 Contents

1 Computer vision techniques

2 Image formation
Pinhole camera

3 Projective geometry
2D geometry
Homogeneous coordinates
3D projection
Vanishing points and lines

4 OpenCV Installation

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 14 / 54
 Image formation
Pinhole camera

Figure: A pinhole camera.

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 15 / 54
 Image formation
Dimensionality Reduction Machine (3D to 2D)

Figure: A pinhole camera.

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 16 / 54
 Projective geometry
What is lost?

What is lost?
Length
Which one is closer?

Dr. Abdul Basit (CSIT - UoB) Figure: What weVision

Computer lost after projection. September 29, 2016 17 / 54
 Projective geometry
Length cannot be preserved?

Figure: Lenght is lost.

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 18 / 54
 Projective geometry
Length cannot be preserved?

What is lost?
Length
Angles

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 19 / 54
 Projective geometry
What is preserved?

What is preserved?
Straight lines are still straight

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 20 / 54
 Contents

1 Computer vision techniques

2 Image formation
Pinhole camera

3 Projective geometry
2D geometry
Homogeneous coordinates
3D projection
Vanishing points and lines

4 OpenCV Installation

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 21 / 54
 Contents

1 Computer vision techniques

2 Image formation
Pinhole camera

3 Projective geometry
2D geometry
Homogeneous coordinates
3D projection
Vanishing points and lines

4 OpenCV Installation

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 21 / 54
 2D Modeling Transformation
2D Translation

Translation is a simple straight line movement of the object

Figure: 2D translation.

old coordinates are (x, y) and the new coordinates are (x’, y’)
0
x = x + Tx
0
y = y + Ty

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 22 / 54
 2D Modeling Transformation
2D Translation

Consider the following equation

0
x =x +3
0
y =y +1

This can be rewritten with matrices

 0    
x x 3
0 = +
y y 1

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 23 / 54
 2D Modeling Transformation
2D scaling

Scaling alters the size of the object. Sx and Sy are the scaling factors:
0 0
Old coordinates are (x, y ) and the new coordinates are (x , y )

0
x = x ∗ Sx
0
y = y ∗ Sy

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 24 / 54
 2D Modeling Transformation
2D scaling

Figure: 2D scaling.

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 25 / 54
 2D Modeling Transformation
2D Rotation

Example of a 2D rotation through an angle w where the coordinates

x, y go into x’, y’.
Note that w is positive for a counterclockwise rotation and that that
rotation is about the origin (0, 0).

Figure: 2D rotation.

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 26 / 54
 2D Modeling Transformation
Formula for rotation

0 0
Old coordinates are (x, y ) and the new coordinates are (x , y )
θ = initial angle, α = angle of rotation.

x = r cos θ
y = r sin θ

0
x = r cos(θ + α) = r cos θ cos α − r sin θ sin α
0
y = r sin(θ + α) = r sin θ cos α + r cos θ sin α
hence,
0
x = x cos α − y sin α
0
y = y cos α + x sin α
Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 27 / 54
 2D Modeling Transformation
Martix representation of transformations

0 0
Scaling: x = x ∗ Sx , y = y ∗ Sy , which can be written as:
 T  
0 0 x S 0
× x
 
x y =
y 0 Sy
Rotation:
0
x = x cos α − y sin α
0
y = y cos α + x sin α

which can be written as:

 T  
 0 0  x cos α sin α
x y = ×
y − sin α cos α

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 28 / 54
 2D Modeling Transformation
Martix representation of transformations

So if we wanted to scale and then rotate the object we could do the

following:

 T  
0 0 x S 0
× x
 
x y =
y 0 Sy
 0 T  
 00 00  x cos α sin α
x y = 0 ×
y − sin α cos α

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 29 / 54
 2D Modeling Transformation
Matrix representation of transformations

But this is the same as:

 T    
 00 00  x Sx 0 cos α sin α
x y = × ×
y 0 Sy − sin α cos α
 T  
x Sx cos α Sx sin α
= ×
y −Sy sin α Sy cos α

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 30 / 54
 2D Transformations
Matrix representation of transformations

What about translation?

0
x = x ∗ Tx
0
y = y ∗ Ty

There is no way to express this as a multiplicative transformation with

a 2D matrix.
Therefore we go into hyperspace (N + 1 dimensions) , i.e., we embed
2D space into 3D space.
We express all coordinates as (xh, yh, z). This is called a
”Homogeneous” coordinate system.

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 31 / 54
 Contents

1 Computer vision techniques

2 Image formation
Pinhole camera

3 Projective geometry
2D geometry
Homogeneous coordinates
3D projection
Vanishing points and lines

4 OpenCV Installation

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 31 / 54
 2D transformations
Homogeneous coordinates

Translation (represented by T = (Tx , Ty )):

 T  
 0 x 1 0 0
0 
x y 1 = y  ×  0 1 0
1 Tx Ty 1
which gives
0
x = x + 1 ∗ Tx
0
y = y + 1 ∗ Ty
1=1

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 32 / 54
 2D transformations
Homogeneous coordinates

Scaling (represented by S = (Sx , Sy ))

 T  
 0 x Sx 0 0
0 
x y 1 = y  ×  0 Sy 0
1 0x 0 1
which gives
0
x = x ∗ Sx
0
y = y ∗ Sy

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 33 / 54
 2D transformations
Homogeneous coordinates

Rotation (represented by R(q):

 T  
 0 x cos q sin q 0
0 
x y 1 = y  × − sin q cos q 0
1 0 0 1
which gives
0
x = x cos q − y sin q
0
y = y cos q + x sin q

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 34 / 54
 Homogeneous coordinates
Projection world to image coordinates

Figure: Projection from world to image coordinates.

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 35 / 54
 Homogeneous coordinates

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 36 / 54
 Homogeneous coordinates
Scaling

Invariant to scaling

Point in Cartesian is ray in Homogeneous

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 37 / 54
 Homogeneous coordinates
Projection Matrix

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 38 / 54
 Contents

1 Computer vision techniques

2 Image formation
Pinhole camera

3 Projective geometry
2D geometry
Homogeneous coordinates
3D projection
Vanishing points and lines

4 OpenCV Installation

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 38 / 54
 Projection Matrix

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 39 / 54
 Projection Matrix
Remove assumption: known optical center

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 40 / 54
 Projection Matrix
Remove assumption: square pixels

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 41 / 54
 Homogeneous coordinates
Oriented and translated camera

Figure: Oriented and translated camera.

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 42 / 54
 Projection Matrix
Allow camera translation

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 43 / 54
 Projection Matrix
3D rotation of points

Rotation around the coordinate axes, counter-clockwise:

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 44 / 54
 Projection Matrix
Allow camera rotation

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 45 / 54
 Projection
Degrees of freedom

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 46 / 54
 Projection
Field of View (Zoom, focal length)

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 47 / 54
 Projection Matrix
Beyond Pinholes: Radial Distortion

Figure: Radial distortion. Reprinted from

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 48 / 54
 Contents

1 Computer vision techniques

2 Image formation
Pinhole camera

3 Projective geometry
2D geometry
Homogeneous coordinates
3D projection
Vanishing points and lines

4 OpenCV Installation

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 48 / 54
 Projective geometry
Vanishing points and lines

Parallel lines in the world intersect in the image at a “vanishing point”

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 49 / 54
 Projective geometry
Vanishing points and lines

Figure: Vanishing points and lines.

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 50 / 54
 Projective geometry
Vanishing points and lines

Figure: Vanishing points and lines.

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 51 / 54
 Things to remember

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 52 / 54
 Contents

1 Computer vision techniques

2 Image formation
Pinhole camera

3 Projective geometry
2D geometry
Homogeneous coordinates
3D projection
Vanishing points and lines

4 OpenCV Installation

Dr. Abdul Basit (CSIT - UoB) Computer Vision September 29, 2016 53 / 54
The End

59 / 59