GEO1016

Photogrammetry and 3D Computer Vision

Lecture
Camera Calibration

Liangliang Nan
Today’s Agenda
c

• Review: Camera models

• Camera calibration
• A1: Camera calibration

2
Images
c

A color image: R, G, B channels

“vector-valued” function

3
Pinhole camera model
c

4
Pinhole camera model
c

• 3D point P = (X, Y, Z)T projected to 2D image p = (x, y)T

camera coordinate
system

𝑋 𝑌
𝑥=𝑓 , y=𝑓
𝑍 𝑍

5
Perspective projection model
c
𝑋 𝑌
• Pinhole camera 𝑥=𝑓 ,
𝑍
y=𝑓
𝑍
• Change of unit: physical measurements -> pixels
– If k = l, camera sensor’s pixels are exactly square

𝑋 𝑌
𝑥 = 𝑘𝑓 , y = 𝑙𝑓
𝑍 𝑍
Denote 𝛼 = 𝑘𝑓, 𝛽 = 𝑙𝑓

𝑋 𝑌
𝑥=𝛼 , y=𝛽 - x, y: image coordinates (pixels)
𝑍 𝑍
- k, l: scale parameters (pixels/mm)
- f : focal length (mm) 6
Perspective projection model
c
𝑋 𝑌
• Pinhole camera 𝑥=𝑓 ,
𝑍
y=𝑓
𝑍 𝑋 𝑌
• Change of unit: physical measurements -> pixels 𝑥=𝛼 ,
𝑍
y=𝛽
𝑍
• Change of coordinate system
– Image plane coordinates have origin at image center
– Digital image coordinates have origin at top-left corner

7
Perspective projection model
c
𝑋 𝑌
• Pinhole camera 𝑥=𝑓 ,
𝑍
y=𝑓
𝑍 𝑋 𝑌
• Change of unit: physical measurements -> pixels 𝑥=𝛼 ,
𝑍
y=𝛽
𝑍
• Change of coordinate system
• Account for skewness
– Image frame may not be exactly rectangular due to sensor manufacturing errors

θ: skew angle between x- and y-axis

8
Perspective projection model
c

• Rewrite in matrix-vector product form

𝐏 = 𝑋, 𝑌, 𝑍 T , 𝐩 = 𝑥, 𝑦, 1 T

(homogeneous coordinates)

𝛼 −𝛼 cot 𝜃 𝑐𝑥 𝑓𝑥 𝑠 𝑐𝑥
!
𝐩 = 𝐾𝐏, 𝐾 = 0 𝑐𝑦 = 0 𝑓𝑦 𝑐𝑦
"#$ %
0 0 1 0 0 1
Intrinsic parameter matrix 9
Perspective projection model
c

• Camera motion
– World frame may not align with the camera frame
– Camera can move and rotate
Camera frame
&
𝐏 & = 𝑅' 𝐏 ' + 𝐭 &'
World frame
1 3 2 4

1. Coordinates of 3D scene point in camera frame.

2. Coordinates of 3D scene point in world frame.
3. Rotation matrix of world frame in camera frame.
4. Position of world frame’s origin in camera frame.

10
Perspective projection model
c

• The complete transformation

Internal (intrinsic) parameters

External (extrinsic) parameters

- R: rotation matrix of the world coordinate system defined in the camera coordinate system
- t: the position of world coordinate system’s origin in camera coordinate system
(Note: t is often mistakenly interpreted as the position of the camera position in the world coordinate system) 11
Today’s Agenda
c

• Review: Camera models

• Camera calibration
• A1: Camera calibration

12
General Idea
c

• Why is camera calibration necessary?

– Given 3D scene, knowing the precise 3D to 2D projection requires
• Intrinsic and extrinsic parameters
– Reconstructing 3D geometry from images also requires these
parameters

Internal (intrinsic) parameters

External (extrinsic) parameters 13

General Idea
c

• Why is camera calibration necessary?

• What information do we have?
– Images only

14
General Idea
c

• Why is camera calibration necessary?

• What information do we have?
• Camera calibration
– Recovering K
– Recovering R and t
• How many parameters

Internal (intrinsic) parameters

External (extrinsic) parameters

15
General Idea
c

• How many parameters to recover?

– How many intrinsic parameters?

𝑓𝑥 𝑠 𝑐𝑥
𝐾= 0 𝑓𝑦 𝑐𝑦 Internal (intrinsic) parameters
0 0 1
16
General Idea
c

• How many parameters to recover?

– How many intrinsic parameters?
– How many extrinsic parameters?

External (extrinsic) parameters

17
General Idea
c

• How many parameters to recover: 11

– 5 intrinsic parameters
• 2 for focal lengths
• 2 for offset (image center, or principal point)
• 1 for skewness
– 6 extrinsic parameters
• 3 for rotation
• 3 for translation 𝑓𝑥 𝑠 𝑐𝑥
𝐾= 0 𝑓𝑦 𝑐𝑦 ,
0 0 1
18
General Idea
c

• What information to use?

– Corresponding 3D-2D point pairs

19
General Idea
c

• What information to use?

– Corresponding 3D-2D point pairs
• How many pairs do we need?

20
General Idea
c

• What information to use?

– Corresponding 3D-2D point pairs
• How many pairs do we need?
– How much information does each pair of corresponding point provide?

21
m1, m2, m3: the three rows of the projection matrix M
General Idea
c

• What information to use?

– Corresponding 3D-2D point pairs
• How many pairs do we need?
– Each 3D-2D point pair -> 2 equations
– 11 unknown -> 6 point correspondence
– Use more to handle noisy data

22
m1, m2, m3: the three rows of the projection matrix M
General Idea
c

12 x 1

2n x 12

Constraints from one pair Equations from n pairs

What is the dimension of the P matrix?

What is the dimension of m?
23
Details: the derivation of the linear system
c

• The equations

24
Details: the derivation of the linear system
c

• The equations

25
Details: the derivation of the linear system
c

• The equations
For every pair of 3D-2D corresponding points

Given n pairs of 3D-2D corresponding points

26
Details: the derivation of the linear system
c

• The equations

Simplified notation

27
General Idea
c

• How to solve it?

– It is a homogeneous linear system
– It is overdetermined

28
General Idea
c

• How to solve it?

– m = 0 is always a trivial solution
– If m ≠ 0 is a solution, then any k * m is also a solution

29
General Idea
c

• How to solve it?

– m = 0 is always a trivial solution
– If m ≠ 0 is a solution, then any k * m is also a solution
– Constrained optimization

30
SVD
c

• Singular Value Decomposition

– Generalization of the eigen-decomposition of a square matrix to any
m by n matrix

𝑈: an m by m orthogonal matrix
𝐷: an m by n diagonal matrix; entries on diagonal called singular values
𝑉: an n by n orthogonal matrix
31
SVD
c

• Geometric meaning

Example (square matrix)

A U D VT
Transformation Rotation Scaling Rotation
 Calibration: solve for projection matrix
c

SVD of P

U2nx2n D2nx12 VT12x12

Last column of V gives m

(Why? See page 593 of Hartley & Zisserman. Multiple View Geometry in Computer Vision)
33
 Least-squares solution of homogeneous equations
c

Page 593 of Hartley & Zisserman. Multiple View Geometry in Computer Vision 34
Camera parameters from project matrix
c

SVD-solved projection matrix

SVD-solved projection matrix is known up to scale, i.e., The true values of project matrix

35
Camera parameters from project matrix
c

Solving for the intrinsic and extrinsic parameters

36
Camera parameters from project matrix
c

Intrinsic parameters: Extrinsic parameters:

37
Find 3D-2D corresponding points
c

• At least 6 3D-2D point pairs

– 3D points with known 3D coordinates
– Corresponding image points with known 2D coordinates

tape measure
38
Find 3D-2D corresponding points
c

• Calibration rig - a special apparatus

– P1, … Pn with known positions in [Ow, iw, jw, kw]

39
Find 3D-2D corresponding points
c

• Calibration rig - a special apparatus

– P1, … Pn with known positions in [Ow, iw, jw, kw]
– p1, … pn known positions in the image
– At least 6 pairs
image
• Goal
– Intrinsic parameters
– Extrinsic parameters

40
Calibration
c

• Always solvable?

41
 Calibration
c

• Always solvable?
– {Pi} cannot lie on the same plane
– {Pi} cannot lie on the intersection curve of two quadric surfaces

42
See Section 1.3 of Forsyth & Ponce. Computer Vision: A Modern Approach
A1: Camera calibration
c

43
Next lecture
c

• Epipolar geometry

44