Course Notes 2: Camera Calibration∗

1 Camera Calibration
To precisely know the transformation from the real, 3D world into digital images requires
prior knowledge of many of the camera’s intrinsic parameters. If given an arbitrary
camera, we may or may not have access to these parameters. We do, however, have
access to the images the camera takes. Therefore, can we find a way to deduce them
from images? This problem of estimating the extrinsic and intrinsic camera parameters
is known as camera calibration.

Figure 1: The setup of an example calibration rig.

Specifically, we do this by solving for the intrinsic camera matrix K and the extrinsic
parameters R, t from Equation 1.
! "
pi = K R t Pi
(1)
= M Pi

We can describe this problem in the context of a calibration rig, such as the one shown
in Figure 1. The rig usually consists of a simple pattern (i.e. checkerboard) with known
dimensions. Furthermore, the rig defines our world reference frame with origin OW and
axes iW , jW , kW . From the rig’s known pattern, we have known points in the world
reference frame P1 , ..., Pn . Finding these points in the image we take from the camera
gives corresponding points in the image p1 , ..., pn .
∗
Most contents are from
- R. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision (2nd Edition)
- K. Hata and S. Savarese. Course notes of Stanford CS231A

1
Course Note GEO1016: Photogrammetry and 3D Computer Vision

Denote the three rows of M as mT1 , mT2 , mT3 , i.e.,

# T%
m1
M = mT2 &
$ (2)
mT3

We set up a linear system of equations from the n correspondences such that for each
correspondence Pi , pi : # T %
Pi m 1
' ( PT
ui ) i m3 *
pi = = M Pi = $) * (3)
vi T
&
P m2 i
PT
i m3

As we see from the above equation, each correspondence gives us two equations and,
consequently, two constraints for solving the unknown parameters contained in M . Be-
fore, we know that the camera matrix has 11 unknown parameters. This means that we
need at least 6 correspondences to solve this. However, in the real world, we often use
more, as our measurements are often noisy. To explicitly see this, we can derive a pair of
equations that relate ui and vi with Pi

PTi m1 −ui (PTi m3 ) = 0

PTi m2 −vi (PTi m3 ) = 0

Given n pairs of these corresponding points, the entire linear system of equations
becomes

PT1 m1 −u1 (PT1 m3 ) = 0

PT1 m2 −v1 (PT1 m3 ) = 0
..
.
PTn m1 −un (PTn m3 ) = 0
PTn m2 −vn (PTn m3 ) = 0

This can be formatted as a matrix-vector product shown below:

# %
PT1 0T −u1 PT1
) 0T PT −v1 PT * # %
) 1 1 * m1
) .. *$ &
) * m2 = P m = 0 (4)
) T .T *
$Pn 0 −un PTn & m3
0T PTn −vn PTn

(An expanded version of this equation is given in Equation 10). When 2n > 11, this
homogeneous linear system is overdetermined. For such a system m = 0 is always a
trivial solution. Furthermore, even if there were some other m that were a nonzero
solution, then ∀k ∈ R, km is also a solution. Therefore, to constrain our solution, we
complete the following minimization:

minimize $P m$2
m
(5)
subject to $m$2 = 1

2
Course Note GEO1016: Photogrammetry and 3D Computer Vision

To solve this minimization problem, we simply use singular value decomposition. If we

let P = U ΣV T , then the solution to the above minimization is to set m equal to the last
column of V . The derivation for this solution is outside the scope of this class and you
may refer to Section 5.3 of Hartley & Zisserman1 on pages 592-593 for more details.
After reformatting the vector m into the matrix M , we#now% want to#explicitly
% solve
rT1 tx
for the extrinsic and intrinsic parameters. Denote R = $rT2 &, t = $ty &, and since
rT3 tz
# % # %
f x s u0 α −α cot θ u0
K = $ 0 f y v0 & = $ 0 β
sin θ
v0 &, we know our SVD-solved projection matrix
0 0 1 0 0 1
(denoted by M) is known up to scale, which means that the true values of the project
matrix M are some scalar multiple of M, i.e.,
# T %
αr1 − α cot θrT2 + u0 rT3 αtx − α cot θty + u0 tz
ρM = M = $ β
rT + v0 rT3
sin θ 2
β
t + v0 t z
sin θ y
&. (6)
T
r3 tz

Dividing by the scaling parameter gives

# T %
αr1 − α cot θrT2 + u0 rT3 αtx − α cot θty + u0 tz
1 1
M= M = $ β
rT + v0 rT3
sin θ 2
β
t + v0 t z
sin θ y
&. (7)
ρ ρ T
r3 tz
# T %
! " a 1 b 1
Because we have obtained all the entries of M, denote M = A b = $aT2 b2 &, solving
aT3 b3
for the intrinsics gives
1
ρ=±
$a3 $
2
u0 = ρ (a1 · a3 )
v0 = ρ2 (a2 · a3 )
(8)
(a1 × a3 ) · (a2 × a3 )
cos θ = −
$a1 × a3 $ · $a2 × a3 $
2
α = ρ $a1 × a3 $ sin θ
β = ρ2 $a2 × a3 $ sin θ
The extrinsics are
a2 × a3
r1 =
$a2 × a3 $
r2 = r3 × r1 (9)
r3 = ρa3
t = ρK −1 b
We leave the derivations as a class exercise or you can refer to Section 1.3.1 of the
Forsyth & Ponce textbook. Section 2 explains the derivation of Equation 4 using light
linear algebra.
1
R. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision (2nd Edition). Cam-
bridge University Press, 2004

3
Course Note GEO1016: Photogrammetry and 3D Computer Vision

With the calibration procedure complete, we warn against degenerate cases. Not all
sets of n correspondences will work. For example, if the points Pi lie on the same plane,
then the system will not be able to be solved. These unsolvable configurations of points
are known as degenerate configurations. More generally, degenerate configurations
have points that lie on the intersection curve of two quadric surfaces. Although this is
outside the scope of the class, you can find more information in Section 1.3 of the Forsyth
& Ponce textbook 2 .

2 [Detail] Derivation Using Light Linear Algebra

Given any pair of 3D-2D correspondence, i.e., [X, Y, Z]T → [u, v]T , we have (in homoge-
neous coordinates)
# %
# % # % X
su m11 m12 m13 m14 ) *
$ sv & = $ m21 m22 m23 m24 & ) Y *
$ Z &
s m31 m32 m33 m34
1

su = m11 X + m12 Y + m13 Z + m14

⇒ sv = m21 X + m22 Y + m23 Z + m24
s = m31 X + m32 Y + m33 Z + m34
m11 X + m12 Y + m13 Z + m14
u=
m31 X + m32 Y + m33 Z + m34
⇒
m21 X + m22 Y + m23 Z + m24
v=
m31 X + m32 Y + m33 Z + m34
(m31 X + m32 Y + m33 Z + m34 ) u = m11 X + m12 Y + m13 Z + m14
⇒
(m31 X + m32 Y + m33 Z + m34 ) v = m21 X + m22 Y + m23 Z + m24
m31 uX + m32 uY + m33 uZ + m34 u = m11 X + m12 Y + m13 Z + m14
⇒
m31 vX + m32 vY + m33 vZ + m34 v = m21 X + m22 Y + m23 Z + m24

m11 X + m12 Y + m13 Z + m14 − m31 uX − m32 uY − m33 uZ − m34 u = 0

⇒
m21 X + m22 Y + m23 Z + m24 − m31 vX − m32 vY − m33 vZ − m34 v = 0
2
D. Forsyth and J. Ponce. Computer Vision: A Modern Approach (2nd Edition). Pearson, 2012.

4
Course Note GEO1016: Photogrammetry and 3D Computer Vision

With n pairs of 3D-2D corresponding points, i.e., {[Xi , Yi , Zi ]T } → {[ui , vi ]T }, (1 ≤ i ≤

n), we have the system of linear equations as has been given in Equation 4 as:
# %
m11
) m12 *
) *
) m13 *
# %))
* # %
*
X 1 Y1 Z 1 1 0 0 0 0 −u1 X1 −u1 Y1 −u1 Z1 −u1 ) m14 * 0
) 0 * ) m21 * ) *
) 0 0 0 X1 Y1 Z1 1 −v1 X1 −v1 Y1 −v1 Z1 −v1 * ) * ) 0 *
) .. *)) m22 * *=) . *
) . *) * ) .. *
) * m23 * ) *
$ X n Yn Z n 1 0 0 0 0 −un Xn −un Yn −un Zn −un & ) ) m24 * $ 0 &
0 0 0 0 Xn Yn Zn 1 −vn Xn −vn Yn −vn Zn −vn ) ) m31 *
* 0
) *
) m32 *
) *
$ m33 &
m34
(10)

3 [Optional] Handling Distortion in Calibration

So far, we have been working with ideal lenses which are free from any distortion. How-
ever, as seen before, real lenses can deviate from rectilinear projection, which requires
more advanced methods. This section provides just a brief introduction to handling
distortions.
Often, distortions are radially symmetric because of the physical symmetry of the
lens. We model the radial distortion with an isotropic transformation:
#1 %
0 0 ' (
λ u
QPi = $ 0 λ 0& M Pi = i = Pi
1
(11)
vi
0 0 1
Let q1 , q2 , and q3 denote the three rows of matrix Q, we can rewrite this into a system
of equations as before, we get
ui q 3 P i = q 1 P i
vi q 3 P i = q 2 P i
This system, however, is no longer linear, and we require the use of nonlinear op-
timization techniques, which are covered in Section 22.2 of Forsyth & Ponce. We can
simplify the nonlinear optimization of the calibration problem if we make certain assump-
tions. In radial distortion, we note that the ratio between two coordinates ui and vi is
not affected. We can compute this ratio as
m 1 Pi
ui m 3 Pi m1 Pi
= m 2 Pi
= (12)
vi m 3 Pi
m2 Pi
Assuming that n correspondences are available, we can set up the system of linear
equations:
v1 (m1 P1 )−u1 (m2 P1 ) = 0
..
.
vn (m1 Pn )−un (m2 Pn ) = 0

5
Course Note GEO1016: Photogrammetry and 3D Computer Vision

Similar to before, this gives a matrix-vector product that we can solve via SVD:
# %
v1 PT1 −u1 PT1 ' T (
) .. .. * m1 = 0 (13)
$ . . & mT
2
vn Pn −un PTn
T

Once m1 and m2 are estimated, m3 can be expressed as a nonlinear function of

m1 , m2 , and λ. This requires to solve a nonlinear optimization problem whose complexity
is much simpler than the original one.