PERSPECTIVE CAMERA MODEL

Formalizing the pinhole camera model: Properties of the perspective projection Linear formulation Calibration Distortions

PERSPECTIVE PROJECTION

u= f

X Z Y v= f Z

Pinhole geometry without image reversal

PROJECTION IS NON LINEAR

HOMOGENEOUS COORDINATES
Homogeneous representation of 2D point:

u= f

X Z Y v= f Z

x = (x1 , x2 , x3 ) represents x1 , x2 x x3 3

Homogeneous representation of 3D point:

x x = ( x1 , x2 , x3 , x4 ) represents x1 , x2 , 3 x x4 x4 4

Reformulate it as a linear operation.

Projections become linear transformations.

SIMPLE PROJECTION MATRIX

v
u

INTRINSIC AND EXTRINSIC PARAMETERS

1. Camera may not be at the origin, looking down the z-axis
Extrinsic parameters

u= f

X Z Y v= f Z

x f y = 0 z 0 f = 0 0

0 f 0 0 f 0

X 0 0 c Y x y 0 0 c with u = , v = Z z z c 1 0 1 X 0 1 0 0 0 c 0 1 0 0 Yc 0 Z 1 0 0 1 0 c 1

2. One unit in camera coordinates may not be the same as one unit in world coordinates
Intrinsic parameters
X x Transformation 1 0 0 0 Transformation Y y = representing 0 1 0 0 representing z intrinsic parameters 0 0 1 0 extrinsic parameters Z 1

LINEAR CAMERA MODEL

X x Transformation 1 0 0 0 Transformation Y y = representing 0 1 0 0 representing z intrinsic parameters 0 0 1 0 extrinsic parameters Z 1 X 1 0 0 0 Y = K 0 1 0 0 Rt , 0 0 1 0 Z 1 where K is a 3x3 matrix and Rt a 4x4 matrix.

PRINCIPAL POINT

u ' = u + pu = fX / Z + pu
pv v
u

v' = v + pv = fY / Z + pv
p
pu

f K=

pu pv 1

INHOMOGENEOUS SCALING
u ' = u + pu = u X / Z + pu v' = v + pv = vY / Z + pv u K= pu pv 1

AXIS SKEW

u K=

pu pv 1

The pixels are not necessarily square.

Encodes the non-orthogonality of the u and v directions.

ROTATION / TRANSLATION
X Xc ~ Y = R Y C with R t R = I c Z Z c Xc X Y R T Y ~ c = with T = RC Z c 0 1 Z 1 1

FULL PROJECTION MATRIX

x u y = 0 z 0 X pu 1 0 0 0 R T Y ~ t v pv 0 1 0 0 0 1 Z with T = RC and R R = I 0 1 0 0 1 0 1 X p11 p12 p13 p14 Y = p21 p22 p23 p24 Z p31 p32 p33 1 1 ~ ~ x = K[R RC]X = KR[I C]X = [M m ]X = PX s

Rotations and translations also expressed in terms of matrix multiplications in projective space.

CAMERA PARAMETERS
Internal Parameters: Horizontal and vertical scaling (2) Principal points (2) Skew of the axis (1) External Parameters: Rotations (3) Translations (3) 11 free parameters.

CALIBRATION GRID

LEAST-SQUARES MINIMIZATION
For 1 i n Pu ( X i ) = ui + u ,i Pv ( X i ) = vi + v ,i Minimize

LINE CAMERA
1D camera under 2D translation, parameterized by (Xc, Yc), the camera center coordinates. 100 points taken at random in [400;1000]x[-500;+500] True camera position at (0, 0)

i =1

2 u ,i

2 v ,i

= ( Pu ( X i ) ui ) + ( Pv ( X i ) vi )
2 i =1

Yc

Xc

with respect to the unknown calibration parameters.

ENERGY LANDSCAPE
White cross: True camera position; Black cross: Global minimum of the objective function.

NUMERICAL OPTIMIZATION

p2 p 1

p0

The global minimum of the object function is close to the true camera pose.

NOISY CORRESPONDENCES
White cross: True camera position; Black cross: Global minimum of the objective function.

DECOMPOSITION
Finding the camera center:
~ C ~ 0 = PC = [M m ] MC = m 1

Finding the camera orientation and internal parameters:

M = KR where K is triangular superior and R is orthogonal
The global minimum of the object function is now further from the true camera pose.

K and R can be computed by RQ decomposition

OPEN GL

GAMES

Two scenes rendered with a shading language developed at Stanford.

Cross platform 3D graphics library that takes advantage of specialized graphics hardware. Polygons are affected by the current color, transformation, drawing mode, etc.

4x4 HOMOGENEOUS MATRICES

MODEL VIEW MATRIX

Transforms the viewpoint and objects within the scene.

X x p11 p12 p13 p14 y p p22 p23 p24 R T Y = 21 z p31 p32 p33 1 0 1 Z b c d 1 h a Display point if and only if hmin h hmax

Example:
glMatrixMode(GL_MODELVIEW); // glLoadIdentity(); // glTranslatef(10.5, 0, 0); // glRotatef(45, 0, 0, 1); // DrawCube(); //
origin set the current matrix load the identity matrix translate 10.5 units along x-axis rotate 45 degrees CCW around z-axis cube is defined centered around

Where will this end up? Answer:

On the x-axis, rotated 45 degrees CCW.

Hardware acceleration to quickly perform multiplications of 4 vectors and 4x4 matrices

PROJECTION MATRIX
Sets up a perspective projection. A few available options:
// sets up a user defined viewing frustrum glFrustrum (...); // Calculates viewing frustrum for you, given field-of-view in degrees, aspect ratio, and near and far clipping planes. gluPerspective (fovy, aspect, near, far); // Creates an orthographic projection. Useful for 2D rendering. glOrtho (...);

Z BUFFER
Occlusion handling Creating shadows Voxelization
The Magic of the Z-Buffer: A survey. T. Theoaris et al. 2001.

Example:
glMatrixMode(GL_PROJECTION); glLoadIdentity();
gluPerspective(64,windowWidth,windowHeight,4,4096);

CAMERA AT INFINITY

WEAK PERSPECTIVE

As focal length and distance to camera increase, the image remains the same size but perspective effects diminish.

AFFINE CAMERAS

MOVING BACK THE OPTICAL CENTER

Original projection :
t~ r1 C t ~ r2 C t~ r3 C The optical center moves back by tr3 : ~ ~ r1t r1t C r1t r1t (C tr3 ) t t t ~ t ~ Pt = K r2 r2 (C tr3 ) = K r2 r2 C ~ r3t r3t r3t (C tr3 ) dt

m11 P = m21 0

m12 m22 0

m13 m23 0

t1 t2 1

The projection is linear in Euclidean space Parallel world lines project to parallel image lines The optical center is a point at infinity

r1 t ~ P0 = K R RC = K r2 r t 3
t

ZOOMING IN
Zooming in to conserve the image size : ~ r1t r1t C dt d r1t t t 0 d ~ dt t t Pt = K r2 r2 C = d 0 K r2 d0 t r t d 0 d dt 1 r3 t 3 In the limit : ~ r1t r1t C t M T t ~ P = lim Pt = K r2 r2 C = t 0 1 0 d0
t~ r1 C t ~ r2 C d0

QUALITY OF THE APPROXIMATION

Valid approximation if The range of depths is small compared to the average depth. The field of view is limited Usually true when using a long focal length

RADIAL DISTORTION

UNDISTORTING

Short focal length Long focal length Large distortion. Small distortion.

xd x = L(r ) y y d r = x2 + y2

x = xc + L( r )( x xc ) = yc + L( r )( y yc ) y with : r 2 = ( x xc ) 2 + ( y yc ) 2 L( r ) = 1 + 1r + 2 r 2 + 3 r 3 + ...

MAKING STRAIGHT LINES STRAIGHT

UNDISTORTING

PROJECTIVE CAMERAS

Once the image is undistorted, the camera projection can be formulated as a projective transform. The pinhole camera model applies.