Cameras

CSE 803– Fall 2024, MSU

Xiaoming Liu

Thank many researchers who have made their slides and course materials available
1
 Let’s Take a Picture!

Photosensitive Material
Idea 1: Just use film Result: Junk
Slide inspired by S. Seitz; image from Michigan Engineering 4
 Let’s Take a Picture!

Photosensitive Material
Idea 2: add a barrier
Slide inspired by S. Seitz; image from Michigan Engineering 5
 Let’s Take a Picture!

Photosensitive Material
Idea 2: add a barrier
Slide inspired by S. Seitz; image from Michigan Engineering 6
 Let’s Take a Picture!

Photosensitive Material
Film captures all the rays going through a point (a
pencil of rays).
Result: good in theory!

Slide inspired by S. Seitz; image from Michigan Engineering 7

 Camera Obscura
• Basic principle known to
Mozi (470-390 BCE),
Aristotle (384-322 BCE)
• Drawing aid for artists:
described by Leonardo
da Vinci (1452-1519)

Gemma Frisius, 1558

Source: A. Efros 8
Camera Obscura

An image of the New Royal Palace A freestanding room-sized camera

at Prague Castle projected onto obscura at the University of North
an attic wall by a hole in the tile Carolina at Chapel Hill. One of the
roofing pinholes can be seen in the panel
to the left of the door.

9
 Camera Obscura

From Grand Images Through a Tiny Opening, Photo

District News, February 2005

Abelardo Morell, Camera Obscura Image of Manhattan

View Looking South in Large Room, 1996

http://www.abelardomorell.net/project/camera-obscura/ 10
Source: D Fouhey
 11
Abelardomorell.com
 Camera Obscura

Hotel room View Out of

contrast enhanced Hotel Room Window

https://www.youtube.com/watch?time_continue=227&v=WfRiBSEfCFQ&feature=emb_logo

Source: A. Torralba, W. Freeman Accidental Pinhole and Pinspeck Cameras, CVPR 2012 12
 Projection

P O

How do we find the projection P of a point X?

Form visual ray from X to camera center and intersect it with
camera plane
Source: L Lazebnik 13
 Projection

X
X’
P O

Both X and X’ project to P. Which appears in the image?

Are there points for which projection is undefined?

Source: L Lazebnik 14
 Quick Aside: Remember This?

d
b θ
a θ c

𝑎 𝑑 𝑏𝑑
= 𝑎=
𝑏 𝑐 𝑐
15
 Projection Equations

f y
X (x,y,z)

O z
P x

Coordinate system: O is origin, XY in image, Z sticks out.

XY is image plane, Z is optical axis.
(x,y,z) projects to (fx/z,fy/z) via similar triangles
Source: L Lazebnik 16
 Some Facts About Projection
3D lines project to 2D lines
The projection of any 3D
parallel lines converge at a
vanishing point
Distant objects are smaller

List of properties from M. Hebert 17

Some Facts About Projection

Let’s try some fake images

18
 Some Facts About Projection

Slide by Steve Seitz 19

 Some Facts About Projection

Slide by Steve Seitz 20

 Some Facts About Projection

Illusion Credit: RN Shepard, Mind Sights: Original Visual Illusions, Ambiguities, and other Anomalies 21
 What’s Lost?
Is she shorter or further
away?
Are the orange lines we
see parallel / perpendicular
/ neither to the red line?

Inspired by D. Hoiem slide 22

 What’s Lost?
Is she shorter or further
away?
Are the orange lines we
see parallel / perpendicular
/ neither to the red line?

Adapted from D. Hoiem slide 23

 What’s Lost?
Be careful of drawing conclusions:
• Projection of 3D line is 2D line; NOT 2D line is
3D line.
• Can you think of a counter-example (a 2D
line that is not a 3D line)?

• Projections of parallel 3D lines converge at VP;

NOT any pair of lines that converge are parallel
in 3D.
• Can you think of a counter-example?

24
Source: D Fouhey
 Do You Always Get Perspective?

25
Source: D Fouhey
 Do You Always Get Perspective?

Y location of 𝒇𝒚 𝒇𝒚 𝒇𝒚 𝒇𝒚
blue and red
𝒛𝟐 𝒛𝟏 𝒛 𝒛
dots in image: 26
Source: D Fouhey
 Do You Always Get Perspective?
When plane is fronto-parallel
(parallel to camera plane),
everything is:
• scaled by f/z
• otherwise is preserved.

27
Source: D Fouhey
 What’s This Useful For?

Things looking different when viewed from

different angles seems like a nuisance. It’s
also a cue. Why?
28
Source: D Fouhey
 Projection Equation

f y
X

P O z
x
(x,y,z) → (fx/z,fy/z)
I promised you linear algebra: is this linear?
Nope: division by z is non-linear
(and risks division by 0) 29
Adapted from S. Seitz slide
 Homogeneous Coordinates (2D)
Trick: add a dimension!
This also clears up lots of nasty special cases

Physical Homogeneous Physical

Point Point Point
𝑥 𝑢
𝑢/𝑤
𝑦 𝑣
Concat 𝑤 Divide 𝑣/𝑤
w=1 by w
What if w = 0?

Adapted from M. Hebert slide 30

 Homogeneous Coordinates
λ[x,y,w]
Triple / Double /
Equivalent Equals [x,y,w]
𝑢 𝑢 # 𝑢 𝑢 #
𝑣 ≡ 𝑣# ↔ 𝑣 = 𝜆 𝑣# y
𝑤 𝑤# 𝑤 𝑤#
𝜆≠0 z
Two homogeneous coordinates are x
equivalent if they are proportional
to each other. Not = !
31
Source: D Fouhey
 Benefits of Homogeneous Coords
General equation of 2D line:

𝑎𝑥 + 𝑏𝑦 + 𝑐 = 0
Homogeneous Coordinates
𝑎 𝑥
!
𝒍 𝒑 = 0, 𝒍 = 𝑏 ,𝒑 = 𝑦
𝑐 1

Slide from M. Hebert 32

 Benefits of Homogeneous Coords

• Lines (3D) and points (2D → 3D) are now the

same dimension.
• Use the cross (x) and dot product for:
• Intersection of lines l and m: l x m
• Line through two points p and q: p x q
• Point p on line l: lTp
• Parallel lines, vertical lines become easy
(compared to y=mx+b)

33
Source: D Fouhey
 Benefits of Homogeneous Coords

What’s the intersection?

0x + 1y - 2 = 0

1x + 0y - 1 = 0
[0,1,-2] x [1,0,-1] = [-1,-2,-1]
Converting back (divide by -1)
(1,2)
34
Source: D Fouhey
 Benefits of Homogeneous Coords

What about parallel lines?

0x + 1y - 2 = 0
0x + 1y - 1 = 0

[0,1,-2] x [0,1,-1] = [1,0,0]

Any point [x,y,0] is at infinity
All operations generate valid results!
35
Source: D Fouhey
36
Cameras
CSE 803– Fall 2024, MSU

Xiaoming Liu

37
 Recap: Homogeneous Coords
Line of y=2 in
ax+by+c=0:
0x + 1y - 2 = 0
𝑎, 𝑏, 𝑐 = 0,1, −2
𝑥, 𝑦 = (1,2)
Append 1
𝑢, 𝑣, 𝑤 = (1,2,1) Point-on-line test: lTp

Divide by w 𝑎, 𝑏, 𝑐 ! 𝑢, 𝑣, 𝑤 =
𝑥, 𝑦 = (1,2) 0,1, −2 ! 1,2,1 = 0

38
Source: D Fouhey
 Recap: Homogeneous Coords

𝑎" , 𝑏" , 𝑐" = (0,1, −2) Line y=2

𝑎# , 𝑏# , 𝑐# = (1,0, −1) 0x + 1y - 2 = 0
Intersection: l1 x l2
Line x=1
1x + 0y - 1 = 0

[0,1,-2] x [1,0,-1] = [-1,-2,-1]

Converting back (divide by -1)
(1,2)
39
Source: D Fouhey
 Benefits of Homogeneous Coords

0x + 1y - 3 = 0
0x + 1y - 2 = 0
0x + 1y - 1 = 0

Intersection of y=2, y=1

[0,1,-2] x [0,1,-1] = [1,0,0]
Does it lie on y=3? Intuitively?
[0,1,-3]T[1,0,0] = 0
40
Source: D Fouhey
 Benefits of Homogeneous Coords
Translation is now linear / matrix-multiply
𝑢′ 1 0 𝑡$ 𝑢 𝑢 + 𝑡$
If w = 1 𝑣′ = 0 1 𝑡% 𝑣 = 𝑣 + 𝑡%
𝑤′ 0 0 1 1 1
𝑢′ 1 0 𝑡$ 𝑢 𝑢 + 𝑤𝑡$
Generically 𝑣′ = 0 1 𝑡% 𝑣 = 𝑣 + 𝑤𝑡%
𝑤′ 0 0 1 𝑤 𝑤

Rigid body transforms (rot + trans) now linear

𝑢′ 𝑟"" 𝑟"# 𝑡$ 𝑢
𝑣′ = 𝑟#" 𝑟## 𝑡% 𝑣
Source: D Fouhey 𝑤′ 0 0 1 𝑤 41
 3D Homogeneous Coordinates

Same story: add a coordinate, things are

equivalent if they’re proportional

𝑢
𝑥 𝑢/𝑡
𝑣
𝑦 𝑣/𝑡
𝑤
𝑧 𝑤/𝑡
𝑡

42
Source: D Fouhey
 Projection Matrix

Projection (fx/z, fy/z) is matrix multiplication

f
O
𝑥
𝑓𝑥 𝑓 0 0 0
𝑦 𝑓𝑥/𝑧
𝑓𝑦 ≡ 0 𝑓 0 0 𝑧 → dis
𝑓𝑦/𝑧
𝑧 0 0 1 0 1

Slide inspired from L. Lazebnik 43

 Projection Matrix

Projection (fx/z, fy/z) is matrix multiplication

f
O
𝑥
𝑓𝑥 𝑓 0 0 0
𝑦 𝑓𝑥/𝑧
𝑓𝑦 ≡ 0 𝑓 0 0 𝑧 →
𝑓𝑦/𝑧
𝑧 0 0 1 0 1

Slide inspired from L. Lazebnik 44

 Why ≡ ≠ =

X
X’
P O

Project X and X’ to the image and

compare them
𝑓𝑥 # 𝑓𝑥 #
𝑓𝑥 𝑓𝑥
YES 𝑓𝑦 ≡ 𝑓𝑦 ′ NO 𝑓𝑦 = 𝑓𝑦 ′
𝑧 𝑧′ 𝑧 𝑧′
45
 Typical Perspective Model

P: 2D homogeneous X: 3d homogeneous
point (3D) point (4D)

𝑓 0 𝑢$
𝑷≡ 0 𝑓 𝑣$ 𝑹%&% 𝒕%&' 𝑿(&'
0 0 1

46
Source: D Fouhey
 Typical Perspective Model

R: rotation between t: translation

world system and between world
camera system and camera
𝑓 0 𝑢$
𝑷≡ 0 𝑓 𝑣$ 𝑹%&% 𝒕%&' 𝑿(&'
0 0 1

47
Source: D Fouhey
 Typical Perspective Model

f focal length u0,v0: principal

point (image coords
of camera origin on
retina)
𝑓 0 𝑢$
𝑷≡ 0 𝑓 𝑣$ 𝑹%&% 𝒕%&' 𝑿(&'
0 0 1

48
Source: D Fouhey
 Typical Perspective Model

Intrinsic Extrinsic
Matrix K Matrix [R,t]
𝑓 0 𝑢$
𝑷 ≡ 0 𝑓 𝑣$ 𝑹%&% 𝒕%&' 𝑿(&'
0 0 1

𝑷 ≡ 𝑲 𝑹, 𝒕 𝑿 ≡ 𝑴!"# 𝑿#"$

49
Source: D Fouhey
 Other Cameras – Orthographic

Orthographic Camera (z infinite)

1 0 0
𝑷= 0 1 0 𝑿&$"
0 0 0

Image Credit: Wikipedia 50

 Other Cameras – Orthographic

Why does this make things easy and

why is this popular in old games?

1 0 0 𝑥
𝑷= 0 1 0 𝑦
0 0 0 𝑧

51
Source: D Fouhey
 The Big Issue

Photosensitive Material
Film captures all the rays going through a point (a
pencil of rays).
How big is a point?

Slide inspired by S. Seitz; image from Michigan Engineering 57

 Math vs. Reality

• Math: Any point projects to one point

• Reality (as pointed out by the class)

• Don’t image points behind the camera / objects
• Don’t have an infinite amount of sensor material

• Other issues
• Light is limited
• Spooky stuff happens with infinitely small holes

58
 Limitations of Pinhole Model

Ideal Pinhole
-1 point generates 1 image
-Low-light levels

Finite Pinhole
-1 point generates region
-Blurry.
Why is it blurry?

Slide inspired by M. Hebert 59

 Limitations of Pinhole Model

Slide Credit: S. Seitz 60

 Adding a Lens

• A lens focuses light onto the film

• Thin lens model: rays passing through the center
are not deviated (pinhole projection model still
holds)
Slide Credit: S. Seitz 61
 Adding a Lens

focal point

• All rays parallel to the optical axis pass

through the focal point

Slide Credit: S. Seitz 62

 What’s The Catch?

“circle of
confusion”

• There’s a distance where objects are “in focus”

• Other points project to a “circle of confusion”

Slide Credit: S. Seitz 63

 Thin Lens Formula
We care about images that are in focus.
When is this true? Discuss with your neighbor.
When two paths from a point hit the same image location.

object lens image

Diagram credit: F. Durand
plane 64
 Thin Lens Formula
Let’s derive the relationship between object distance D, image
plane distance D’, and focal length f.

D D′
f
y
y′

object lens image

Diagram credit: F. Durand
plane 65
 Thin Lens Formula
One set of similar 𝑦′ 𝑦 𝑦′ 𝐷 # − 𝑓
triangles: #
= =
𝐷 −𝑓 𝑓 𝑦 𝑓
D D′
f
y
y′

object lens image

Diagram credit: F. Durand
plane 66
 Thin Lens Formula
Another set of 𝑦′ 𝑦 𝑦′ 𝐷′
similar triangles: = =
𝐷′ 𝐷 𝑦 𝐷
D D′
f
y
y′

object lens image

Diagram credit: F. Durand
plane 67
 Thin Lens Formula
Set them 𝐷′ 𝐷′ − 𝑓 1 1 1
equal: = + =
𝐷 𝑓 𝐷 𝐷′ 𝑓
D D′
f
y
y′

object lens image

Diagram credit: F. Durand
plane 68
 Thin Lens Formula
Suppose I want to take a picture of a lion with D big?
Which of D, D’, f are fixed?

How do we take pictures of things at different distances?

D D′
f
1 1 1
+ =
𝐷 𝐷′ 𝑓

object lens image

plane
Diagram credit: F. Durand 69
 Depth of Field

http://www.cambridgeincolour.com/tutorials/depth-of-field.htm

Slide Credit: A. Efros 70

 Controlling Depth of Field

Changing the aperture size affects depth of field

A smaller aperture increases the range in which
the object is approximately in focus
Diagram: Wikipedia 71
 Controlling Depth of Field

If a smaller aperture makes everything

focused, why don’t we just always use it?

Diagram: Wikipedia 72
 Varying the Aperture

Small aperture = large DOF Large aperture = small DOF

Slide Credit: A. Efros, Photo: Philip Greenspun 73
Varying the Aperture

74
 Field of View (FOV)

Photo. Material
𝜙
𝑑

𝑑
𝜙 = tan)'
2𝑓
tan-1 is monotonic increasing.
How can I get the FOV bigger?
75
Source: D Fouhey
 Field of View

Slide Credit: A. Efros 76

 Field of View

Slide Credit: A. Efros 77

 Field of View and Focal Length

Large FOV, small f

Camera close to car

Small FOV, large f

Slide Credit: A. Efros, F. Durand Camera far from the car 78
 Field of View and Focal Length

wide-angle standard telephoto

Slide Credit: F. Durand 79

 Dolly Zoom
Change f and distance at the same time

Video Credit: Goodfellas 1990 80

 More Bad News!

• First a pinhole…
• Then a thin lens model….

Slide: L. Lazebnik 81
 Lens Flaws: Radial Distortion
Lens imperfections cause distortions as a function
of distance from optical axis

Less common these days in consumer devices

Photo: Mark Fiala, U. Alberta 82

 Radial Distortion Correction
y

Photo. Material
f
z r
y'
Ideal Distorted
𝑦 𝑦
𝑦′ = 𝑓 𝑦′ = (1 + 𝑘' 𝑟 * + ⋯ )
𝑧 𝑧 83
Source: D Fouhey
 Vignetting

Photo. Material
What happens to the light between the
black and red lines?

Slide inspired by L. Lazebnik Slide 84

 Vignetting

Photo credit: Wikipedia (https://en.wikipedia.org/wiki/Vignetting) 85

 Lens Flaws: Spherical Abberation
Lenses don’t focus light perfectly!
Rays farther from the optical axis focus closer

Slide: L. Lazebnik 86
 Lens Flaws: Chromatic Abberation
Lens refraction index is a function of the
wavelength. Colors “fringe” or bleed

Image credits: L. Lazebnik, Wikipedia 87

 From Photon to Photo

• Each cell in a sensor array is a light-sensitive diode that

converts photons to electrons
• Dominant in the past: Charge Coupled Device (CCD)
• Dominant now: Complementary Metal Oxide
Semiconductor (CMOS)

Slide Credit: L. Lazebnik, Photo Credit: Wikipedia, Stefano Meroli 89

 From Photon to Photo
Rolling Shutter: pixels read in sequence
Can get global reading, but $$$

90
Source: D Fouhey
 Preview of What’s Next
Bayer grid
Demosaicing:
Estimation of missing components
from neighboring values

Slide Credit: S. Seitz 91

Human Luminance Sensitivity Function
 Historic milestones
• Pinhole model: Mozi (470-390 BCE),
Aristotle (384-322 BCE)
• Principles of optics (including lenses):
Alhacen (965-1039 CE)
Alhacen’s notes
• Camera obscura: Leonardo da Vinci
(1452-1519), Johann Zahn (1631-1707)
• First photo: Joseph Nicephore Niepce (1822)
• Daguerréotypes (1839)
• Photographic film (Eastman, 1889)
• Cinema (Lumière Brothers, 1895)
Niepce, “La Table Servie,” 1822
• Color Photography (Lumière Brothers, 1908)
• Television (Baird, Farnsworth, Zworykin, 1920s)
• First consumer camera with CCD
Sony Mavica (1981)
• First fully digital camera: Kodak DCS100 (1990)
Slide Credit: S. Lazebnik 92
Old television camera
 First digitally scanned photograph

• 1957, 176x176 pixels

Slide Credit: http://listverse.com/history/top-10-incredible-early-firsts-in-photography/ 93

 Historic Milestone
Sergey Prokudin-Gorskii (1863-1944)
Photographs of the Russian empire (1909-1916)

Slide Credit: S. Maji 94

 Historic Milestone

Slide Credit: S. Maji 95

 Future Milestone

Your job in homework 1:

Make the left look like the right.

96
97