Computational Imaging

Lecture 04：Cameras

Qilin Sun（孙启霖）

香港中文大学（深圳）

点昀技术（Point Spread Technology）

Todays
Topic
GAMES204 Computational Imaging, Qilin Sun
GAMES204 Computational Imaging, Qilin Sun
 nautilus eye
wikipedia

GAMES204 Computational Imaging, Qilin Sun

 https://www.dailypaws.com/cats-kittens/behavior/common-cat-behaviors/why-do-cats-eyes-glow-in-the-dark
GAMES204 Computational Imaging, Qilin Sun
 The dark blue, teal, and gold tapetum
lucidum from the eye of a cow

https://encyclopaediafelidae.com/through-cat-eyes/
GAMES204 Computational Imaging, Qilin Sun
 National Geographic
GAMES204 Computational Imaging, Qilin Sun
GAMES204 Computational Imaging, Qilin Sun
GAMES204 Computational Imaging, Qilin Sun
GAMES204 Computational Imaging, Qilin Sun
 Nimrud lens - 2700 years old
GAMES204 Computational Imaging, Qilin Sun
 lensmaker’s
magnification:
equation:

GAMES204 Computational Imaging, Qilin Sun

GAMES204 Computational Imaging, Qilin Sun
 Ideal Astigmatism

Seperical
Abberation Distortion

Coma
Chromatic
Abberation

GAMES204 Computational Imaging, Qilin Sun

 Camera Optics: Lens Aberrations

Sharp Image Blurred Image due to optical aberrations

GAMES204 Computational Imaging, Qilin Sun

GAMES204 Computational Imaging, Qilin Sun
GAMES204 Computational Imaging, Qilin Sun
GAMES204 Computational Imaging, Qilin Sun
 Camera Aperture
Unit: f-number N=f/D

GAMES204 Computational Imaging, Qilin Sun

 Camera Aperture

GAMES204 Computational Imaging, Qilin Sun

GAMES204 Computational Imaging, Qilin Sun
 circle of confusion
GAMES204 Computational Imaging, Qilin Sun
GAMES204 Computational Imaging, Qilin Sun
GAMES204 Computational Imaging, Qilin Sun
 http://photographywisdom.com/
GAMES204 Computational Imaging, Qilin Sun
GAMES204 Computational Imaging, Qilin Sun
GAMES204 Computational Imaging, Qilin Sun
 For a fixed sensor size, decreasing the focal length
increases the field of view.
GAMES204 Computational Imaging, Qilin Sun
GAMES204 Computational Imaging, Qilin Sun
 Credit: lensvid.com
GAMES204 Computational Imaging, Qilin Sun
 Mobile Phone: Maintain FOV on Smaller Sensor?

To maintain FOV, decrease focal length of lens

in proportion to width/height of sensor
GAMES204 Computational Imaging, Qilin Sun
 Andrew McWilliams

GAMES204 Computational Imaging, Qilin Sun

 Camera Field of View (FOV)

James Webb– What’s the focal length?

A: 131.4m
GAMES204 Computational Imaging, Qilin Sun
 Ø For historical reasons, usually refer to angular
FOV by focal length of a lens used on a 35mm-
format film (36 x 24mm)

Ø Examples of focal lengths on 35mm format:

Ø 17mm is wide angle 104°

Ø 50mm is a “normal” lens 47°

Ø 200mm is telephoto lens 12°

Ø Note: When we say current cell phones have

approximately 28mm “equivalent” focal length,
this uses the above convention. The physical
focal length is often 5-6 times shorter, because
the sensor is correspondingly smaller
GAMES204 Computational Imaging, Qilin Sun
 Ernst Abbe 1873:

spot radius (image space)

Diffraction Airy pattern

GAMES204 Computational Imaging, Qilin Sun

 Ernst Abbe 1873:

spot radius (image space)

Ø Microscope objectives today: NA 1.4-1.6 à d=λ/2.8

Ø Small f-number (large NA) = high resolution but shallow depth of field
Ø inherent tradeoff between “3D” information and 2D resolution
Ø space-bandwidth product (uncertainty principle)

GAMES204 Computational Imaging, Qilin Sun

GAMES204 Computational Imaging, Qilin Sun
 From Canon EF Lens Work III

In this sequence,
distance from subject increases
with focal length to maintain
image size of human subject.

Notice the dramatic change in

background perspective.

Perspective Composition – Camera Position / Focal Length

GAMES204 Computational Imaging, Qilin Sun
 Up close and zoomed wide
with short focal length

Perspective Composition
GAMES204 Computational Imaging, Qilin Sun
 Tips: Photographer’s Mindset

Walk back and zoom in

with long focal length

Perspective Composition
GAMES204 Computational Imaging, Qilin Sun
 “Choose your perspective before
you choose your lens.”
— Ming Thein, mingthein.com

Perspective Composition
GAMES204 Computational Imaging, Qilin Sun
 Tip 1: Make sure you have a strong subject
Ø Make it prominent, e.g. 1/3 of your image

Tip 2: Choose a good perspective relationship (relative size) between your

subject and background (or foreground)
Ø Complement, don’t compete with the subject

Tip 3: Change the zoom and camera distance to your subject

Ø Implement: actively zoom, and move your camera in/out
Ø Even works with your smartphone!

GAMES204 Computational Imaging, Qilin Sun

GAMES204 Computational Imaging, Qilin Sun
GAMES204 Computational Imaging, Qilin Sun
 Ø Highly non-linear response:
Ø Here shown for slide film
Ø Terminology:
Ø Transmittance T: portion of
light transmitted (0…1)
Ø Density: log(1/T)
Ø Contrast: ratio between max.
and min. discernible intensity
Ø Example:
Ø Transmittance 0.001
Ø Density: 3
Ø Contrast of film:
Ø 1,000:1 or more

GAMES204 Computational Imaging, Qilin Sun

 Photon to electron converter
photoelectric effect!

Point Spread Technology

# electrons
QE =
# photons

www.olympus-lifescience.com wikipedia

GAMES204 Computational Imaging, Qilin Sun

 The photosite response is mostly linear
Ø non-linear when potential well is saturated
(over-exposure)

Number of electrons
Ø non-linear near zero (due to noise)
We will see how to deal with these issues in a
later lecture (HDR imaging).

under-exposure
(non-linearity due
to sensor noise)

over-exposure What does this slope equal?

(non-linearity due
to sensor saturation)
Number of photons
GAMES204 Computational Imaging, Qilin Sun
 Approach
Ø Pixels are covered by color filters
Ø Twice as many green pixels as red or blue ones
Ø Interpreted as “intensity”
Ø Missing color values are interpolated for every pixel
Ø More in later lecture
Ø Note: camera specs report the total number of pixels in
a sensor
Ø I.e, a image recorded by a 1-chip camera only
consists of 1/3 measured values, 2/3 are
interpolated

GAMES204 Computational Imaging, Qilin Sun

 Bayer pattern

Any combination possible tradeoffs?

wikipedia
GAMES204 Computational Imaging, Qilin Sun
 Common Semiconductor sensors used in cameras:
Ø Charge Coupled Devices (CCD)
Ø Complementary Metal-Oxide-Semiconductor (CMOS)
Ø Foveon

Others
Ø Charge Injection Devices (CID)
Ø Photomultipliers
Ø Cathode Ray Tubes (CRT)
Ø Single Photon Avalance Diode Array(SPAD)
Ø Quanta Image Sensor (QIS)
Ø Two-bucket Sensor
Ø Photonic Mixer Device (PMD)
Ø ...

GAMES204 Computational Imaging, Qilin Sun

 CCD: High Sensivity for infrared
GAMES204 Computational Imaging, Qilin Sun
GAMES204 Computational Imaging, Qilin Sun
 photon sensor defects = quantization
additive noise
noise fixed pattern noise “noise”

RAW image
photons

GAMES204 Computational Imaging, Qilin Sun

 Ø Noise is (usually) bad!

Ø Many sources of noise: heat, electronics, amplifier gain,

photon to electron conversion, pixel defects, read, …

Ø Different noise follows different statistical distributions,

two crucial ones:

Ø Gaussian

Ø Poisson

GAMES204 Computational Imaging, Qilin Sun

 Ø Thermal, read, amplifier

Ø Additive, signal-independent!

+ =

GAMES204 Computational Imaging, Qilin Sun

 Ø Signal dependent

Ø Poisson distribution:

wikipedia

GAMES204 Computational Imaging, Qilin Sun

GAMES204 Computational Imaging, Qilin Sun
 Ø e.g., Andor iXon Ultra 897: cooled to -100° C

Ø Scientific CMOS & CCD

Ø Reduce pretty much all noise, except for photon noise

GAMES204 Computational Imaging, Qilin Sun

GAMES204 Computational Imaging, Qilin Sun
 High-Speed Photography
Ø Short exposure
Ø Bright strobe illumination
Ø Gun synced to camera

Harold Edgerton Mark Watson

GAMES204 Computational Imaging, Qilin Sun
 Long-Exposure Photography

https://www.demilked.com/best-long-exposure-photos/
GAMES204 Computational Imaging, Qilin Sun
 Ø Q =T x E
Ø Exposure = time x irradiance (e.g. 1/250, 1/60, 1, 15, bulb)
Ø Exposure time (T)
Ø Controlled by shutter
Ø Irradiance (E)
Ø Power of light falling on a unit area of sensor
Ø Controlled by lens aperture and focal length

wikipedia
¼ sec, f/3.3 2 sec, f/6.3,
GAMES204 Computational Imaging, Qilin Sun
 Exposure bracketing with +/– 1 stop exposure

https://www.dpmag.com/how-to/tip-of-the-week/how-and-why-to-use-auto-exposure-bracketing/
GAMES204 Computational Imaging, Qilin Sun
GAMES204 Computational Imaging, Qilin Sun
 Ø F-Number
f/# = N = f/D
where f is the focal length, and D is the diameter of the entrance pupil

Ø Common F-stops on real lenses: 1.4, 2, 2.8, 4.0, 5.6, 8, 11, 16, 22, 32
Ø 1 stop doubles exposure
Ø Notation: an f-stop of, e.g. 2 is sometimes written f/2, or F:2

GAMES204 Computational Imaging, Qilin Sun

 Aperture size:
Ø Change the f-stop by opening / closing the aperture (if camera has
iris control)

Shutter speed:
Ø Change the duration the sensor pixels integrate light

ISO gain:
Ø Change the amplification (analog and/or digital) between sensor
values and digital image values

GAMES204 Computational Imaging, Qilin Sun

 Example: these pairs of aperture and shutter speed
give equivalent exposure

F-Stop

Shutter

If the exposure is too bright/dark, may need to adjust

f-stop and/or shutter up/down.

GAMES204 Computational Imaging, Qilin Sun

 Third variable for exposure

Image sensor: trade sensitivity for noise

Ø Multiply signal before analog-to-digital conversion
Ø Linear effect (ISO 200 needs half the light as ISO 100)

GAMES204 Computational Imaging, Qilin Sun

 Sensor
sensitivity
–
Analog gain
applied before
ADC!

Image becomes very grainy because noise is amplified. bobatkins.com

GAMES204 Computational Imaging, Qilin Sun
GAMES204 Computational Imaging, Qilin Sun
 Motion blur: handshake, subject movement
Doubling shutter time doubles motion blur

Gavin Hoey
http://www.gavtrain.com/?p=3960
GAMES204 Computational Imaging, Qilin Sun
 Motion blur: handshake, subject movement
Doubling shutter time doubles motion blur

London

GAMES204 Computational Imaging, Qilin Sun

 Ø Pixel is electronically reset to start exposure
Ø Fills with photoelectrons as light falls on sensor
Ø Reading out pixel electronically “ends” exposure
Ø Problem: most sensors read out pixels sequentially, takes time (e.g. 1/30 sec) to read
entire sensor
Ø If reset all pixels at the same time, last pixel read out will have longer
exposure
Ø So, usually stagger reset of pixels to ensure uniform exposure time
Ø Problem: rolling shutter artifact

GAMES204 Computational Imaging, Qilin Sun

 Credit: David Adler, B&H Photo Video Credit: Soren Ragsdale
https://www.bhphotovideo.com/explora/video/tips-and-solutions/rolling-shutter- https://flic.kr/p/5S6rKw
versus-global-shutter

GAMES204 Computational Imaging, Qilin Sun

 The Slow Mo Guys, https://youtu.be/CmjeCchGRQo

GAMES204 Computational Imaging, Qilin Sun

 Delay from top to bottom
of sensor (e.g. 1/30 sec)
dpreview.com
GAMES204 Computational Imaging, Qilin Sun
 The Slow Mo Guys, https://youtu.be/CmjeCchGRQo

GAMES204 Computational Imaging, Qilin Sun

 All sensor pixels exposed at Row-by-row readout of image
same time Ø shorter exposure times per pixel
Ø motion artifacts

http://lfa.mobivap.uva.es/~fradelg/phd/notes/global-shutter.html
GAMES204 Computational Imaging, Qilin Sun
 Photographers must trade off depth of field
GAMES204 Computational Imaging, Qilin Sun
and motion blur for moving subjects
 Shallow Depth of Field Can Create a Stronger Image

From Peterson, Understanding Exposure

200mm, f/4, 1/1000 (left) and f/11, 1/125 (right)
GAMES204 Computational Imaging, Qilin Sun
GAMES204 Computational Imaging, Qilin Sun
GAMES204 Computational Imaging, Qilin Sun
 Ø Low Resolution LED Array
Ø About 1000-2000 LEDs

Ø Conventional LCD

Ø Diffusion Optics

Wolfgang Heidrich [BrightSite Technology]

GAMES204 Computational Imaging, Qilin Sun

GAMES204 Computational Imaging, Qilin Sun
 200 mm telephoto 35 mm wide-angle

50 mm telephoto 16 mm fisheye

From Kolb, Mitchell and Hanrahan (1995)

GAMES204 Computational Imaging, Qilin Sun

 200 mm telephoto

Notice shallow depth of field (out of focus background)

GAMES204 Computational Imaging, Qilin Sun
 16 mm fisheye

Notice distortion in the corners (straight lines become curved)

GAMES204 Computational Imaging, Qilin Sun
 Monte Carlo approach
Ø At every sensor pixel, compute integral of rays
incident on pixel area arriving from all paths
through the lens

Algorithm (for a pixel)

Ø Choose N random positions in pixel
Ø For each position x’, choose a random position on
the back element of the lens x’’
Ø Trace a ray through from x’ to x’’, trace
refractions through lens elements until it misses
the next element (kill ray) or exits the lens (path
trace through the scene)
Ø Weight each ray according to radiometric
calculation on next slide to estimate irradiance
E(x’)
GAMES204 Computational Imaging, Qilin Sun
 Back element of lens

Sensor Plane

GAMES204 Computational Imaging, Qilin Sun

Todays
Topic
Thank You!

Qilin Sun（孙启霖）

香港中文大学（深圳）

点昀技术（Point Spread Technology）