Chapter2 Image Formation

Reading: Szeliski, Chapter 2

 What are we tuned to?
The visual system is tuned to process structures
typically found in the world.
What is a natural image?
The structure of ambient light
What is a natural image?
 What is a natural image?
The visual system seems to be tuned to a set of images:
 What is a natural image?
The visual system seems to be tuned to a set of images:

Did you saw this image?

The visual system seems to be tuned to a set of images:

Demo inspired from D. Field

 6 images

http://www.alexfito.com/

http://www.rowland.harvard.edu/images/ModPurp.jpg

Not all these images are the result of sampling a real-world plenoptic function
• Proposition 1. The primary task of early vision is to deliver a small set of useful
measurements about each observable location in the plenoptic function.

• Proposition 2. The elemental operations of early vision involve the measurement

of local change along various directions within the plenoptic function.

• Goal: to transform the image into other representations (rather than pixel values)
that makes scene information more explicit

Cavanagh, Perception 95
What are “visual features”

Shape, color, texture, etc

 2.1 Photometric Image Formation
• Discrete color or intensity values
• Where do these value come from?
– Geometry, projection
– Camera optics, sensor properties
– Lighting, surface properties
Images as Functions
 Images as Functions
• We can think of an image as a function, f, from R2 to R:
– f( x, y ) gives the intensity at position ( x, y )
– Realistically, we expect the image only to be defined over a
rectangle, with a finite range:
• f: [a,b]x[c,d]  [0,1]

• A color image is just three functions pasted together.

We can write this as a “vector-valued” function:
 r ( x, y ) 
f ( x, y )   g ( x , y ) 
 

 b( x, y )  
Images as functions
 What is a digital image?
• We usually work with digital (discrete) images:
– Sample the 2D space on a regular grid
– Quantize each sample (round to nearest integer)
• If our samples are  apart, we can write this as:
• f[i ,j] = Quantize{ f(i , j ) }
• The image can now be represented as a matrix of integer
values
Photometric Image Formation

Perspective projection Light scattering

Lens optics Bayer color filter array

Photometric Image Formation
 2.2 Lighting
• Point light source
– Single location (small light bulb)
– Infinity: the sun --directional light
• Area light source
– A finite rectangular area emitting light equally in all
directions
• Environment map
• Light direction to color mapping
 2.2.2 Reflectance and Shading

many models for reflectance and shading

BRDF: Bidirectional Reflectance Distribution Function
 BRDF
• BRDF is reciprocal

• For isotropic surface,

no preferred directions for light transport
 BRDF
• Light existing a surface point:

Foreshortening factor
 Diffuse Reflection
• Also called Lambertian or matte reflection
– Light is scattered uniformly in all directions, i.e.
– BRDF is constant:

Think about the inverse problem

 Specular Reflection
• Depends on the direction of outgoing light
• Mirror surface:
– Specular reflection direction

vr
cos( s )  (vr  si )
 Specular Reflection
• Amount of light
– Phone model
– Micro-facet model

– Larger , , more specular surface with

hightlights ; Smaller, softer gloss
 Phone Shading Model
• Diffuse
• Specular
• Ambient:
– Does not depend on surface orientation
– Color and both ambient illumination and
the object
 Phone Shading Model
Lr (vr ;  )  ka ( ) La ( ) The recent advent of
programmable pixel shaders
 kd ( ) Li ( )(vi  ni )  makes the use of more
i
complex models feasible.
 ks ( ) Li ( )(vr  si ) ke
i
Example
 Realistic Rendering

Ioannis Gkioulekas, et al, Siggraph’13

The recent advent of programmable pixel shaders makes the use of more complex
models feasible.
 Optics
• Lens, sensor
• Ideal pinhole camera
• More complex: focus, exposure, vignetting,
aberation,…,
Thin lens model
Thin lens: low, equal curvature on both sides
Optical axis
 Thin lens model

object

Focus plane
 Thin lens model

object

Circle of confusion

Pinhole camera
Pinhole Camera Model

object
Pinhole Camera Model

object
Pinhole Camera Model

object
 2.3 3D to 2D Projection
book: pp32-60

3D view of world perspective

• 3D perspective: the most commonly used

projection in computer vision and computer graphics
Pinhole Camera Model

object

xf
u
z
yf
v
z
Pinhole Camera Model

object
 Pinhole Camera Model
• Using homogeneous (projective) coordinate
xf
– u
z
yf
v
z
f 0 0 0  x 
u   fx / z    
    0 f 0 0   y 
w  v   z  fy / z  
1  1  0 0 1 0  z 
    0  
 0 0 1  1 

wx = Kp
 Camera Intrinsics
• Imperfect camera
• image sensor
f s cx 
K   0 af c y 
 0 0 1 
e.g ., (cx , c y )  (W / 2, H / 2)

• s: possible skew between sensor axes

• a: aspect ratio
(c x , c y )
• : optical center Five intrinsic parameters
• F : focal length
 Camera Intrinsics
• Focal length
• Actual focal length, e.g. 18~55mm,
• Conventional sensor width: 35 mm
• Digital Image: integer values, [0,W) x [0,H)
Field of view

Focal length Sensor width

 Extrinsic Parameters
• World Coordinate system to Camera
Coordinate system
p c  [R | t]p w
j
Extrinsic parameters

i i Cw  R 1t
O j
k
C
k
 Extrinsic Parameters

with wx = Kp
wx  K[R | t]p w
M  K[R | t]
Camera Matrix
 2.3 Digital Camera
• Process chart
 2.3 Digital Camera
• Process chart
 2.3.2 Color
• Light from different parts of the spectrum is
somehow integrated into discrete RGB color
values

wx  K[R | t]p w
 2.3.2 Color
• Primary and Secondary Colors
• Additive colors (projector, monitor)
• Subtractive colors (printing, printing)
 CIE color matching
• Commission Internationale d’Eclairage (CIE)
• Color matching experiments

pure colors to the

R=700.0nm,
G=546.1nm, and
B=435.8nm
 XYZ Color Space

Y=1 for pure R (1,0,0)

 XYZ Color Space

• Y=1 for (1,1,1)

 XYZ Color Space
• Chromaticity coordinates

• Yxy (luminance plus the two most distinctive

chrominance components)
Chromaticity Diagram
 L*a*b* Color Space
• Human visual system is roughly logarithmic
• Differences in luminance or chrominance are
more perceptually uniform
• Non-linear mapping from XYZ to L*a*b* space
L*a*b* Color Space
 Color Cameras
• Spectral response function

• Make sure to generate the standard color values

HDTV, new monitors, new standard ITU-R BT.709

 Color Filter Arrays
• Separate sensors for three primary colors

Bayer RGB pattern: (a) color filter array layout; (b) interpolated pixel values
 Bayer Pattern, 1976
• Green filters are twice as many as red and
blue filters
• Human visual system is much more sensitive
to high frequency detail in luminance than
chrominance
• Luminance is mostly determined by green
value
 Color Balance
• Move the white point of a given image closer
to pure white (R=G=B)
– Multiply RGB values by a different factor
– Color twist, general 3x3 transform matrix
– Exercise 2.9 (optional)
 Gamma
• CRT Monitor: non-linear relationship between
the voltage and the resulting brightness is
determined by gamma
  2.2

• Pre-map the sensed luminance Y through an

inverse gamma
1
 0.45

 Gamma Compensation

Noise added during transmission or quantization will be reduced in

the darker regions of the signal where it was more visible
 Other Color Spaces
• XYZ, RGB for spectral content of color signals
• Others for image coding and computer
graphics
– YUV, YCrCb, HSV
 YUV Color Space
• YUV for video transmission
– Luma

– Two lower frequency chroma channels

 YCrCb Color Space
• Closely related to YUV
• Different scale factor to fit within the 8-bit
range for digital signals

• Useful for careful image de-blocking, et al.

 HSV Color Space
• Hue: direction around a
color wheel
• Saturation: scaled
distance from the diagonal
• Value: mean or maximum
color value

More suitable for color picking

 Color Ratios
• Suitable for algorithms that only affect the
value/luminance and not saturation or hue

• After processing, scale rgb back by the color

ratio Ynew/Yold

Color FAQ, http://www.poynton.com/ColorFAQ.html

 2.3.3 Compression
• Converting signal into YCbCr (or related
variant)
• Compress the luminance signal with higher
fidelity than the chrominance signal