Light and

Shading
CSE 803– Fall 2024, MSU

Xiaoming Liu

Thank many researchers who have made their slides and course materials available
 Recap: Projection
Image 𝑷= 𝑲 𝑹, 𝒕 𝑿 World
Intrinsic Extrinsic
X
Photo. Material

Source: D Fouhey
 Recap: Lenses
Pinhole Model Reality: Lenses

Photosensitive Material

Mathematically correct Necessary in practice

Not quite correct in practice Introduce complications
Reasonable approximation Complications fixable

Source: D Fouhey
 Today

• A little bit about light and how you represent it

• A little bit about lighting and how it works
 Your Very Own Camera

Controls pupil

Aperture letting in light

Lens…

Where’s the film/CCD?

Slide Credit: NIH
 Your Very Own Camera

Where’s the film/CCD?

Slide Credit: NIH
 What is Retina/Film Made Of?
Cross-section of eye Cross section of retina

Pigmented
epithelium
Ganglion axons
Ganglion cell layer
Bipolar cell layer

Receptor layer
Slide Credit: J. Hays
 Two Type of Photo Receptors
Cones
cone-shaped
less sensitive
operate in high light
color vision

Rods
rod-shaped
highly sensitive
operate at night
gray-scale vision

Slide Credit: J. Hays

 Rod / Cone Sensitivity

Slide Credit: J. Hays

 Rod/Cone Distribution

Diagram Credit: B. A. Wandell, Foundations of Vision

 Electromagnetic Spectrum

Why do we see light in these wavelengths?

Slide Credit: J. Hays
 The Physics of Light
.

A. Ruby Laser B. Gallium Phosphide Crystal

# Photons

# Photons
400 500 600 700 400 500 600 700
Wavelength (nm.) Wavelength (nm.)

C. Tungsten Lightbulb D. Normal Daylight

# Photons

# Photons

400 500 600 700 400 500 600 700

Slide Credit and Copyright: S. Palmer

 The Physics of Light

Red Yellow Blue Purple

400 700 400 700 400 700 400 700

Slide Credit and Copyright: S. Palmer
 Artificial Cones

Estimate RGB
at ‘G’ cells from
neighboring
values

Slide Credit: S. Seitz

Slide by Steve Seitz
 Color Image

Slide Credit: J. Hays

 Color Image
Combined Red

Green Blue

Slide Credit: J. Hays

Images in Python
R
G
B
 Images in Python
Images are matrix / tensor im R
im[0,0,0]
G
top, left, red

im[y,x,c]
B
row y, column x, channel c
im[H-1,W-1,2]
bottom right blue

Slide inspired by James Hays

 5 Things To Always Remember

1. Origin is top left

2. Rows are first index (what’s the fastest
direction for accessing?)
3. Usually referred to as Height x Width
4. Typically stored as uint8 [0,255]
5. for y in range(H): for x in range(W): will run 1
million times for a 1000x1000 image. A 4GHz
processor can do only 4K clock cycles per
pixel per second.

Source: D Fouhey
 Representing Colored Light

Discussion time: how many numbers do you

actually need for colored light? Assume all tuples
(R,G,B) are legitimate colors (they are).
Image Credit: http://en.wikipedia.org/wiki/File:RGB_illumination.jpg
 One Option: RGB
Pros Cons
1. Simple 1. Distances don’t
2. Common make sense R
2. Correlated

Slide Credit: J. Hays, RGB cube: https://en.wikipedia.org/wiki/RGB_color_model

 RGB

Photo credit: J. Hays

 Another Option: HSV
Pros Cons
1. Intuitive for 1. Not as good as H
picking colors other better (S=1,V=1)

2. Sort of common spaces

3. Fast to convert

S
(H=1,V=1)

V
(H=1,S=0)

Slide Credit: J. Hays, HSV cylinder: https://en.wikipedia.org/wiki/HSL_and_HSV

 HSV

Photo credit: J. Hays

 Another Option: YCbCr/YUV
Pros Cons
1. Great for 1. Not as good as Y
transmission / other better smart (Cb=0.5,
Cr=0.5)
compression color spaces

Y=0 Y = 0.5 Cb
(Y=0.5,
Cr=0.5)

Cr
(Y=0.5,
Cb=05)

UV planes
Slide Credit: J. Hays, YUV cube: https://en.wikipedia.org/wiki/YUV
 YCbCr

Photo credit: J. Hays

 Another Option: Lab
Pros Cons
1. Distances 1. Complex to L
correspond with calculate (don’t (a=0,b=0)

human judgment write it yourself,

2. Safe lots of fp
calculations)
a
(L=65,b=0)

b
(L=65,a=0)

Slide Credit: J. Hays, Lab diagram cube: https://en.wikipedia.org/wiki/CIELAB_color_space

 Lab

Photo credit: J. Hays

 Why Are There So Many?

• Each serves different functions

• RGB: sort of intuitive, standard, everywhere
• HSV: good for picking, fast to compute
• YCbCr/YUV: fast to compute, compresses well
• Lab: the right(?) thing to do, but “slow” to compute
• Pick based on what you need and don’t sweat
it: color really isn’t crucial
 Only Images

• Almost all of this class is about ordinary RGB

images because this has driven a lot of
applications
• However, there are lots of other images
 Depth Map
FAR

2.3m

NEAR

Source: D Fouhey
 Surface Normals

Room
[0.06,0.99,0.12]

Legend
Source: D Fouhey
 Science Data
Magnetic Field in:
x, y, z
Light at 9 ~wavelenths:
9.4nm, 13.1nm, 17.1nm
19.3nm, 21.1nm, 30.4nm
33.5nm, 160nm, 170nm
NASA Solar Dynamics
Observatory observing solar flare
 Volumes

Volumes: images with

more dimensions.

Emerge in 3D
reconstruction,
medical imaging,
temporal data

From: Girdhar et al., Learning a predictable and generative vector representation for objects.
ECCV 2016
 Other Images

• A small part of computer vision in this class is

really only for ordinary images
• The rest is easily generalized to other images
• Really transformative stuff will happen when
good vision techniques get traction in other
areas
 So Far

How do we represent light

and its storage on film?
Photo. Material

Source: D Fouhey
 Now

How does the scene

cause that light?

Now
Photo. Material

Source: D Fouhey
 Light and Surfaces
What happens when
light hits a surface?

Surface

Source: D Fouhey
 Light and Surfaces
What happens when
light hits a surface?
1. Absorbed
It’s absorbed and
converted into some other
Surface
form of energy (e.g., a
black shirt getting hot in
the sun)

Source: D Fouhey
 Light and Surfaces
What happens when
light hits a surface?
2. Transmitted
Possibly bouncing around
before going through or
Surface
out (e.g. lenses bend and
go through, milk bounces
around)

Source: D Fouhey
 Light and Surfaces
What happens when
light hits a surface?
3. Reflected
It’s reflected back, in one
or more directions with
Surface
varying amounts (e.g.,
mirror, or a white surface)

Source: D Fouhey
 Light and Surfaces
What happens when
light hits a surface?
4. Everything
All of the above! Real
surfaces often have
Surface
combinations of all of
these options.

Source: D Fouhey
 Modeling Light and Surfaces
Opaque Reflections
Bi-directional reflectance
function: % reflected given
incident angle to light
reflected angle to the
Surface
viewer.

Note: have not specified

𝜙! , 𝜃! 𝜙" , 𝜃" form of function.
Source: D Fouhey
 Specular and Diffuse Reflection
Same lighting, as close as possible camera
settings, but different location

Source: D Fouhey
 Specular and Diffuse Reflection
Diffuse Specular

Totally
Basically different
same

Source: D Fouhey
 Diffuse Reflection
Lambertian Surface
Light depends only on
orientation of surface
ϕi, θi
to light. Result of random
Surface
small facets. Looks
identical at all views.

𝜙! , 𝜃!
Source: D Fouhey
 Diffuse Reflection
N Lambert’s Law
N: surface normal
𝜃! S: source direction and
S strength
ρ: how much is reflected
Surface 𝐵 = 𝜌𝑵 ⋅ 𝑺
𝐵 = 𝜌 𝑺 cos 𝜃
Source: D Fouhey
 Specular Reflection
Specular Surface
Light reflected like a
mirror, but spreads out
in a “lobe” around the
reflection ray
Surface

Source: D Fouhey
 Specular Reflection
Phong Model
V V: vector to viewer
R: reflection ray
R α: shininess constant

𝐵= !
𝑉 𝑅 "
Surface

Source: D Fouhey
 BRDFs can be incredibly complicated…

Slide Credit: L. Lazebnik

 What Can This Be Used For
Shape from Shading
Lambert’s Law: for every pixel i
𝐵# = 𝜌𝑵# ⋅ 𝑺

Reflected Surface Illumination

Light Orientation Global,
(1 dim) (3? dim) (3 dim)
Given: illumination and light, recover normals
Potential problems?
Source: D Fouhey
 Shape From Shading
𝐵# = 𝜌𝑵# ⋅ 𝑺

1D, fixed actually 2D 3D, fixed

unknown

• System of equations that’s underdetermined

(N equations, 2N unknowns, N+3 known)
• Solution: Add more equations that enforce
smoothness or finding a single surface.
Source: D Fouhey
 Realistic Shape From Shading
𝐵# = 𝜌𝑵# ⋅ 𝑺

1D, fixed 2D 3D, unknown

unknown

• System of equations that’s underdetermined

(N equations, 2N+3 unknowns)
• Solution: need prior beliefs to disambiguate.
Source: D Fouhey
 Extended SfS
Given a photo collection of unconstrained 2D face
images, reconstruct a 3D face model.

55
Roth, Tong, Liu, CVPR 15,16, PAMI 17
Algorithm Overview

56
 Photometric Stereo
Lambertian reflectance model
I x = ρ x (ka + kd ℓ ⋅ n x )
Image Albedo Ambient Diffuse

57
 Photometric Stereo: Illumination Matrix
• Each row is the intensity of the projected shape.
• Missing data for obscured locations.
• Use matrix completion* to fill in missing data.

F M

p
…

* Lin et al., The augmented lagrange multiplier method for exact recovery of corrupted low-rank matrices. UIUC Tech. Report,
2009. 58
 Photometric Stereo:
Lighting / Shape Estimation

• Decompose illumination matrix, M, into

• LT: Lighting (nx4)
• S: Shape (4xp)

M LT
= x
S

ρ( 1, nx, ny, nz )T
( ka, kdlx, kdly, kdlz )

59
Results: Jennifer Lawrence

60
 Shape from Shading in Practice

https://www.youtube.com/watch?v=4GiLAOtjHNo
3min video