Lecture 6

Spectral Image Formation

Illumination
200 150 100 50 30 20 10 400 500 600 700 0 400 500 600 700

Color Signal

Image Formation
Radiometric Units Photometric Units Image Formation Model Illuminant Linear Approximations Surface Linear Approximations

Reflectance
1 0.75 0.5 0.25 0 400 500 600 700

Cone Sensitivities
1 0.75 0.5 0.25 0 400 500 600 700

Cone Absorbtions
1 0.75 0.5 0.25 0 L M S

Radiometric Units
Radiant Flux Joules/Sec = Watt
Light emitted from a source (in all directions) .

Radiometric Units
Irradiance Watt/m2
Light density incident on a plane (from all directions).
(How many photons reach a given surface area in a given amount of time).

Radiant Intensity (Density) Watt/sr (sr = steradian)

Light emitted from a source per solid angle .

Radiance Watt/m2/sr
Power per unit solid angle per unit area.
(Radiance is independent of distance).

Radiance and Irradiance Units

Term Radiant Flux Radiant Intensity Irradiance Defining Equation Application
Total quantity of light emitted from a point Total quantity of light emitted from a point in a given solid angle Light density incident on a plane Light emitted or reflected from an extended source in a given direction

Radiometry Photometry
Source Radiance
Symbol: L Units: W/(sr m2)

SI Unit

Source Luminance
Symbol: Lv Units: lm/(sr m2)

Q F= t
I= F

watt

watt /sr

0.8 0.6

E=

F Ar

watt /m2

Radiance

L=

I As cos( )

watt

/sr/m2

y 0.4 0.2 0 300

400

500

600

700

800

- wavelength [nm]

Q = energy (joules) t = time (sec) = solid angle (steradian) A = area (meter 2 ) = angle incident to plane
watt=joule/sec
http://www.calculator.org/properties/luminance.prop

Luminous Efficiency Function

Courtesy P. Catrysse

Radiometry Photometry
X v = K m X l ( ) V ( )d
Photometric term Radiometric term

Radiometry Photometry
X v = K m X l ( ) V ( )d
For X, we can pair the Radiometric and Photometric pairs:

QUANTITY

RADIOMETRIC

PHOTOMETRIC

V() is the Photopic Luminous Efficiency function ( Y() ).

This equation represents a weighting, wavelength by wavelength of the radiant spectral term by the visual response at that wavelength. The constant Km is a scaling factor = 683 lm/W.

power power per unit solid angle power per unit area power per area per solid angle

Radiant Flux watt (W)

Luminous Flux lumen (lm)

Radiant Intensity Luminous Intensity W/sr lm/sr = candela (cd) Irradiance W/m2 Radiance W/m2/sr Illuminance lm/m2 = lux (lx) Luminance lm/m2/sr = cd/m2 = nit

Basic Unit in Photometry is the Lumen and the Candela Monochromatic light 555nm with radiant intensity of 1 Watt = 683 Lumens. Monochromatic light 555nm with radiant intensity of 1 Watt/sr = 683 Candela.

Luminance and Illuminance Units

Term Defining Equation Application SI Unit

Photometry of Scenes: Illuminance

Typical illuminance produced by various sources (lux = lm/m2) Direct sunlight Open shade Overcast/dark day Twilight Full moon Starlight Dark night 110,000 11,000 110 - 1,100 1.1 - 11 0.11 0.0011 0.00011

Total quantity of Luminous Fv = K m Fe ( )V ( )d light emitted from Flux a point

lumen candela (cd) lumens/m2 (lux)

Luminous Intensity Illuminance

Iv = Ev =

Fv Lv Ar

Total quantity of light emitted from a point in a given solid angle Light density incident on a plane Light emitted or reflected from an extended source in a given direction

Luminance

Lv =

I As cos( )

cd/m2

Q = energy (joules) t = time (sec) = solid angle (steradian) A = area (meter 2 ) = angle incident to plane
watt=joule/sec
http://www.electro-optical.com/whitepapers/candela.htm

Courtesy P. Catrysse

Photometry of Scenes: Luminance

Luminance of outside scenes (cd/m2) Sun 6x108 Visual saturation 49,000 Just below saturation 25,000 Outdoor building faade 10,000 Blue sky (morning) 4,600 Concrete sidewalk in sun 3,200 in shadow 570 in deep shadow 290

Photometry of Scenes: Luminance

Intensity Ranges (orders of magnitude):

Natural Light 12 Natural scene 4

Human Visual System: operating range 14 Single view 5

Luminance of interior scenes (cd/m2) Interior room (fluorescent lighting) floor/walls 90 in shadow 10 Interior room (no lighting) floor/walls in shadow in closet door

Technical devices (e.g. displays) : Absolute Dynamic range 2

30 5 1

Spectral Image Formation

Illumination
200 150 100 50 0 400 500 600 700 30 20 10 0 400 500 600 700

Illuminants (CIE standard illuminants)

A - tungsten (2856K)
200 150 100 50 0 400 500 600 700

Color Signal

B - Direct sunlight (4870K)

200 150 100 50 0 400 500 600 700

C - Avg sunlight (4870K)

200

D65 - Avg daylight (6500K)

200 150 100 50

Reflectance
1 0.75 0.5 0.25 0 400 500 600 700

Cone Sensitivities
1 0.75 0.5 0.25 0 400 500 600 700

150 100 50 0 400 500 600 700

400

500

600

700

E - flat spd (5500K)

200 150 100 50 0 400 500 600 700

Cone Absorbtions
1 0.75 0.5 0.25 0 L M S

Surface Reflectances
Yellow Red

Dichromatic Reflection Model

1 0.8 0.6 0.4 0.2 400 1 0.8 0.6 0.4 0.2 400

1 0.8 0.6 0.4 0.2

Incident light

Interface reflection Body reflection

Body reflection
500 600 700

500

600

700

400 1 0.8 0.6 0.4 0.2

Blue

Gray

Colorant particles (pigments)

500 600 700

500

600

700

400

Wavelength (nm)

(Shafer 85)

Reflection Model
Incident light Interface reflection Body reflection

Interface reflection - mirror like reflection at the surface Body reflection - reflected randomly between color particles. Reflection is equal in all directions

Types of Surfaces: Specular surface = Interface reflection is non-zero object appears glossy. Lambertian surface (matte) = surface with no interface reflection, only body reflection. Interface reflection - reflects all wavelengths equally and in the same direction, thus this reflection takes on the same color as the illuminant (and the same SPD).

Spectral Image Formation

Illumination
200 150 100 50 0 400 500 600 700 30 20 10 0 400 500 600 700

Image Formation Equation

Assuming Lambertian Surfaces e() Illuminant s() Surface reflectance l(),m(),s() Cone responsivities

Color Signal

Sensors Illuminant Surface

Reflectance
1 0.75 0.5 0.25 0 400 500 600 700

Cone Sensitivities
1 0.75 0.5 0.25 0 400 500 600 700

L = l( )e( )s( )
Output

M = m( )e( )s( ) S = s( )e( )s( )

Cone Absorbtions
1 0.75 0.5 0.25 0 L M S

Image Formation Equation

Assuming Lambertian Surfaces e() Illuminant s() Surface reflectance l(),m(),s() Cone responsivities

Sensors
L M = S L l ( ) L L m( ) L L s ( ) L

Illuminant
O 0 0 e( ) O

Surface
M s ( ) M

End this section!

Output

rhvs = R thvs diag (e)s

Affects of Illumination
1 0.8 0.6 0.4 0.2 0 400 1 0.8 0.6 0.4 0.2 0 400 550 700 550 700

Yellow

1 0.8 0.6 0.4 0.2 0 400 1 0.8 0.6 0.4 0.2

Red

1 0.8 0.6 0.4 0.2

Blue

Affects of Illumination
Illuminant 1 Illuminant 2

550

700

0 400 1 0.8 0.6 0.4 0.2

550

700

Tungsten

1 0.8 0.6 0.4 0.2 0 400 550 700

0 400

550

700

0 400

550

700

1 0.8 0.6 0.4 0.2

Blue Sky

L M S
1 0.8 0.6 0.4 0.2 1 0.8 0.6 0.4 0.2 550 700 0 400

L M S
1 0.8 0.6 0.4 0.2 550 700 0 400

L M S

0 400

550

700

0 400

550

700

L M S

L M S

L M S

Color Constancy

There was a farmer haf a dog and Bingo was his name - oh.... The big brown fox jumped over the lazy black dog. Roses are red Violets are blue sugar is sweet and so are you

White paper - reflects 90% Black ink reflects 2% Indoor: 100 units illumination Outdoor: 10,000 units illumination.

Outdoors, black ink reflects more than white paper indoors yet the ink still looks black.

Color Constancy
Absolute level of cone responses does not define an objects color appearance.

Camera vs Perceived
Camera sensors Camera output

The level of sensor responses relative to responses to other objects in the scene defines the color appearance of an object..

Scene

L M S L M S

Perceived Scene HVS Cones

L M S

Additional Variations due to Camera Sensors

Color Correction

Known illuminant

Camera responses QImaging Kodak Nikon

Visual responses

Estimate
Reflectance functions

Derive

Color Correction
Assume camera sensors are known

Surface reflectance estimation

Acquired Image

Estimate Reflectance

Estimate Illuminance

e() = illuminant s() = surface reflectance color signal

c() = e()s()

Derive new image from reflectances and standard illumination

Derive new image by transforming from estimated to standard illuminance

Problems: 1) There is no way to distinguish between the following illuminant-surface pairs: e() = e() f() s() = s() / f() c() = e()s() = [e() f() ][s() /f()]= c()

Corrected Image simulates surfaces viewed under standard (white) illumination

Corrected Image

Surface reflectance estimation

Illuminant + Surface reflectance Estimation

Linear Models
2) Visual systems receives LMS cone absorption values (or sensor output values) and not SPDs, thus metameric pairs add to the confusion:
Judd et al 64 Cohen 64 Maloney 86 Marimont & Wandell 93
Assume: Likely Illuminants and Surfaces

ri = R () C()
i

( Ri() = Spectral sensitivity of sensor i )

Metameric pair

Represent Illuminants and Surfaces using low dimensional linear representation

c1() = e()s1()

c2() = e()s2()

Ri() C1()

Ri() C2()

Sensor response

Linear Model - Illuminants

Likely Illuminants:
1

Linear Model - Illuminants

Find a basis of SPDs

e1() e2() e3() .

Daylight Tungsten CWF

Relative intensity

0.8

0.6

such that a linear combination gives a good approximation for every illuminant. Chose a linear model that minimizes:

0.4

0.2

0 400

500

600

700

[ e() - i ei() ]2

i=1

Wavelength (nm)

For all illuminants e() .

Judd, MacAdam & Wyszecki (1964) Modeling of Daylight

n = dimensions of the linear model (# of basis finctions)

Standard daylight Model

e( ) = i ei()
i=1 3

Principle Component Analysis (PCA)

Data Set pi

Matrix representation:
e0

e1 e2 e3

1 2 3
e0 = mean(pi) Minimizes (pi e0)2
i

e = Be

Principle Component Analysis (PCA)

Data Set pi

Principle Component Analysis (PCA)

Data Set pi

e1 e0

~ pi - Mean zero data set

Minimizes (pi e0 w1e1)2
i

e1 = ?

Principle Component Analysis (PCA)

Data Set pi

Principle Component Analysis (PCA)

Data Set pi

e1

e1

e2

e2

Finding ei() : Minimizes (pi e0 w1e1 w2e2)2

i

Create covariance matrix

C = pi pit

Diagonalize using Singular Value Decomposition (SVD) : C = UDVt Where D is a diagonal matrix of eigen values and U,V are matrices of eigenVectors. U = V = [ e1() e2() e3() .]

Standard daylight Model

e( ) = i ei()
i=1 3

Simple Illuminant Estimation

e(l) = illuminant to be estimated 3 color sensors: R1() R2() R3() Be = [ e1() e2() e3()] matrix of illuminant basis Given the sensor responses: r = R () e() i = 1,2,3
i
i

140 120

Relative power

e1() = mean

100 80 60 40 20 0 -20 350 450 550 650 750

In matrix representation: e2() e3()

r1 r2 = r3

R1 R2 R3
r=Re

Wavelength (nm)

r = R(Be) = (RBe)
estimate : estimate e :

RBe = matrix 3x3

= (RBe)-1r e = Be

Linear model for Surface Reflectance

Linear models for special sets: inks, geological materials, etc Surface reflectances are relatively smooth, so linear models can be used to approximate. Example: Macbeth Color Checker

Linear model for Surface Reflectance

Chose a linear model, i.e. basis functions si, to minimize:

[ s() - i si() ]2 i=1

Where s is the surface reflectance function = surface coefficients n = dimension of linear model (# of basis functions)

s1 s2 . . . sn

1 . . .

reflectance

Matrix notation:

s = Bs

wavelength

Example: Linear model for Macbeth color checker

Approximating surface reflectance using a linear model

A minimum of 3 basis functions are needed.

0.1 0.05 0 -0.05 -0.1 400 0.1 0.05 0 -0.05 -0.1 3 400 500 500 600

s1()

0.1 0.05 0 -0.05 700 -0.1 400 0.1 0.05 0 -0.05 600 700 -0.1 400 500 500

s2()

600

700

s ()

s4()
600 700

Using 1,2,3 basis functions (top to bottom)

Approximating surface reflectance using a linear model

Surface and Illumination Estimation of a Scene

Simplifying Assumptions: 1) Likelihood of surfaces and illuminants are given (for example using linear models). 2) Illuminant does not change rapidly over the scene. 3) Sensor sensitivities are known.

Macbeth Chart

n=1

n=4

Problem: assuming a 3D linear model for surface reflection and for illuminants, find 3 surface reflection coefficients for each of the p points in the scene. Case 1: Illuminant is known
If the illuminant e is given, then there are 3p measurements and 3p unknowns. For every point :

n=2

n=5

r1 = R1()e() s() = R1()e() jsj()

r2 = R2()e() s() = R2()e() jsj()

j=1 3 j=1 3 j=1

n=3

n=6

r3 = R3()e() s() = R3()e() jsj()

Surface and Illumination Estimation of a Scene r1 = R1()e() jsj()

3

Case 2: Illuminant is unknown, 1 surface is known

if illuminant is unknown: p points in scene 3p sensor responses are given 3p + 3 unknowns Assume surface s() is known.
3

r2 = R2()e() jsj()

j=1 3 j=1 3 j=1

r3 = R3()e() jsj()

r1 = R1() e() s() = R1() j=1 jej() s()

As a matrix equation:

r2 = R2() e() s() = R2() j=1 jej() s()

r = e
where the (i,j) entry of matrix e is

r3 = R3() e() s() = R3() j=1 jej() s()

As a matrix equation:

Ri ( )e( )s ( )
j

r = s
where the (i,j) entry of matrix s is

Solving for : if n=3 then if n>3 then

1 e r

R i ( )e ( )s( )
j

= e r

(e is the pseudo-inverse)

Compute s for the known s(), then solve for :

= s r
Calculate the illuminant:

e( ) = i ei( )
i= 1

e is the surface matrix for illumination e.

Proceed as in Case 1.

Case 3: Illuminant is unknown, no surface is known

Some assumption must be made to solve for illuminants and surfaces.

Option 2: Uniform perfect reflector brightest surface in scene has a flat SPD. (McCann et al 77)
Calculate brightness of every surface (RGB -> Y) Find surface (pixel) rbright of maximum brightness. Calculate s for brightest surface (flat SPD), then :

Option 1: Gray world assumption average of all surface in scene is gray. (Buchsbaum 80, Land 86)
Using linearity: if s1 under e produces response r1 and s2 under e produces response r2 then s1+s2 under e produces response r1+r2 Thus, under gray world assumption, p if

rbright = s
Calculate :
1 = sr

As in Case 2

s
i=1

then

r
i=1

averages to gray
i

Calculate illuminant:

e( ) = i ei( )
i= 1

is the response to a gray surface.

and calculate all reflectances as in Case 1.

Calculate s for gray surface (flat SPD), then : p As in Case 2 ri = s

i =1 1 Calculate : = s ri i=1 p

Calculate illuminant:

e( ) = i ei( )
i= 1

and calculate all reflectances as in Case 1.

Option 3: Variations on Gray world assumption Average of all surface in scene is not exactly gray. The more colors in scene and larger std - the more likely to average to gray.
(Lee 01, Lam 04)

Additional Methods for Illuminant estimation

Intersection of convex sets of possible illuminants (Forsyth 92) Illuminant estimation using additional sensors (Wandell 87) Illuminant estimation using several illuminants (DZmura & Iverson 93) Illumination from specularities (DZmura & Lennie 86, Lee 86, Tominaga & Wandell 89 90) Object-based illumination classification (Hel-Or & Wandell 02)

Illumination from Specularities

Incident light Interface reflection Body reflection

Illumination from Specularities

Dichromatic Surface Model Color signal at any location is a linear combination of Interface reflectance and Body reflectance. 0.8 0.6 0.4 0.2 0
Body Reflectance Interface Reflectance
1

0.8

0.6

g
0.4

0.2

0 0

0.2

0.4

0.6

0.8

0.2

0.4

0.6

0.8

Illumination from Specularities

Illumination from Specularities

1 0.8 0.6

1 0.8 0.6

g
0.4 0.2 0 0

g
0.4 0.2 0 0

0.2

0.4

0.6

0.8

0.2

0.4

0.6

0.8

Object Based Illumination Classification

Object Based Illumination Classification

0.008 0.006 0.004 0.002 0.000 400

Illuminants

Classic approach

Content Based approach

RG(B) values of diff surfaces under 2 illuminants

Illumination Cluster CIE-A

500

600

700

Wavelength (nm)

Illumination Cluster CIE-C

Object Based Illumination Classification

Object Based Illumination Classification

0.008

Illuminants

SPD of Human Skin

Munsell Surfaces o Skin Surfaces +

0.006 0.004

Mahanalobis Distances
0.002 0.000 400

500

600

700

Wavelength (nm)

Clusters under 2 Illuminants: Munsell Surfaces vs Skin Surfaces

Object Based Illumination Classification

Illumination Correction
Change image acquired under one illumination, to appear as if taken under a different illumination.

color correction for images normalization of images computer graphics - computer generated images.
Using linear models:
3

ri = Ri ( )e( )s( )

using s() = si ( )i we have: Illumination Classification Clusters under 8 Illuminants: Munsell Surfaces vs Skin Surfaces
i =1

r = e

where (i,j) entry of e is Ri( )e( )s j ( )

Same surface under two illuminants:

r = e r = e

r = e ( e )1 r
ie a linear transformation of sensor responses.

Illumination Correction
Examples: Tungsten Blue Sky 0.8119 0.2271 0.0550 -0.0803 1.1344 0.1282 0.0429 -0.0755 1.8091

Illumination Correction
Given two images, find the illuminant transformation M between them.

r =

Assume ri and ri are sensor outputs in the two images corresponding to same surface reflectance.

Notice diagonal elements are dominant. X1 0 0 0 X2 0 0 0 X3 Compensation for illuminants using pure diagonal transformation = scaling of sensor responses = Von Kries Coefficient Law

ri = M ri
Build sensor response matrixes:

r1 r2 . . . rn

r1 r2 . . . rn

Solve *

= r1 r2 . . . rn

r1 r2

. . . r n

Illumination Correction
White Balance

ANSI IT8.7 (Kodak-Q60)

Standard CIELAB Printing primaries Vendor area

RGB = (215,253,178)

Apply transformation:
253/215 0

0 0

0 1 0 0 253/178

RGB = (253,253,253)

Columns 1-3, 5-7 and 9-11 have 108 standardized CIELAB values Accuracy to 10 Eab Produced by Kodak, Agfa, others
ftp://ftp.kodak.com/gastds/Q60DATA/TECHINFO.PDF

Altona TestSuite1.2a

Color Constancy is not Complete

Asymmetric Color Matching Experiment Bg1 Bg2

test

match

Memory match or Dichoptic match

ftp://ftp.kodak.com/gastds/Q60DATA/TECHINFO.PDF

Bg

Bg + green

When does color constancy kick in ?

To make match patch appear equal to test, more green must be added. Color Constancy says added green must equal green (as if adding illuminant to patch). In practice: compensation for illuminant is not complete.

Helson 38, Judd 40, Brainard & Wandell 91

Around 3-4 Surfaces (?)