Algorithms and Applications in

Computer Vision
Lihi Zelnik-Manor
[email protected]

`
Color



Color
Readings:
Forsyth and Ponce, Chapter 6
Szeliski, 2.3.2
Slide Credits
Trevor Darrell
Kristen Grauman: 3-48, 50-75, 79-86
Bob Woodham: 49, 87-90
and others, indirectly (Steve Palmer, Brian
Wandell, etc!)
Today: Color
Measuring color
Spectral power distributions
Color mixing
Color matching experiments
Color spaces
Uniform color spaces

Perception of color
Human photoreceptors
Environmental effects, adaptation

Using color in machine vision systems
Color and light
Color of light arriving at camera depends on
Spectral reflectance of the surface light is leaving
Spectral radiance of light falling on that patch

Color perceived depends on
Physics of light
Visual system receptors
Brain processing, environment
Color and light
Newton 1665
Image from http://micro.magnet.fsu.edu/
White light:
composed of about
equal energy in all
wavelengths of the
visible spectrum
Image credit: nasa.gov
Electromagnetic spectrum
Human Luminance Sensitivity Function
Measuring spectra
Foundations of Vision, B. Wandell
Spectroradiometer: separate input light into its
different wavelengths, and measure the energy
at each.
400 500 600 700
Wavelength (nm.)
# Photons
(per ms.)
Spectral power distribution
The power per unit area at each wavelength
of a radiant object

Figure Stephen E. Palmer, 2002
Spectral power distributions
.
#

P
h
o
t
o
n
s
D. Normal Daylight
Wavelength (nm.)
B. Gallium Phosphide Crystal
400 500 600 700
#

P
h
o
t
o
n
s
Wavelength (nm.)
A. Ruby Laser
400 500 600 700
400 500 600 700
#

P
h
o
t
o
n
s
C. Tungsten Lightbulb
400 500 600 700
#

P
h
o
t
o
n
s
Some examples of the spectra of light sources
Stephen E. Palmer, 2002
Spectral reflectances
for some natural
objects: how much
of each wavelength is
reflected for that
surface
Forsyth & Ponce, measurements by E. Koivisto
The color viewed is also affected by the surfaces
spectral reflectance properties.
Surface reflectance spectra
Some examples of the reflectance spectra of surfaces
Wavelength (nm)
%

P
h
o
t
o
n
s

R
e
f
l
e
c
t
e
d

Red
400 700
Yellow
400 700
Blue
400 700
Purple
400 700
Stephen E. Palmer, 2002
The Psychophysical Correspondence
There is no simple functional description for the perceived
color of all lights under all viewing conditions, but ...
A helpful constraint:
Consider only physical spectra with normal distributions
area
Wavelength (nm.)
# Photons
400 700 500 600
mean
variance
Stephen E. Palmer, 2002
The Psychophysical Correspondence
Mean Hue

yellow green blue
#

P
h
o
t
o
n
s

Wavelength
Stephen E. Palmer, 2002
The Psychophysical Correspondence
Variance Saturation

Wavelength
high
medium
low
hi.
med.
low #

P
h
o
t
o
n
s

Stephen E. Palmer, 2002
The Psychophysical Correspondence
Area Brightness

#

P
h
o
t
o
n
s

Wavelength
B. Area Lightness
bright
dark
Stephen E. Palmer, 2002
Color mixing
Source: W. Freeman
Cartoon spectra for color names:
Additive color mixing
Colors combine by
adding color spectra
Light adds to black.
Source: W. Freeman
Examples of additive color systems
http://www.jegsworks.com
http://www.crtprojectors.co.uk/
CRT phosphors
multiple projectors
Superposition
Additive mixing:
The spectral power
distribution of the
mixture is the sum of the
spectral power
distributions of the
components.
Figure from B. Wandell, 1996
Subtractive color mixing
Colors combine by
multiplying color
spectra.
Pigments remove
color from incident
light (white).
Source: W. Freeman
Examples of subtractive color systems
Printing on paper
Most photographic film


Today: Color
Measuring color
Spectral power distributions
Color mixing
Color matching experiments
Color spaces
Uniform color spaces

Perception of color
Human photoreceptors
Environmental effects, adaptation

Using color in machine vision systems
Why specify color numerically?
Accurate color reproduction is commercially valuable
Many products are identified by color

Few color names are widely recognized by English speakers
11: black, blue, brown, grey, green, orange, pink, purple, red, white, and
yellow.
Other languages have fewer/more.
Common to disagree on appropriate color names.

Color reproduction problems increased by prevalence of digital
imaging e.g. digital libraries of art.
How to ensure that everyone perceives the same color?
What spectral radiances produce the same response from people under
simple viewing conditions?
Forsyth & Ponce
Color matching experiments
Goal:
What spectral radiances produce same
response in human observers?

Color matching experiments
Foundations of Vision, by Brian Wandell, Sinauer Assoc., 1995
After Judd & Wyszecki.
Observer adjusts weight (intensity) for primary
lights (fixed SPDs) to match appearance of test
light.
Color matching experiments
Goal:
What spectral radiances produce same
response in human observers?

Assumption:
Under simple viewing conditions only test
light affects perception
Ignoring additional factors for now like adaptation,
complex surrounding scenes, etc.
Color matching experiment 1
Slide credit: W.
Freeman
Test light Primary lights
Color matching experiment 1
p
1
p
2
p
3

Slide credit: W.
Freeman
Test light Primary lights
Color matching experiment 1
p
1
p
2
p
3

Slide credit: W.
Freeman
Test light Primary lights
Color matching experiment 1
p
1
p
2
p
3

The primary color
amounts needed
for a match
Slide credit: W.
Freeman
Test light Primary lights
Color matching experiment 2
Slide credit: W.
Freeman
Test light Primary lights
Color matching experiment 2
p
1
p
2
p
3

Slide credit: W.
Freeman
Test light Primary lights
Color matching experiment 2
p
1
p
2
p
3

Slide credit: W.
Freeman
Test light Primary lights
Color matching experiment 2
p
1
p
2
p
3

p
1
p
2
p
3

We say a
negative
amount of p
2

was needed to
make the match,
because we
added it to the
test colors side.
The primary color
amounts needed
for a match:
p
1
p
2
p
3

Test light Primary lights
Color matching
What must we require of the primary lights chosen?
How are three numbers enough to represent entire
spectrum?


1
2 1 2 3
3
e
color e P P P
e
(
(
(
(
(
(
(
(
(

(
(

Primary
lights
spectrum
weights
Metamers
Lights forming a perceptual match still may be physically different
Match light: a combination of primaries
Test light: any light
Metamers: pairs of lights that match perceptually but not physically
Different
spectrum
Same
primary
mixture
weights
How to compute the weights of the primaries
to match any new spectral signal?
e
1
e
2
e
3

?
Given: a choice of three
primaries and a target
color signal
Find: weights of the
primaries needed to
match the color signal
p
1
p
2
p
3

Challenge: we cannot use manual tuning for all colors in the world
Computing color matches
1. Select primaries


2. Collect weights for all
monochromatic lights:




Put them in a big matrix:

( )
( )
( )
1 1
2 1
3 1
c
c
c

(
(
(
(

1 1 1
2 1 2
3 1 3
( ) ( )
( ) ( )
( ) ( )
N
N
N
c c
c c
c c



| |
|

|
|
\ .
Computing color matches
1. Select primaries


2. Collect weights for all
monochromatic lights:




3. Compute the weights of
any color by:
( )
( )
( )
1 1
2 1
3 1
c
c
c

(
(
(
(

( )
( )
1
1 1 1 1
2 1 2 2
3 1 3 3
( ) ( )
( ) ( )
( ) ( )
N
N
N
N
t
c c e
c c e
c c e
t

| |
|
|
| | | |
|
| |
|
| |
|
| |
\ . \ .
|
|
|
\ .
|
|
|
.
|

\
|
=
) (
) (
1
N
t
t
t

Arbitrary new spectral signal is linear combination of the
monochromatic sources.
t
Computing color matches
Ct e =
Color matching functions specify how to match a unit of
each wavelength, so:

(
(
(
(

|
|
|
.
|

\
|
=
(
(
(

) (
) (
) (
) ( ) (
) ( ) (
) ( ) (
2
1
3 1 3
2 1 2
1 1 1
3
2
1
N
N
N
N
t
t
t
c c
c c
c c
e
e
e

Kristen Grauman
Why is computing the color
match for any color signal for a
given set of primaries useful?
Want to paint a carton of Kodak film
with the Kodak yellow color.
Want to match skin color of a
person in a photograph printed on
an ink jet printer to their true skin
color.
Want the colors in the world, on a
monitor, and in a print format to all
look the same.

Adapted from W. Freeman
Computing color matches
Image credit: pbs.org
Today: Color
Measuring color
Spectral power distributions
Color mixing
Color matching experiments
Color spaces
Uniform color spaces

Perception of color
Human photoreceptors
Environmental effects, adaptation

Using color in machine vision systems
Standard color spaces
Use a common set of primaries/color
matching functions
Linear color space examples
RGB
CIE XYZ
Non-linear color space
HSV
CIE LAB


RGB color space
Single wavelength primaries
Good for devices (e.g., phosphors for monitor), but
not for perception
RGB color matching functions
CIE XYZ color space
Established by the commission international
declairage (CIE), 1931
Usually projected to display:
(x,y) = (X/(X+Y+Z), Y/(X+Y+Z))

CIE XYZ Color matching functions
Distances in color space
Are distances between points in a color space
perceptually meaningful?
Distances in color space
Not necessarily: CIE XYZ is not a uniform color space, so
magnitude of differences in coordinates are poor
indicator of color distance.
McAdam ellipses:
Just noticeable differences in color
Uniform color spaces
CIE XYZ
CIE Luv CIE Lab
CIE LAB color space
Established by the CIE in 1948 and then 1976
Goal: perceptually uniform

HSV color space
Hue, Saturation, Value (Brightness)
Nonlinear reflects topology of
colors by coding hue as an angle
Matlab: hsv2rgb, rgb2hsv.
Image from mathworks.com
Today: Color
Measuring color
Spectral power distributions
Color mixing
Color matching experiments
Color spaces
Uniform color spaces

Perception of color
Human photoreceptors
Environmental effects, adaptation

Using color in machine vision systems
Color
Color of light arriving at camera depends on
Spectral reflectance of the surface light is leaving
Spectral radiance of light falling on that patch
Color perceived depends on
Physics of light
Visual system receptors
Brain processing, environment
Human photoreceptors
-Rods responsible for intensity
-Cones responsible for color
-Fovea: small region (1 or 2) at
the center of the visual field
containing
the highest density of cones
(and no rods).
Less visual acuity in the
periphery
Adapted from Seitz, Duygulu
Stephen E. Palmer, 2002
Cones
cone-shaped
less sensitive
operate in high light
color vision
Two types of light-sensitive receptors
Rods
rod-shaped
highly sensitive
operate at night
gray-scale vision

Slide credit: Alyosha Efros
Human photoreceptors
React only to some
wavelengths, with different
sensitivity (light fraction
absorbed)
Brain fuses responses from
local neighborhood of
several cones for perceived
color
Sensitivities vary from
person to person, and with
age
Color blindness: deficiency
in at least one type of cone
Wavelength (nm)
S
e
n
s
i
t
i
v
i
t
y

Three kinds of cones
Human photoreceptors
Possible
evolutionary
pressure for
developing
receptors for
different
wavelengths in
primates
Osorio & Vorobyev, 1996
Trichromacy
Experimental facts:
Three primaries will work for most people if we
allow subtractive matching; trichromatic nature
of the human visual system
Most people make the same matches for a given
set of primaries (i.e., select the same mixtures)
Environmental effects & adaptation
Chromatic adaptation: we adapt to a particular
illuminant
Assimilation, contrast effects, chromatic induction:
nearby colors affect what is perceived; receptor
excitations interact across image and time
Afterimages

Color matching ~= color appearance
Physics of light ~= perception of light
Chromatic adaptation
If the visual system is exposed to a certain
illuminant for a while, color system starts to
adapt / skew.
Chromatic adaptation

http://www.planetperplex.com/en/color_illusions.html
Brightness perception
Edward Adelson
http://web.mit.edu/persci/people/adelson/illusions_demos.html
Edward Adelson
http://web.mit.edu/persci/people/adelson/illusions_demos.html
Brightness perception
Edward Adelson
http://web.mit.edu/persci/people/adelson/illusions_demos.html
Brightness perception
Content 2008 R.Beau Lotto
http://www.lottolab.org/articles/illusionsoflight.asp
Look at blue
squares
Look at yellow
squares
Content 2008 R.Beau Lotto
http://www.lottolab.org/articles/illusionsoflight.asp
Look at blue
squares
Look at yellow
squares
Content 2008 R.Beau Lotto
http://www.lottolab.org/articles/illusionsoflight.asp
Content 2008 R.Beau Lotto
http://www.lottolab.org/articles/illusionsoflight.asp
Content 2008 R.Beau Lotto
http://www.lottolab.org/articles/illusionsoflight.asp
Content 2008 R.Beau Lotto
http://www.lottolab.org/articles/illusionsoflight.asp
Content 2008 R.Beau Lotto
http://www.lottolab.org/articles/illusionsoflight.asp
After images
Tired photoreceptors send out negative
response after a strong stimulus
http://www.sandlotscience.com/Aftereffects/Andrus_Spiral.htm
Source: Steve Seitz
After images
Tired photoreceptors send out negative
response after a strong stimulus
http://www.sandlotscience.com/Aftereffects/Andrus_Spiral.htm
Source: Steve Seitz
Name that color
High level interactions affect perception and processing.
Today: Color
Measuring color
Spectral power distributions
Color mixing
Color matching experiments
Color spaces
Uniform color spaces

Perception of color
Human photoreceptors
Environmental effects, adaptation

Using color in machine vision systems
Color as a low-level cue for CBIR
Blobworld system, Carson et al, 1999
R
G B
Color histograms:
Use distribution of
colors to describe
image

No spatial info
invariant to
translation,
rotation, scale

Color intensity
P
i
x
e
l

c
o
u
n
t
s


Color as a low-level cue for CBIR
Color as a low-level cue for CBIR
Compute distance between histograms:
Intersection

Similar
Different
Color-based image retrieval
Given collection (database) of images:
Extract and store one color histogram per image

Given new query image:
Extract its color histogram
For each database image:
Compute intersection between query histogram and
database histogram
Sort intersection values (highest score = most similar)
Rank database items relative to query based on this sorted
order



Color-based image retrieval
Example database
Example retrievals
Color-based image retrieval
Example retrievals
Color-based image retrieval
http://images.google.com/
Search for similar images. Try:

Buildings
Dogs
Concert
Color-based skin detection
M. Jones and J. Rehg, Statistical Color Models with Application to Skin Detection,
IJCV 2002.

Color-based appearance models for body
tracking
D. Ramanan, D. Forsyth, and A. Zisserman. Tracking People by Learning
their Appearance. PAMI 2007.
Slide credit: L.
Lazebnik
Viewing Colored Objects
Slide Credits
Trevor Darrell
Kristen Grauman: 3-48, 50-75, 79-86
Bob Woodham: 49, 87-90
and others, indirectly (Steve Palmer, Brian
Wandell, etc!)
Today: Color
Measuring color
Spectral power distributions
Color mixing
Color matching experiments
Color spaces
Uniform color spaces

Perception of color
Human photoreceptors
Environmental effects, adaptation

Using color in machine vision systems