Good morning!

Colors
and the Human Visual System
(“The day I found out: There is no reason to be blue…”)
Colors
Elmar Eisemann and the Human Visual System
Computer Graphics and Visualization
Delft University of Technology

Today Today

• How do we see? • How do we see?

– Human visual system - Basics – Human visual system - Basics

• How can we represent colors? • How can we represent colors?

General Overview
– Color spaces – Color spaces

• How do we process colors? • How do we process colors?

– Human visual system – Advanced – Human visual system – Advanced

• What can we do with colors? • What can we do with colors?

 What do we see ? Physical Definition
Observer
• Light is registered •Light = Distribution of power over a spectrum
by our eyes…
and perceived as color

Electromagnetic Wave

Light Spectrum Physical Definition

•Visible colors between •Light = Distribution of power over a spectrum

380 nm (violette) and 720 nm (red)
•Outside visible range
– Below 380 nm : ultra-violet
– Above 720 nm : infra-red
 Simple example Examples of Spectra

Energie
P()
Peak is what
people often
Saturation or Chroma

e2 refer to…

e1
Luminance =
area under the curve 
400 nm Hue 720 nm

Everything we see is light… Everything we see is light…

Observer

Electromagnetic Wave
Spectral
Power
Distribution

Illuminant D65
Reflectance
(nm)
Stimulus Spectrum

Spectral
Power
Distribution

12
 Everything we see is light… Everything we see is light…
…but not all of
Neon Lamp
it, as our eyes
are not perfect.
Spectral
Power
Observer
Distribution

Illuminant F1
Spectral
Reflectance
Power
Spectrum
Distribution Stimulus
Under D65

Spectral
Power
Distribution
Under F1

15

Eye Biology Eye Biology

Ganglion Horizontal
Cells Cells
Bipolar
Cells Rod
Cone

Light
Light

Amacrine
Cells
Retina Optic Nerve
17
Rods Cones
 Eye Biology Eye Biology
•Cones : • Retina consists of Cones & Rods
– Chromatic perception (3 types-LMS)
– Concentrated in center of retina
– 6 to 7 million in retinal center
- 3 times full HD

•Rods :
Periphery
– Achromatic perception Center – fovea Boundary region
Mix of both (more rods)
only cones Mix of both
– Low-light vision
Curcio, C. A., Sloan, K. R., Kalina, R. E., Hendrickson, A. E., 1990. Human photoreceptor topography. J Comp Neurol 292, 497-523

Night Vision Night Vision

• During night, all cats are gray… • Who sees noise in the night?
Cones shut off!

Rods Cones Rods Cones

 During the Day 3 Cone types
• Rods saturate!
• All that we have left are cones…

Rods Cones

•For “color”, cones are the interesting case…

Receptor and Incoming Light Receptor and Incoming Light

• Multiply incoming light and receptor • Sensitivity = probability of excitation

and integrate

Incoming light

Probability that a photon

“gets through”

Report as ratio of
maximal stimulation
Multiplication Receptor
 Eye Biology 3 Cone types

• Visual system combines cone responses

The Spectrum is much larger Today

• How do we see?
– Human visual system - Basics

• How can we represent colors?

– Color spaces

• How do we process colors?

– Human visual system – Advanced

https://www.quora.com/What-is-the-difference-between-microwave-range-and-millimeter-range-of-frequencies
• What can we do with colors?
 What do we want? A map for colors?

• Something like a map of colors • Is a 2D map enough?

Where is black?

Color Map? We only have 3 Cones!

• How many dimensions?

3! Different spectra,
but same perception:
metameres

Screens work with

RGB mask
Printers work with
CMY ink
34
@Nicolas Holzschuch
 Except… some people only have 2… 3 Cone Types > 2 Cone Types

Color blindness

4 Cones exist! Different Species Different View

But it seems that
it is usually not useful…
 Different Species Different View Different Species Different View
Confirmed:
Deer See Ultraviolet, What Does This Mean To Hunters?
Ultraviolet
Good hunter… Bad hunter…

https://bowhunting.net/2019/02/confirmed-deer-see-ultraviolet-what-does-this-mean-to-hunters/

We can distinguish ~10M colors on a screen Today: Cheap screens have 16M colors

• Is that too much?

• values of green mapped out in steps of 10

Today: Cheap screens have 16M colors Today: Cheap screens have 16M colors

• True Color claims to have 16M colors, • True Color claims to have 16M colors,
is that too much? is that too much?

• values of green mapped out in steps of 5 • values of green mapped out in steps of 2

Today: Cheap screens have 16M colors What about a “Screen” Color Space?

• True Color claims to have 16M colors,

is that too much?

• values of green mapped out in steps of 1

Yes, on a standard screen 16M is enough…

What about a “Screen” Color Space? What about Screen RGB?

• Is this a good basis to describe all colors? • Control color on screen with knobs

• Experiments used special lamps:

• Roughly: ~700, ~520, ~420 nm

• Notice: 3 dimensions because of 3 cone types

Red Green Blue

49
@Nicolas Holzschuch

RGB Space RGB Space

• Extremely simple • Extremely simple

• Defined per device • Defined per device
• Direct connection to hardware • Direct connection to hardware

• Easy to understand… • Easy to understand…

• So what is the problem???

 RGB has some issues Color Spaces

• Not device independent • Device dependent: the description of color information is

related to the characteristics of a particular device
• Some colors do not exist on some devices – Set of primaries
– Technology

• Device independent: the description of color information is

not dependent on the characteristics of a particular device

Why was RGB ever used? RGB can be remapped to HLS

• CRT Screens typically used the same • HLS

phosphor to produce light (Hue, Lightness, Saturation)

Energie
P()


• While working on one device RGB can be ok 400 nm 720 nm
 RGB can be remapped to HLS Demo Color Picker
Energie
P()
• HLS
(Hue, Lightness, Saturation) 400 nm 
720 nm

Don’t learn this!

RGB Space Device independence?

• Extremely simple • To make a device independent color

• Defined per device space, we can use a reference.
• Direct connection to hardware
• First attempt:
• Easy to understand • Use lights with ~700, ~520, ~420 nm

• Typically referred to as (linear) sRGB

• Not suited for image transfer
– Typically it is converted into a device
independent representation first
 Making a reference Making a reference

• Screen: ~700, ~520, ~420 nm • Screen: ~700, ~520, ~420 nm

• Control color on screen with knobs • Control color on screen with knobs
• Match pure color (monochromatic) • Match pure color (monochromatic)

Added the option to

Turns out: for some wavelengths, it was change color of the stimulus
impossible to find combinations of RGB

Red Green Blue Red Green Blue Red Green Blue

Why is it enough to match pure colors? Making a reference

• Light is additive! • Screen: ~700, ~520, ~420 nm

• Control color on screen with knobs
• Match pure color (monochromatic)
= +

Hence, we can represent any spectrum

as the sum of monochromatic light
Added the option to
change color of the stimulus

Red Green Blue Red Green Blue

 What about the result? What about the result?

The French The French

way to put way to put
labels…  labels

Values can be negative!

64 65
@Nicolas Holzschuch @Nicolas Holzschuch

Looking for a new reference system! Quantifying Color – XYZ space

•What we want:
Spectral distribution of the light
– 3D color space
x , y , z   CIE color matching functions
– Device independent
– All colors should be covered with values in [0,1]3
• Also means: all coefficients positive
– Linear conversion with respect to sRGB

•Commission Internationale de l’Éclairage

– www.cie.co.at Mimic perception with 2 axes
– 1931 Y=Luminance
Z=~S Cone
…and some more constraints
 XYZ Space sRGB Color Space

• How did they come up with these curves?

3D representation is
complex to draw…

sRGB Chromaticity What about Screen RGB?

r=R/(R+G+B)
• Constant energy: g=G/(R+G+B) The French
way to put
labels… 
Idea:
White = gray
just darker.
Hence,
normalize by
the sum to
eliminate one
dimension

71
@Nicolas Holzschuch
 sRGB Chromaticity sRGB Chromaticity
r=R/(R+G+B) r=R/(R+G+B)
• Constant energy: g=G/(R+G+B) • Constant energy: g=G/(R+G+B)

• What about the

pure sRGB colors?

B R

XYZ Chromaticity CIE 1931 chromaticity diagram

• Create virtual colors around all visible colors

X
x
X Y  Z

Y
G y
X Y  Z

B
 CIE XYZ CIE XYZ
•X, Y, Z is device independent •X, Y, Z is device independent
•Y : roughly luminance (perceived brightness) •Y : roughly luminance (perceived brightness)
•Z : roughly the exitation of S-cones •Z : roughly the exitation of S-cones
•X,Z : chroma values •X,Z : chroma values

• Advantages:
– XYZ covers all visible colors with positive values
– Conversion to RGB is linear
• 3x3 Conversion Matrix
• Screens come with a driver…

How to convert your device to XYZ? CIE 1931 chromaticity diagram

• If your device has 3 emitting elements

(e.g., RGB), map selected colors on X
screen to englobe all producible colors x
X Y  Z

Y
• “Reddest Red”, “Bluest Blue”, “Greenest y
X Y  Z
Green” (“brightest Bright” and “darkest
Dark”)

• This defines a 3x3 matrix

• Invert to go back and forth
 Different Devices Different Devices

• Each device can cover a certain color region • When creating content keep gamut in mind!

Gamut

Pitstop Is such a space perfect for all purposes?

• RGB: • E.g.:
– Simple • “Would this change be visible?”
– But device dependent • “If I compress the information, will I see artifacts?”
• linear sRGB
– Device independent
– But negative values for some wavelengths • Not tied to the visual system yet
• XYZ:
– Device independent
– Easy conversion
– Y represents roughly luminance
– But a rather mathematical construction
 Nobody’s perfect! But why were the curves the same…?

• We all have some bias in our visual system:

– Green is slightly better distinguishable than red

– Blue perception is very weak!
– Typical Luminance

Proof – bluring a single color channel Proof – bluring a single color channel

Blue channel blurred Green channel blurred

 Towards a Perceptual Space Perceptually Uniform Spaces

• Measure perception: •CIE, 1976

116Y /Y•L*a*b*
n  16 si Y /Yn 0.008856 and L*u*v*
1
3

MacAdam’s Ellipses L*903.3Y  /Yn  sinon


a*500 X /X n  Y /Yn   u*13L*(u'u'n )
1 1
3 3

v*13L*(v' v'n )

b*200 Y /Yn  Z/Zn   4X
1 1

– indistinguishable
3 3
u'
X15Y 3Z
colors grouped v' 9Y
X15Y 3Z

– ellipse contour = CIE could not agree on which one to pick…

just noticeable
differences L*a*b has the advantage that axes are almost orthogonal

L*u*v is easier to transform…

Blur test… What are these ab-axes?

• Roughly a, b represent change between

• red-green and blue-yellow

Blurred a Blurred b

• Lab color space

 Localized Perception:
Opposing Colors
Opposing Colors
• Receptors are not working independently!
• Cones response is not interpreted directly
• Instead converted to differences:

Opposing Colors Mach Banding

• Receptors are not working independently! • Local contrast produces false intensity ramps
Extreme Case Color Space Summary

RGB: Simple, but device dependent

linear sRGB: negative values
HLS: Simple for artists
XYZ: The device independent standard
L*u*v* : Simple conversion, based on perception
L*a*b* : Orthonormal basis, slightly more costly

In practice:
anchored in XYZ conversion when needed

Questions Today

• How do we see?
– Human visual system - Basics

• How can we represent colors?

– Color spaces

• How do we process colors?

– Human visual system – Advanced

• What can we do with colors?

Lab and LUV use same Luminance How do we see intensity differences?
White point
• Luminance formula: • First, we need a unit: 1 cd/m2
“Perceivedlight emitted by a
common candle”

• How to get such a formula?

• Show gradients and check linearity:

• New question, similar to color:

What intensity differences can we perceive?

Sensitivity to luminance Consequence of the Weber-law

• Weber-law:
just-noticeable difference is L ΔL
proportional to magnitude 100 cd/m2 1 cd/m2
1 cd/m2 0.01 cd/m2
smallest
Ernst Heinrich Weber
detectable
difference
• Adding or subtracting luminance depends
Background Constant
on the background luminance
luminance

L
Typical stimuli: ΔL

111 112
Make Luminance (more) perceptually uniform Assuming the Weber law

• Fechner’s Law: • The response of the visual system is:

Where R is the perceptual response to L
1
1 R(L)  0
L
dl
dR(L)  L(l)
L(L)
response R

1 Derivative of
response Detection
ΔL threshold

1
R(L)  0
L
dl
luminance L
L(l) • Local contrast is perceived logarithmically!
This tells you “how many different
grays you perceive on the way to L”

114

BUT WAIT! There was a cubic root Brightness Perception is complex…

White point
• Local contrast sometimes logarithmic (Weber)

• Global changes rather cubic root

• Dark values rather linear

• I think you are lying…

• Luminance formula: I remember an
illustration from
last week!
• This is for global brightness appearance
Stevens’ Power Law Cornsweet Effect

Cornsweet Effect Effects of locality

• Explore 2D space of possibilities:

Sinusoid
 Contrast Sensitivity Function Brightness Perception is complex…

• Local contrast kind of logarithmic (Weber)

• Global changes rather cubic root

• Dark values rather linear
-> somewhere in between: simple root!
Contrast

Frequency

Brightness via cubic root Brightness Perception is complex…

• VERY common: Gamma Correction • Local contrast kind of logarithmic (Weber)

• Encoding for compression (e.g., JPEG)
• Global changes rather cubic root
• MAGIC Coincidence: • Dark values rather linear
CRT screen phosphors had Gamma ~2! -> somewhere in between: simple root!

• Your screens still do the same!

• That is why I wrote linear sRGB because the
real definition of sRGB is roughly quadratic
with a gamma in R, G, B
 Stevens’ Power Law CIE 1931 chromaticity diagram

• There is something still to discuss.

Where is white?
Pure colors
• CIE actually
defined
the space to
have white at
around
X=Y=Z=1/3
• Saturation vs. Brightness

Perception of Color depends on Context Big Spanish Castle Illusion

Big Spanish Castle Illusion

Relatively
fast.

Ferwerda 1998

Adaptation state in color spaces?

You need to define a White point

Can take
minutes!

Ferwerda 1998
 CIE 1931 chromaticity diagram CIE 1931 chromaticity diagram

• Let’s start with white. Where is it? • Let’s start with white. Where is it?

Pure colors

• We adapt
throughout
There is a whole
range of whites! the day!
D65 D65
Morning
to evening

Dress illusion Pit Stop

• Color spaces for devices work well

• Perception of color is complex

– Difference between
global (cube root) and local perception (log)

Roughly: Weber Law describes what is going on

for the case of background intensity

What about local effects and adaptation state?

 Eye Biology High-level Processing

• Our brain interprets the reported response

• This high-level processing is not well
understood but has a strong effect.

Completion/Inpainting

• Blind spot –
we are actually all blind at/in some point…
(Where the nerves connect in our eye)

Visual system invents information all the time!

 Who did you see? Higher-Level Processing
Right LGN

Left LGN
Visual
Cortex

LGN = Lateral Geniculate Nucleus 145

Avec la contribution de Fredo Durand

Higher-Level Processing Strawberry Illusion

• Memory has an impact [Delk & Fillenbaum] • Kitaoka: Grey strawberry illusion

• Known shapes appear slightly in known color

 Perception Some small things are known…

• Adelson Checkerboard Illusion • How do you explain this image?

Assumption made by our brain: Light comes from above…

We are trained to recognize certain shapes Pattern Recognition

• Pigeons outperform doctors…

• Levenson RM, Krupinski EA, Navarro VM, Wasserman EA (2015) Pigeons (Columba livia) as
Trainable Observers of Pathology and Radiology Breast Cancer Images. PLoS ONE 10(11):
e0141357. https://doi.org/10.1371/journal.pone.0141357
 Conclusion Today

• Even today: • How do we see?

– Higher-level processing not understood – Human visual system - Basics
– Lower-level some models exist
• How can we represent colors?
– Much more research is needed! – Color spaces

• BUT: Some insights can be used! • How do we process colors?

So let’s look at some more “practical” cases – Human visual system - Advanced

• What can we do with colors?

– Illusions and Illustrations

Color perception depends on: Is the rainbow map a good choice?

– Spectral distribution of stimulus

– Structure of image/surrounding

– Temporal aspects

• Structures arise that are not actually there…

155
 Uniform Color Ramps? My currently favorite map discussion…

• Problem of Rainbow mappings

Opposing: (Blue->Yellow, Red->Green)

https://www.nytimes.com/interactive/2020/10/30/opinion/election-results-maps.html

Are opposing colors a good choice? Good choices?

• Hard to distinguish levels in the middle

Parula colormap

These can be made nicely in perceptual color spaces

E.g., “Go from dark blue to light yellow in unit steps”
 Good Color Ramps? Color Discussions

• http://colorbrewer2.org/ • color.adobe.com
• Color combination community

Spectral Distribution Color perception depends on:

For color composition: Ask artists!

– Spectral distribution of stimulus

– Structure of image/surrounding

– Temporal aspects

[Cohen-Or et al. SIGGRAPH 06]

164
Color Optimization for Clusters [Lee et al. TVCG 2013] Compensation for Surrounding

• Mittelstaedt et al. 2015

• Left: initial input

• Middle: Optimization
• Right: Left image with enhanced contrast

HDR Tonemapping HDR Tone Mapping

Contrast Restoration by Adaptive Countershading
Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel
EG 2007
 Contrast Perception Hybrid Images [Oliva et al. SIGGRAPH06]

• Different images for different distances

Willet et al. Hybrid-image visualization for

large viewing environments. 2013

Color perception depends on: Afterimages

• Adding afterimages can boost brightness

[Ritschel & Eisemann 12]
– Spectral distribution of stimulus

– Structure of image/surrounding

– Temporal aspects

172
 [Didyk, Eisemann, Ritschel, Myszkowski, Seidel – SIGGRAPH 2010]

Subtle Gaze Direction Apparent Resolution

• [Bailey et al. TOG 2012]
Gaze Direction

• Attractive Flicker
[Waldner et al.
TVCG2014]

low res

high res

low res

high apparent resolution

Summary: Acknowledgments

• Human Visual System – Basics

– Eye optics, receptors • Guillaume Thierry
• Color spaces • Fredo Durand
– RGB, HSV, Luv, Lab • Nicolas Holzschuch
• Human visual system – Advanced
– Adaptation, Contrast perception, for some of the material of this session
higher-level processing

• Application examples
– Background Contrast, Temporal integration…
Thank you very much!