Digital Color: Acquisition, Perception, Coding and Rendering

Chapter 1

Colorimetry and Physiology – The LMS Specification ¹

To test and improve the quality of color images, we need to know how the human visual system operates. Colorimetry is a method that quantitatively assesses the changes that the engineer makes to an image. Recent advances in this field are due to a better understanding of visual mechanisms.

This chapter first describes the physiological mechanisms that are transferred from the retina in the eye to the human brain, which produce the physiological perception of color. Then it presents two approaches to colorimetry: first, as recommended by the International Commission on Illumination (CIE), and second, deriving directly from the physiology of the visual system that results in the ability to specify stimuli and color differences. Finally, the chapter outlines the difficulties in defining the appearance of color and the advantages in modeling the human visual system.

1.1. Physiological basis

Light detected by the eye excites the photoreceptors that are photosensitive cells. It produces biochemical changes and yields a signal, which is relayed by different classes of post-synaptic retinal neurons. The post-synaptic retinal neurons are organized in a layer in radial and transversal directions. The information is conveyed along the radial direction of the receptors toward the bipolar cells and then toward the ganglion cells whose axons form the optic nerve.

Horizontal cells and amacrine cells form a transversal network, whose action modulates direct signals. Then, the signal travels in the form of trains of action potentials through the optic nerve toward the visual cortex, where a visual image is formed.

1.1.1. The photoreceptors

1.1.1.1. Spectral sensitivity of cones, the monovariant response of a photoreceptor

Each photon absorbed by a cone triggers a cascade of chemical reactions producing a signal at the output of the cone, regardless of the wavelength associated with that photon. As a result, the amplitude of the response of the cone to light depends only on the number of absorbed photons, and not on the wavelength associated with the photons. While light consist of wavelengths in the visible spectrum with wide energy distributions, the response of a single cone is monovariant, it varies only in amplitude. If the photon has energy close to that required for the isomerization of the photosensitive pigment included in the cone, this is absorbed by the cone. The probability of absorption is determined by the spectral sensitivity of the cone.

The in vitro measurements of the spectral sensitivity of the cones [DAR 83] showed the existence of three families of retinal cones with maxima at 419, 531 and 558 nm. This would correspond to in vivo measurements in a healthy eye, taking into account the filtering effect of the ocular media, which is about 440 nm for S cones sensitive to short wavelengths, 540 nm for M cones, sensitive to middle wavelengths,

Figure 1.1. Spectral sensitivity of the three families of cones

and 565 nm for L cones, sensitive to longer wavelengths. The spectral sensitivity curves are widely spread over the visible spectrum, with the M and L cones being close to each other. It should be noted that there is no retinal cone whose maximum sensitivity lies in the part of the spectrum that appears red to the eye (beyond about 620 nm), indicating that the retinal cones are not simple color receptors. Red, like other colors, is reconstructed by the visual system. The M and L cones give to the eye its maximum light sensitivity of around 555 nm.

1.1.1.2. The retinal mosaic

Through optical or electronic microscopy, it is observed that the retina consists of two types of morphologically distinct photoreceptors: rods and cones. Rods are responsible for the vision at low illumination, and conesathigher illumination. The numerical densityof conesismaximum in the fovea, that is to say in the central area of the retina, where images of the objects that we see are formed, and drops significantly toward the surrounding area. It is also possible to observe the cones in vivo, or at the back of the eye, using adaptive optics that neutralize the aberrations. We can also identify their corresponding families L, M or S [ROO 99, HOF 05]. Among the ten retinas that were examined, it was verified that the S cones were relatively few and that the numerical proportion of L and M cones was on average 2L for 1M, with surprising variations from person to person ranging from 1L for 2M to 16L for 1M, for normal color vision.

1.1.2. Retinal organization

The extraction of the color signal is achieved by comparing the signals from a family of cones with those from another family of cones. This comparison is carried out by post-synaptic retinal neurons.

1.1.2.1. Concept of receptive field of the neuron

Each neuron of the visual system, wherever it is located in the hierarchy of the processing, corresponds to a given area of space seen by the subject. It also corresponds to all the requested photoreceptors, within which the bio-electrical behavior of the neuron is changed. This area is called the receptive field of the neuron.

The receptive fields are small in the fovea, and larger as we move away from it. Each neuron is only a small, circular part of the visual field and the encoding of the signal responsible for each neuron depends on its immediate environment.

A retinal neuron does not perform an absolute coding of the light contained in its receptive field, but a coding related to the light status in the near vicinity. Only a differential signal (contrast) generates a signal in the neuron, which is transmitted to the next neurones in the hierarchy of visual information processing. The contrast may relate both to a difference in light or to a difference in spectral content of light.

1.1.2.2. Two parallel pathways from the retina to the cortex

The chemical contact between photoreceptors, bipolar cells, and horizontal cells is carried out at the terminal portion of the cone and is called synapse. The synapse type determines a fundamental functional dichotomy of the coding of the light signal. Some bipolar cells have synapses that maintain the polarity of the signal coming from the cone, others reverse it. The ON-bipolar cells indicate an increment of light at the center of the receptive field (relative to the surrounding fields). They initiate a neural pathway called the ON pathway. The OFF-bipolar cells indicate a decrease in light at the center of the receptive field (relative to the surrounding fields). They initiate a neural pathway called the OFF pathway. These ON and OFF pathways run in parallel across each unit area of the visual space by encoding all the variations of light and remain independent up to the cortex.

1.1.2.3. At the origin of konio, parvo and magnocellular pathways

Different types of bipolar cells are at the origin of three separate neurophysiological pathways from the retina to the cortex: the koniocellular pathway dedicated to spectral differentiation; the parvocellular pathway dedicated to spectral differentiation and light differentiation; and the magnocellular pathway dedicated to light differentiation.

Midget bipolar (MB) cells are the most numerous. They receive signals from cones L and M. They are distinguished by the type of synapse, one belonging to the ON pathway, others to the OFF pathway (see next section).

For ease of nomenclature, a neuron whose receptive field center is ON (responding to an increase in light), will be encoded by +, and a neuron whose receptive field center is OFF (responding to a decrease in light) will be encoded −. The letter following the sign denotes the majority cone type in the center, either L or M (S cones will be discussed in the next section). Implicitly, the area of the receptive field receives signals from the other family of cones, either M or L, on an antagonist mode. For example, a neuron denoted +L will transmit a signal if the center is brighter than the surrounding area and/or if the spectral composition in the center favors large wavelengths. Thus, the midget bipolar cells are of four main types +L, −L, +M and −M. These neurons are the essential elements of the parvocellular pathway and the main initiators of the spectrum of colors and the range of forms and details.

Specific bipolar cells of S cones are the essential elements of the koniocellular pathway. They perform an encoding of the spectral antagonism by contrasting the signals of short wavelengths (S) to those of larger wavelengths (L and M), but are not involved in the encoding of variations in brightness.

Diffused bipolar cells are not selective of the spectral origin. Some are ON, while others are OFF. These neurons are the essential elements of the magnocellular pathway, which initiates the perception of the motion, flicker and variations in brightness.

1.1.2.4. Functional characteristic of the parvocellular pathway (P)

Mostly, neurons in the parvocellular pathway are sensitive to both spectral variations and light variations, thus combining two perceptually distinct variables: variation in brightness (shape precursor) and variation in chromaticity (color precursor). On the same area of the image, these neurons have two functional organizations of receptive fields: a receptive field sensitive to light variations only and a receptive field sensitive to spectral variations. For example, a neuron +L, indicates an increment of light in the center of its receptive field AND/OR, for an equiluminance stimulation, the neuron indicates that the center is covered with a light of longer wavelength than the one illuminating the surrounding area.

1.1.2.5. Three neural pathways at the end of the retina

Given the broad band of spectral sensitivity of cones, cone antagonism allows us, in addition to a saving of messengers, to decorrelate the signals from the cones and to minimize redundancies within and among the three pathways: magno, parvo, and koniocellular [LEE 99, ZAI 97, SHE 08].

Each cone can simultaneously interact with many types of bipolar cells. This is as if the signal of a cone was specifically transcribed according to the specialization of the neuron that it contacts, thus creating multiple filters of a single piece of information generated by the interaction of the light with the photoreceptor.

In each channel, the ganglion cells collect the signals that come from the bipolar cells modulated by the amacrine cells. These are transmitted to the neurons in the lateral geniculate nucleus of the thalamus without significant functional change and with convergence rates that vary depending on the distance to the fovea. These signals are then projected on the primary visual cortex, where they are combined (summed and/or differentiated). As one goes up the hierarchy of processing, the size of receptive fields increases, taking into account larger areas of the visual field.

Figure 1.2. Schematic representation of three independent neural pathways from the retina to the cortex, from the cones. First level: synaptic level of 3 types of cones, L, M and S (here, minimal version of 3 cones). Second level: the receptive fields of the neurons in the parvocellular pathway combine the signals from L and M cones. Cone antagonism appears as a center-surround organization encoding both light contrast (+ and −) and spectral contrast (L and M) giving rise to four types of units (+L, −L, +M, −M). The receptive fields of neurons in the koniocellular pathway combine the signals of S cones with the signals of the cones that are sensitive to larger wavelengths (L and M) under the form of co-extensive receptive fields that encode the chromatic contrast (S vs LM). The receptive fields of neurons in the magnocellular pathway combine the signals from L and M cones, the center-surround organization encode the light contrast (+ and −). Third level: additive combination of receptive fields of units in the parvocellular pathway, resulting in selective units at the thin chromatic contrast and selective units at the thin light contrast. The encoding of the color contrast is enhanced and refined by the summation of parvo units with the koniocellular units. The encoding of the light contrast is enhanced and refined by the summation of parvo units with magnocellular units (for a color version of this figure, see www.iste.co.uk/fernandez/digicolor.zip)

1.1.3. Physiological modeling of visual attributes related to color

The De Valois and De Valois model [VAL 93] allows us to account for segregation of color and light information mixed in the early stages in the visual pathways. This model distinguishes several successive levels of development of the nerve signal leading to the perception of color.

1.1.3.1. Conesynoptic and antagonism level

This model is based on the existence of three types of cones: L, M and S. The S cones are arranged in a regular grid and are less numerous (for example, by proportion: 10 L for 5 M for 1 S).

Cone antagonism is differential encoding between the center and the surrounding area ofthe receptive field. Each cone features a large number of synaptic sites at its terminal portion. A minimal version of the model suggests that each cone establishes contact with at least eight bipolar cells, four with ON neurons (+L, +M, +S and +LMS) and four with OFF neurons (L, M, −S and −LMS).

1.1.3.2. Third level: perceptual antagonism

Cone antagonism of the second level provides relative, local information of a dichotomous nature (for example, areas of more light or; fewer large wavelengths). The third level reflects the separation of information regarding spectral content and that regarding light content. The signals reaching the cortex are summed taking into account the retinotopic organization (the initial vicinity on the retina is preserved). The summation of the signals of the second level sometimes enhances the contrast of light and sometimes the color contrast. When the units of different spectral origins have the same sign, for example: (+L) + (+M), the light contrast is enhanced. When they are of opposite sign, for example (+L) + (M), their joint action enhances the color contrast signal, forming a receptive field with two color antagonism between the center and the surrounding area, indicating thin variations of the color contrast.

Considering the retinotopic recovery in the cortex, the signals from the S cones are conveyed through the koniocellular pathway. The signals from the magnocellular pathway interact with the signals from parvocellular pathway, and then they refine the color signal and the light signal.

In the visual cortex, four types of information are available. Two relate to the spectral nature of the light and two to its relative intensity between the center and the surrounding area of the receptive field regardless of its spectral nature:

1) the spectral contrast, local, between the large and the shortest wavelengths (L vs M);

2) the spectral contrast, local, between the short wavelengths on the one hand and the largest wavelengths on the other hand (S vs L+M);

3) the local contrast of luminance between the center and the surrounding area of the receptive field of small size (parvocellular);

4) the local contrast of luminance between the center and the surrounding area of the receptive field of large size (magnocellular).

There is a specific associative cortical area for the processing of color information: the area V4. It processes signals from area V1. In this area V4, the receptive fields are large. Thus, the color signals can be interpreted based on the surrounding color context.

The modeling of color vision still needs to be developed to take into account the weight of the signals of the cones, the linear and/or nonlinear mode of summations [VAL 00, GEG 03], the role of the light and color adaptation, [VAL 08] and the nature of color constancy.

1.2. The XYZ colorimetry: the benchmark model of CIE

In colorimetry, light is a stimulus that elicits a perception of color. Colorimetry is a set of methods and data to universally specify colors.

There are two methods to reproduce colors. The additive mixture consists of adding lightbeams. Their images are projected on the retina and their effects are added. This is the method of displaying colors on screens. The subtractive mixture consists of superimposing absorbing dyes on a medium. A part of the radiation is subtracted from the light reflected by the media and, therefore, does not reach the retina. This is the process used in printers.

Colorimetry is based on visual experiments. Experiments show that the majority of colors may be reproduced by the addition of three additive primary colors, red, green and blue, in appropriate amounts. The experiment, called color match, also shows that colors have the properties of vectors, in particular the additive property. Therefore, we apply the laws of linear algebra to colors and additive mixtures of colors. The real additive primary colors consist of the base vectors [R], [G], and [B]. A color is defined by three scalar quantities R, G, and B, called tristimulus values (not to be confused with digital intensities encoding the color of a pixel).

Since the properties of additive mixtures of colors was replicated by different observers, it was decided to establish a reference color specification system. In 1931, the International Commission on Illumination (CIE) adopted the XYZ color system obtained through the linear transformation of the RGB color system. The primaries [X], [Y], and [Z] are virtual. They were chosen so that all colors, including monochromatic radiations, havepositive X, Y and Z tristimulus values, and so that the [Y] axis, alone, carries the luminance. Let and be the spectral tristimulus values or color matching functions. Given a radiation of spectral power distribution (or density) of energy Φ(λ), the tristimulus values X, Y and Z of this radiation are computed as follows:

[1.1]

[1.2]

[1.3]

In case of a display, the function Φ(λ) is given by the measurement of spectral radiance. With the adjustment factor 683, the tristimulus value Y corresponds to the visual luminance of the radiation.

For a reflective material of reflectance ρ(λ), which is illuminated by a source of spectral distribution of energy Φ(λ), the tristimulus values are compared with a white surface of unit reflectance. The tristimulus value Y is in this case equal to the reflectance:

[1.4]

[1.5]

[1.6]

1.3. LMS colorimetry

In colorimetry, light is a stimulus that elicits a perception of color. However, the color vision begins with the absorption of photons by the visual pigments contained in the cones of the retina. Thus, the light, as a stimulus, can be defined by the three signals L, M and S that are generated in the cones, which are the input signals in the visual system.

1.3.1. LMS fundamentals

1.3.1.1. Measurement of the spectral sensitivity of cones by psychophysical techniques

Since their spectral sensitivities overlap, it is impossible to obtain the response of a single family of cones to monochromatic radiation. Instead, a psychophysical method that takes advantage of the reduction to two families of cones among some color-blinds is used. These dichromats feature only S and L cones or S and M cones. The relative spectral sensitivity of L or M cones is measured by temporally alternating between two monochromatic radiations, one serving as a reference, and the other for testing. The pace is fast enough to exclude the S-cones’ response, but slow enough to leave a slight flicker. The flicker sensation is minimal when two radiations have the same visual efficiency. By doing this for all test wavelengths, we obtain the spectral sensitivity of the cone type (either L or M) that followed the flicker.

1.3.1.2. The photosensitive pigments and the cone fundamentals: definition and properties

The absorption phenomena of photons by the molecules of photosensitive pigments are quantum phenomena. It was found that the absorption spectra of all the photosensitive pigments of terrestrial vertebrates, measured in vitro, have almost the same shape on a graph whose horizontal axis is graduated in inverse of the wavelength.

The spectral sensitivity of the cones measured at the entrance of the cornea is called fundamental or cone fundamentals. It is the sensitivity of cones integrated in the eye, including all the filters such as the crystalline lens, the macular pigment and others, which absorb a fraction of the light entering the eye before it reaches the retina. Cone fundamentals should be linked to the color-matching functions through a linear relationship.

The objective measurement of the spectral sensitivity of the cones is recent. For many years, it has been accessible only by indirect experimental methods.

From the late 19th Century, by posing a few assumptions, it was possible to predict the fundamentals from color matches performed by normal and dichromat subjects. The assumptions were the absence of a family of cones among the dichromats, the need for obtaining positive cone spectral responses, and the likelihood of the shape of the absorption spectrum of photosensitive pigments.

The most accurate and complete approach is provided by colorimetry, where each monochromatic radiation is defined by its spectral tristimulus values . It remains to be seen whether the linear relationship maps these values to the spectral response of cone fundamentals.

1.3.1.3. The recommendations of the CIE

The International Commission on Illumination [COM 06] published tables giving the values of relative spectral sensitivity of the fundamentals L, M and S for a field of view of 10° of angular diameter and for a field of 2°, for an average young observer with normal color vision.

Figure 1.3. Experimental color-matching functions and fundamentals linked by a linear relationship

The recommendations incorporate the work of Stockman and Sharpe [STO 00] over several years.

For a field of view of 10° of visual angle, there is an area of diameter 10 cm seen , which is derived from color matches at 57 cm, the experiments performed on 49 observers by Stiles and Burch [STI 59]. These experimental color-matching functions obtained with real monochromatic primaries, red (645.2 nm), green (526.3 nm), and blue (444.4 nm), exhibit high quality. The fundamentals are obtained by linear transformation:

[1.7]

For a field of view of angular diameter 10°, the fundamentals exhibit a maximum of sensitivity at 568.6 nm, 541.3 nm and 447.9 nm.

The values of the color-matching function , tabulated by the CIE are slightly different from what would provide the linear transformation in equation [1.7], but the consequences for colorimetry are negligible. The tables are available on the Website http://www.cvrl.org/.

Unfortunately, for a field of view of angular diameter 2°, high-quality experimental data was not available. However, several physiological factors are altered with the eccentricity. It was decided to adopt a calculation scheme that enables us to predict the fundamentals consistent with known physiological and psychophysical data. The scheme consists of starting from fundamentals of a field of 10°, going up to the absorption of visual pigments included in the cones. This takes into account the absorption phenomena in several structures of the eye, the crystalline lens, the macular pigment and the photosensitive pigment density in the cones, then getting back through the opposite path in order to obtain the fundamentals for a field of view of 2° [COM 06, SCH 07].

Figure 1.4 shows the fundamental calculation scheme validated for a field of view of 2°, and this can be generalized to any field of view of angular diameter between 1 and 10°. The CIE gives all the numerical values and equations necessary for modeling the fundamentals.

Figure 1.4. Derivation procedure of cone fundamentals for a field of view of angular diameter 2°

1.3.1.4. Diversity of observers

About 8% of the male population manifests defects of color vision in various forms and degrees. Women are practically free from that. These subjects, called color-blinds confuse certain colors that most people see clearly. The defect is caused by the absence or modification of a family of photosensitive pigments.

Individual variations are twofold in normal color vision. The numerical variations of cone populations do not significantly affect color vision and color identification, but have an effect on the visibility and the relative brightness of colors. Spectral variations of fundamentals alter color vision. As part of the normal color vision underpinned by three families of cones, there are subfamilies of photosensitive pigments whose maximum spectral sensitivity are slightly offset in wavelength by a few nanometers. Transmission variations of ocular media can also alter the fundamentals.

Recently, according to the color matches from 47 subjects examined by Stiles and Burch, it was proposed to classify the actual observers in seven categories, each accepting color matches that other categories rejected [SAR 10]. These individual variations have an impact on the assessment of colors on novel LED type displays with narrow primaries and could explain discrepancies between individuals.

1.3.2. Application of LMS colorimetry

Many applications directly use the LMS specification of cone excitation. The LMS specification also opens interesting perspectives for analyzing signal processing in visual pathways: color discrimination and color appearance.

1.3.2.1. LMS specification

TheL, M, S specification of a stimulus φλ(λ), based on the amplitude of the excitation of three families of cones, is obtained by a calculation similar to that of tristimulus values X, Y, and Z:

[1.8]

1.3.2.2. The luminance in LMS colorimetry

The luminance is the sum of the activities of two families of cones L and M. The S cones do not contribute to luminance. To obtain the value Y of the luminance, the tristimulus values L and M should be computed using equations [1.8] of section 1.3.2.1, by introducing the factors kL = 1.98 and kM = 1.

1.3.2.3. Real color domain

All the colors ofthe real surfaces are included in aclosed volumeof the space, limited by the position of the optimal colors. In fact, it is physically impossible to achieve surfaces whose spectral reflection factor exceeds 100%. Once this limit is reached, the only way to