Computational Aesthetics in Graphics, Visualization and Imaging (2005)

L. Neumann, M. Sbert, B. Gooch, W. Purgathofer (Editors)

Color Search and Replace

Mark Grundland† and Neil A. Dodgson

Computer Laboratory, University of Cambridge, Cambridge, United Kingdom

Abstract

We present an interactive image enhancement technique to adjust the global color composition of an image by find-
ing and replacing color gradients. We show how color gradient transformations can perform the basic operations
of color editing. To recolor an image, the user designates a mapping of source color gradients to corresponding
target color gradients. Each color gradient can be represented by a spherical parameterization, consisting of its
midpoint color, contrast radius, as well as hue and luminance angles, in order to give the user separate and in-
dependent control over color shift, contrast adjustment, and color variation. Color gradients provide not only a
flexible way of selecting color features but also a powerful way of manipulating image colors, as each mapping
between a source and a target color gradient defines an affine color transformation. To determine the region of
influence of each color mapping, perceptual similarity between colors is evaluated by applying Shepard’s law
of generalization to color differences. Through a feature-based warping approach, our color warping algorithm
applies a continuous, nonlinear, volumetric deformation to the color space in order to approximate the requested
color mappings. By making interactive color correction easier to control, our technique may prove useful in a
variety of color image enhancement tasks in digital photography, video processing, and information visualization.
Categories and Subject Descriptors (according to ACM CCS): I.3.6 [Computer Graphics]: Methodology and Tech-
niques — Interaction Techniques; I.4.3 [Image Processing and Computer Vision]: Enhancement

It’s not blood, it’s red. — Jean-Luc Godard developed graphical search and replace for vector graphics
into a generic approach for composing and editing illustra-
tions. Image search and replace has only recently begun to
1. Introduction
be investigated [Gla03]. Exhibiting structure without syntax,
Search and replace is a fundamental data processing oper- images pose a hard problem. Indeed, the design of an intel-
ation. It involves finding the instances of one element and ligent system for image transformation, which would be at
substituting another element in its place. Search and replace least as powerful as regular expressions for text processing,
is a basic mechanism for propagating change while preserv- could constitute a grand challenge for the convergence of vi-
ing consistency. The user decides what needs to be changed sion, graphics and interaction. To make progress toward that
and what change needs to be made. Automated search and goal, we investigate color search and replace. We present a
replace can be indispensable to certain interactive editing simple interactive image enhancement technique (Figure 1)
tasks. It is best known for helping to make a word proces- for adjusting the global color balance of an image by find-
sor and a database more efficient than a typewriter and a ing and replacing color gradients. To guide the algorithm
filing cabinet. However, from as far back as Sutherland’s in transforming image colors, the user designates a set of
1963 SketchPad [Sut63], drawing systems have enabled the source color gradients to be replaced by a corresponding set
same visual transformation to be applied to all instances of a of target color gradients. Our color correction technique en-
graphical object. In 1988, Kurlander and Bier [KB88] further ables the user to improve the color composition of an im-
age by choosing a harmonious combination of colors and
adjusting their contrasts to highlight the key elements of the
† Contact: Mark @ eyemaginary.com
picture. It can assist in calibrating the effects of scene illumi-
http://www.eyemaginary.com/Portfolio/Publications.html

c The Eurographics Association 2005.
 M. Grundland & N. Dodgson / Color Search and Replace

Figure 1: Image recoloring. The source color gradients (top row) are mapped to the target color gradient (bottom row).

nation, selectively emphasizing scene details, and applying imagination upon the image. These attitudes are a reaction
a color scheme to impart a particular mood or theme. not so much to the physical limitations of the film medium
but rather to the technological limitations of the optical in-
struments used to manipulate it. To orchestrate color com-
2. Motivation
position, photographers and cinematographers still lack ade-
Our aim is to make interactive color correction easier for the quate means both to ask for what they want and to get what
artist to control. Here, we outline the rationale for our work. they asked for. Resisting the temptation to take what they
In so far as an image is designed to communicate expe- are given and declare it to be the manifest truth, they pro-
rience, serving to make the intended impression upon its ceed to apply all manner of make-up, lighting, camera an-
viewer, one can either attempt to accurately reproduce the gles, lenses, filters and film stocks. Even so, it is hard to
actual physical stimulus or to expressively elicit the desired get the colors to look just right, especially in cinema where
psychological response. In the case of color, exact repro- the composition keeps changing twenty-four times a second.
duction is often not possible and the visual artist is charged Electronic imaging can give to the photographer and the cin-
with finding creative ways to compensate. A picture viewed ematographer the fine control over color composition that
in isolation can never quite match the way the colors in the painter and the animator have enjoyed for so long.
the scene appear in their natural context. From sunlight and Color makes visual communication compelling. Often,
starlight, the real world portrays a dynamic range well be- where contrast conveys the message, color evokes the emo-
yond the scope of any other representational medium. To tion. In graphic design [BM91], the principal functions of
capture the splendor of a sunset takes more than a shade of color are to render information visible, legible, memorable
orange. In art, color is seeing and feeling intimately com- and appealing. Color is used to attract and hold attention,
bined. The artist purposefully arranges the simultaneous har- catching the viewer’s eye and directing it to where it needs to
monies and contrasts of a color palette to make the picture go. Each color brings across its own palette of associations
come alive for its viewer, evoking not merely a semblance of and memories, likes and dislikes. These nonverbal seman-
the scene but also a sensation of being there. Even in skilled tics are what makes color composition so difficult to auto-
hands, the result bears the mark of the tools and materials mate. Furthermore, visual artists often apply color in subtle
at hand. While a photograph relies on its subject for its col- ways [Wol01] to exploit the subtleties of visual perception.
ors, each stroke of color on a canvas is there by an express Color mediates important perceptual relations, such as illu-
dispensation from the artist. In the making of a color com- mination and reflection, opacity and transparency, figure and
position, mixing pigments affords a freedom of expression ground, unity and variety. Adjacent colors can appear to in-
that recording light does not yield so readily. As colorists, teract through simultaneous contrast, becoming more intense
the painter and the animator retain a conspicuous advan- or more subdued, popping out or blending together. Alternat-
tage over the photographer and the cinematographer. Many ing colors can appear to vibrate through regular repetition,
photographers still regard a black and white print to be in- shimmering of their accord. As warm and bright colors ad-
herently more evocative than its color counterpart. Others vance while cool and dull colors recede, they add depth to
have responded by hand coloring photographs and even en- the picture. As some colors accord while others clash, they
tire films in search of their personal vision, projecting their impart an ambience to the scene. Artists have developed their

c The Eurographics Association 2005.
 M. Grundland & N. Dodgson / Color Search and Replace

own sophisticated vocabulary [Kri92] to describe the vari- range for each color channel, drawing color mapping curves
ous aspects of color in a composition: tone, chroma, hue, for each color channel, performing color shifts along prede-
contrast, counterpoint, repetition, proportion, scale, balance, termined color axes, and selecting color variations from a
tension, rhythm, articulation, temperament, signification and gallery of precomputed possibilities. Using these color edit-
symbolism. Effective color composition [Kri92] demands ing tools, the user can perform color search and replace but
the exercise of aesthetic judgement, a task more befitting the usually on only one color at a time. The results may depend
aptitudes of an artist than a machine. Traditionally, color im- on the order in which the color changes are applied. The
age processing has been engineered for fidelity rather than outcome can become difficult to control, especially when the
expression, concentrating on automation rather than inter- same color is affected by multiple color transformations. Our
action. We undertake a complementary approach to image color search and replace technique avoids these problems by
recoloring by giving the user better control over the process performing multiple color substitutions simultaneously.
of interactive color correction.
As some texture editing tools [HJO*01, BD02, BCD03]
can be used to find and replace textures, they could also be
3. Related work harnessed for color editing tasks, but only where the image
exhibits the self-similar characteristics of a texture. They
Interactive approaches to global color correction differ in hinder the making of coherent color changes across lumi-
the level of user control that they support. Color transfer nance, color and texture boundaries, which may or may not
[WSM99,RAG*01,MS03,CSN03,GH03,PNS03,GD05] de- be desirable depending on what the user is trying to achieve.
mands the least interaction. To impose a specific color distri- This dilemma is inherent in interactive image recoloring
bution on an image, these techniques extend grayscale his- over an image segmentation [RB02]. It also affects power-
togram specification to color histogram specification. They ful methods that combine region selection and color editing
transform the colors of the source image to resemble a tar- into a single operation [LLW04], where the user paints col-
get color scheme, usually exemplified by a reference image ors over the image with a few rough brush strokes and the al-
which the user supplies. To selectively apply color transfor- gorithm propagates the indicated color changes with respect
mations, the user can always manually select source image to the image boundaries. For instance, to recolor a striped
regions and designate different target color schemes for them shirt, the user needs to annotate the color change for each
[RAG*01]. Color transfer may be accomplished by either stripe separately. Better interactive methods for grouping im-
the 3D transformation of color space [PNS03], the 1D trans- age regions can make annotating color changes somewhat
formation of color axes [WSM99, RAG*01, GD05], or the easier [RB02, BCD03]. We focus on making global color
discrete mapping of color palettes [MS03, CSN03, GH03]. changes easier to control, while leaving image region selec-
In practice, color transfer remains a hit-and-miss operation. tion to tools dedicated to the task.
The common flaw of all the color transfer methods is that
To the best of our knowledge, the closest precedent for
they do not give the user sufficient control to ensure consis-
our color search and replace technique is the Hardeberg et
tently reliable results. Unlike color search and replace, color
al. color substitution method [HFK*02]. The authors present
transfer relies on its algorithm rather than its user to decide
their work in the context of color management tools, survey-
which source color should be mapped to which target color.
ing the color transformations used in color gamut charac-
For instance, a user has no direct way to tell a color trans-
terization. Their system supports color calibration of digi-
fer algorithm to turn a red traffic light green, not without
tal video through the transfer of color properties from one
running the risk that the street lights will turn green instead.
video source to another. Similar to our technique, it enables
The algorithm can give no assurance of producing a mean-
the user to simultaneously find and replace multiple image
ingful color mapping since it lacks understanding of how
colors, performing the desired color substitutions using vol-
color lends meaning to an image. With color transfer, it is
umetric color warping. However, as discussed in the next
difficult to exchange colors, to adjust contrast, or to specify
section, there is a crucial difference that distinguishes our
a target color scheme without access to a ready made exam-
color search and replace technique from their method.
ple of it. Color search and replace resolves these issues by
placing the user in charge of defining the color mapping.
4. Interaction
To replicate the effect of our technique, most other color
correction approaches demand more laborious user interac- Compared with previous interactive color correction meth-
tion. Methods exist for color adjustment of designated hue ods, what renders our color search and replace technique
ranges [IT97], color emphasis of designated colors [GD05] uniquely effective is that it operates on color gradients rather
as well as volumetric editing of color gamuts visualized than just individual colors. A linear color gradient is a
in color space [FH02]. Image processing systems, such as smoothly varying sequence of colors defined by a line seg-
Photoshop, offer powerful tools for selecting image regions, ment in color space. While an individual color can only de-
including selection by color. They also provide facilities note a color position, a color gradient also designates a con-
for making color changes, including adjusting the dynamic trast magnitude and orientation. In the degenerate case, when

c The Eurographics Association 2005.
 M. Grundland & N. Dodgson / Color Search and Replace

b components. Experiments in color selection [MSK04] re-

Adjustment Parameter Transformation port that Lab seems preferable to RGB for rendering aesthet-
color (L, a, b) translation ically pleasing color gradients. For instance, in a Lab color
luminance L translation gradient between saturated hues, the midpoint color appears
color contrast ρ uniform scaling more subdued than the endpoint colors, exhibiting their aver-
luminance contrast ρL nonuniform scaling age luminance as well as a lower saturation. However, comb-
color inversion −ρ reflection ing equal measures of complementary hues need not produce
hue variation θ rotation a neutral gray in Lab since, unlike the additive RGB and sub-
luminance variation φ rotation tractive CMY color spaces, Lab color interpolation does not
obey Grassman’s laws of color mixing. In general, the Lab
color space appears better at modeling small color differ-
Table 1: Summary of color gradient transformations.
ences than large ones. A disadvantage of color interpolation
in Lab is that typical display and printer color gamuts, in-
cluding sRGB, do not form convex polytopes. It is possible
the endpoints of its color span are equal, a color gradient col- for some intermediate colors of a highly saturated color gra-
lapses to represent a particular color. In general, a color gra- dient to be out of gamut when its endpoints remain in gamut.
dient represents a particular range of colors. It can describe Whenever we encounter out of gamut colors, we simply clip
color changes from light to dark, neutral to saturated, cooler their color values to fit the dynamic range of the RGB color
to warmer. As color gradients express color differences, they cube. As clipping risks detail loss, better gamut mapping for
offer a natural basis for color editing. This approach is in color search and replace is a subject for future research.
keeping with human color vision, which is specifically tuned A color gradient is a line segment in color space. Nor-
to perceive color differences. mally, it is parameterized by the rectangular color coordi-
Color gradients provide a flexible way both to select and nates of its endpoints, (L1 , a1 , b1 ) and (L2 , a2 , b2 ). In this pa-
to adjust colors. As areas of flat color are largely absent rameterization, though defining a color gradient is straight-
from natural scenes, color gradients prove more convenient forward, adjusting its color properties becomes needlessly
for selecting object features, encompassing variations in sur- difficult. Both endpoints need to be displaced in tandem to
face orientation, reflectance and illumination. While map- perform a contrast preserving color shift or a color preserv-
ping one color to another [HFK*02] is limited to producing a ing contrast adjustment. For directing color changes, a spher-
color shift by a translation in color space, mapping one color ical parameterization proves more convenient. We therefore
define a color gradient’s location G
= (L, a, b) in color space
gradient to another can express a more general affine color
transformation. A color gradient mapping offers the degrees to be its midpoint color, the average of its endpoint colors:
of freedom (Table 1) required for effective color editing, L = 12 (L1 + L2 ) , a = 12 (a1 + a2 ) , b = 12 (b1 + b2 ) .
providing separate and independent control over color shift,
A color gradient’s displacement ∆G
= (∆L, ∆a, ∆b) in color
contrast adjustment, and color variations.
space determines its Euclidean color distance ∆E =
∆G
,




along with its color difference direction ∆Ĝ = ∆G/
∆G
:
4.1. Color gradient specification
∆L = L1 − L2 , ∆a = a1 − a2 , ∆b = b1 − b2 ,
A color gradient is formed by linear interpolation between a √
∆E = ∆L2 + ∆a2 + ∆b2 .
pair of colors. Its appearance is thus dependent on the choice
of color space. Color gradients demand that convex linear We model a color gradient as the diameter of a color sphere,
combinations of color values should produce a proportion- its color neighborhood in color space. Its color contrast ra-
ally linear perceptual response. Our implementation relies dius ρ is a measure of the color difference, while the lumi-
on the Lab color space [Kue03]. Unlike the default RGB and nance contrast ρL is a measure of the luminance difference.
CMY color spaces for displaying and printing images, the Centered at the midpoint color, the azimuthal and polar an-
Lab color space is approximately perceptually uniform. In gles of its spherical color coordinates (ρ, θ, φ) are the hue
Lab, Euclidean distance between color values approximates angle −π ≤ θ ≤ π and the luminance angle 0 ≤ φ ≤ π:
the perceptual difference between them, thereby providing a ∆E ∆L ∆b ∆L
visually plausible color difference metric. This property jus- ρ= , ρL = , θ = arctan , φ = arccos .
2 2 ∆a ∆E
tifies our use of geometric transformations in color space.
This transformation can be reversed by observing:
As equal increments in color values give rise to perceptu-
ally equal color changes, the smoothness of the color gra- ∆L = 2ρ cos φ, ∆a = 2ρ cos θ sin φ, ∆b = 2ρ sin θ sin φ.
dients is assured, enabling the user to make precise color
adjustments. For color specification, the Lab color space has These parameters directly relate affine color transforma-
the advantage of distinguishing the achromatic luminance L tions with the basic color editing operations on a color gradi-
component from the chromatic red-green a and yellow-blue ent (Table 1), including color shift, contrast adjustment, and

c The Eurographics Association 2005.
 M. Grundland & N. Dodgson / Color Search and Replace

Figure 2: User interface for color search and replace.

color variation. Furthermore, these operations can be con- image, enabling the user to sample colors from them. For ap-
strained to act on just luminance and leave hue unaltered. plying a preconceived color scheme, as in color transfer, it
The user can easily change one color gradient property with- could also be helpful to display an additional reference im-
out affecting another, as each parameter controls a differ- age. To assist in color selection, our implementation uses a
ent color editing operation. A color shift need not entail an standard color spectrum as a reference. Below the images,
overall contrast change while a contrast adjustment need not the color gradient mappings are shown ("Edit Gradients").
entail an overall color change. Moreover, the hue and lumi- The top row lists the source color gradients while the bottom
nance variations can produce subtle color effects that cause row lists the target color gradients. Each column corresponds
neither an overall color shift nor an overall contrast change. to a color gradient mapping. The user can add, delete, and
select the color gradient mappings. For the selected color
4.2. Color editing interface gradient mapping, its source and target color gradients are
displayed ("Find Gradient" and "Replace Gradient").
Our user interface (Figure 2) enables the user to edit the
global color composition of an image by performing basic The user can edit a color gradient through either its rectan-
color editing operations (Table 1) on color gradients. For gular parameterization by selecting one of its endpoint col-
color selection, to change one color for another takes two ors or its spherical parameterization by selecting the color
clicks of the mouse, the first selects the old color and the sec- gradient swatch displayed between them. The contrast prop-
ond chooses the new color. For color correction, to map one erties of the currently selected color gradient can be adjusted
color gradient to another takes as little as three clicks of the using the contrast pickers ("Adjust Contrast"). From top to
mouse, the first and the second choose the endpoints of the bottom, they control color contrast, luminance contrast, lu-
source color gradient while the third applies a color editing minance angle and hue angle. Along with an indicator of the
operation to it and thereby specifies the target color gradient. current parameter setting, each contrast picker previews how
Since direct manipulation relies on continuous feedback, the changing the parameter affects the endpoint colors of the se-
results are updated as each color change is made. Our user lected color gradient. Where a parameter change would send
interface displays the original image as well as the resulting one of the endpoint colors out of gamut, the contrast picker

c The Eurographics Association 2005.
 M. Grundland & N. Dodgson / Color Search and Replace

displays a dotted boundary to warn the user, since a contrast perception of color differences. Such an approach could bet-
adjustment that causes the color gradient to be clipped may ter account for the relative importance of luminance ∆L,
also cause a color shift by displacing its midpoint color. The chroma ∆C, and hue ∆H differences:
color properties of the currently selected color gradient can

√
be adjusted using the color pickers ("Adjust Color"). From C1 = a21 + b21 , C2 = a22 + b22 , C̄ = C1C2 ,
top to bottom, their color bars display the range of available √
∆C = C1 −C2 , ∆H = ∆E 2 − ∆L2 − ∆C2 .
colors along the currently selected color gradient followed
by the luminance, saturation, hue, red-cyan, green-magenta, A weighted color difference formula D can accommodate
and blue-yellow color axes. To aid in color space navigation, different luminance κL , chroma κC , and hue κH weights:

each color bar is centered on the currently chosen color. The  2  2  2
color pickers can be applied to the midpoint color or to the ∆L ∆C ∆H
D= + + .
endpoint colors. While the rest of the interface relies on Lab κL κC κH
color interpolation, our color pickers use HSV, RGB, and For the examples shown in this paper, we only required the
CMY color axes because they are likely to be more familiar default setting κL = κC = κH = 1 so that D = ∆E. However,
to our users. However, according to user studies [DK99], vi- for improved perceptual fidelity, the weights could be cho-
sual feedback has much greater impact on accuracy of color sen κL ≤ κH ≤ κC to reflect psychophysical evidence that,
selection than the choice of color space. More advanced in general, luminance exerts greater effect than hue which in
color pickers [MSK04] can incorporate color relationships, turn exerts greater effect than chroma. To accurately model
groupings, contexts, and juxtapositions. Alternatively, syn- small color differences, the CIE94 formula [Kue03] uses
chronized views of orthogonal color planes can allow the κL = 1, κC = 1 + 0.045C̄, and κH = 1 + 0.015C̄. To ac-
user to draw a color gradient in color space [RHO97]. curately model larger color differences, the GLAB formula
The user can fine tune the results in several ways. Glob- [GL99] uses κL = 0.76, κC = 1 + 0.016C̄, and κH = 1. Color
ally, the user can balance the preservation λ of the original gamut mapping experiments
√ [KIO99] suggest that κL = 1,
image colors with the influence ω that all the color gradi- κC = 2, and κH = 2 yields results that are better than ∆E
ent mappings exert over their surrounding colors (by default and about on par with CIE94. Weighted color differences can
λ = 15 and ω = 5). For each color gradient mapping, the user also serve to make color selection more flexible. The weights
can adjust the additional scope σi of its influence over its could be set to reflect the user’s priorities in searching for
surrounding colors as well as the additional contrast empha- colors. For instance, by setting κL = ∞, the user could se-
sis τi of its surrounding colors (by default σi = 1 and τi = 1). lect a range of hues irrespective of their luminance, a useful
Our interface lets the user adjust these values, λ ("Preserva- strategy for dealing with a color cast. Moreover, different
tion"), ω ("Influence"), σi ("Scope"), and τi ("Emphasis"). weights could be used for different color gradient mappings.
In general, color difference formulas are most effective
5. Algorithm at measuring the perceptual similarity between closely re-
We calculate a continuous, nonlinear, volumetric warping of lated colors. Unrelated colors are perceived to be categori-
color space, which is guided by the mapping of source color cally dissimilar. For example, a yellow can be judged more
gradients G
i to target color gradients G

i . We approximate or less similar to an orange but it no more resembles a red
rather than interpolate the user’s wishes to avoid color dis- than a purple. Hence, perceptual similarity should vary lin-
continuities in cases where one color gradient mapping con- early with small color differences, while vanishing entirely
tradicts another. First, the search step uses perceptual color as color differences become excessive. This observation is
similarity to ascertain each source color gradient’s region of expressed by Shepard’s law of generalization [She87], which
influence Si . Second, the replace step derives the linear trans- is based on evidence in many perceptual domains that the
formation Ti that maps each source color gradient to its tar- likelihood of confusing one stimulus with another decreases
get color gradient. Finally, the combine step generates the exponentially as the perceptual distance between them in-
resulting image
P
from a composite of the original image creases. Previously, this approach has been used to improve

P0 and its color gradient transformations
Pi , where the effect feature detection in color image analysis [RT01]. Using scal-
of each transformation is masked by its region of influence. ing factor δ, we evaluate perceptual similarity 0 ≤ S ≤ 1:
 −D 
S = exp .
5.1. Search step: Color similarity evaluation δ
Without reference to Shepard’s law, Hardeberg et al.
To close the gap between color perception and colorimet-
[HFK*02] used a more abrupt scaling of color differences,
ric measurement, many color difference formulas have been
by setting D = ∆E 2 + ln ∆E in the above equation, to deter-
proposed [Kue03]. A standard measure of color difference
mine the influence of a color change on surrounding colors.
between a pair of colors is their Euclidean distance ∆E in the
Lab color space. For some applications, it could be worth- In color search, perceptual similarity facilitates the selec-
while to more carefully consider how chroma C̄ affects the tion of image regions by color. To determine the region of

c The Eurographics Association 2005.
 M. Grundland & N. Dodgson / Color Search and Replace

Figure 3: Contrast enhancement. The source color gradients (top row) are mapped to the target color gradient (bottom row).

influence of each color gradient mapping, we use its percep- gradients, their normal direction is selected as the axis of
tual similarity map Si . Its extent can be adjusted by the user rotation (ΦLi , Φai , Φbi ) = (∆Ĝi × ∆Ĝ
i )/
∆Ĝi × ∆Ĝ
i
, so that
through the combined scaling factor δi = ω σi . At each pixel, αi = cos ϕi = (∆Ĝi · ∆Ĝ
i ) and βi = sin ϕi =
∆Ĝi × ∆Ĝ
i
:
we evaluate the similarity Si between the pixel color
P0 and 
(1−αi )ΦLi ΦLi +αi (1−αi )ΦLi Φai −βi Φbi (1−αi )ΦLi Φbi +βi Φai

the closest color Q
i:

i present in the source color gradient G  L
Φi = (1−αi )Φi Φi +βi Φi
L a b
(1−αi )Φi Φi +αi (1−αi )Φi Φi −βi Φi .
a a a b

Q
i + ri ∆Ĝi for ri = median{−ρi , (
P0 − G

i =G
i ) · ∆Ĝi , ρi }. (1−αi )ΦLi Φbi −βi Φai (1−αi )Φai Φbi +βi ΦLi (1−αi )Φbi Φbi +αi

In the degenerate case, either the source, the target, or both

5.2. Replace step: Color gradient mapping color gradients specify a single color rather than a range of
colors, exhibiting a color difference that is too small to be
A color gradient mapping can be interpreted geometrically noticeable (we set the perceptual threshold to ε = 1.5 Lab
in color space as an affine transformation [Gol03] that maps units). This situation may arise when the user wishes to edit
one line segment to another. Affine color transformations uniformly colored image regions. Such a color mapping des-
have been previously used in automatic color cast removal ignates a color shift, performed by a translation, as well as a
[KKK*01]. For each color gradient mapping, a translation contrast adjustment, performed by a uniform scaling:
maps the source midpoint color G
i to the target midpoint


max{ρ
i , ε}
color Gi while a linear transformation Ti = Φi Ψi combines Ti = τi when min{ρ
i , ρi } ≤ ε.
the scaling Ψi and the rotation Φi required to align the source max{ρi , ε}
color gradient with the target color gradient. At each pixel

P0 , we calculate the effect
Pi of the color gradient mapping: 5.3. Combine step: Color gamut warping

Pi = Ti (
P0 − G

i .

i) + G As each color gradient mapping affects a different color
space region, we apply interpolation to smoothly combine
To adjust contrast, a general scaling [Gol03] transformation these transformations to produce a volumetric warping of
Ψi applies a parallel scaling factor νi along a designated di- the entire color space. We adapt a feature-based warping ap-
rection (ΨLi , Ψai , Ψbi ) as well as a perpendicular scaling fac- proach [BN92] originally developed for image morphing. In
tor τi in the perpendicular plane. The color gradient con- color gamut mapping [SES95], color warping has been pre-
trast ratio νi = ρ
i /ρi specifies the contrast adjustment of the viously applied to ensure a smooth transition between gamut
source color gradient (ΨLi , Ψai , Ψbi ) = ∆Ĝi , while the user pa- mapping strategies for different subsets of the color space.
rameter τi controls contrast in the perpendicular color plane: In another application, the perceptual uniformity of a color
 
τi +(νi −τi )ΨLi ΨLi (νi −τi )ΨLi Ψai (νi −τi )ΨLi Ψbi space was improved by a free-form deformation [TR98],
  which was optimized to make distances between designated
Ψi = (νi −τi )Ψai ΨLi τi +(νi −τi )Ψai Ψai (νi −τi )Ψai Ψbi .
(νi −τi )Ψbi ΨLi (νi −τi )Ψbi Ψai τi +(νi −τi )Ψi Ψi
b b color samples match their psychophysically estimated color
differences. In our color warping approach, we map each
To adjust color, using Rodrigues formula, a general rotation
pixel color
P0 to a new color
P
by compositing the colors
Pi
[Gol03] transformation Φi directs a rotation around a desig-
produced by the color gradient transformations:
nated unit axis (ΦLi , Φai , Φbi ) through a counterclockwise an-
gle ϕi . To align the directions of the source and target color
P
= w0
P0 + ∑ wi
Pi .

c The Eurographics Association 2005.
 M. Grundland & N. Dodgson / Color Search and Replace

Figure 4: Color transfer. The color scheme of the reference image (bottom left) is applied to the input image (left) to produce
the output image (right). The reference and input images were taken from different camera angles at different times of day.

In determining the new color of a pixel, the influence of a In information visualization, color adjustment is often used
color gradient mapping should be proportional to the simi- in post-processing to draw color distinctions that direct the
larity of its source color gradient to the original image color. viewer’s attention to the relevant structure of the data. Also,
Hence, for each color gradient mapping, its perceptual sim- our technique could assist in the mapping between the dis-
ilarity map Si governs its relative contribution wi to the out- parate color gamuts of different capture and display devices.
come of the color warping. However, to the degree w0 that
the original image color differs from all of the source color
7. Conclusion and Future Work
gradients, it should remain unaffected by any of the color
gradient mappings. Our method enables the user to control Visual search and replace is an emerging approach to image
the tradeoff λ between color change and color preservation: editing. We have presented a novel color search and replace
Si technique designed to make interactive color correction eas-
if λ ≥ ∑ Si then wi = and w0 = 1 − ∑ wi , ier to control. The central contribution of our work is to es-
λ
S tablish color gradient transformation as a primitive opera-
if λ ≤ ∑ Si then wi = i and w0 = 0. tion for image recoloring in interactive image enhancement.
∑ Si
Through a spherical parameterization, a color gradient trans-
formation has the flexibility to perform all the basic tasks of
6. Results and Applications color correction, including color shift, contrast adjustment,
We demonstrate our technique on several examples. In Fig- and color variation. For color editing, our user interface of-
ure 1, we amplify subtle variations of shading to dramat- fers separate control over color and contrast changes. To
ically recolor an image. Figure 2 shows how we can edit perform multiple color gradient mappings simultaneously,
the color composition of an image, painting the background our color warping algorithm relies on a perceptual measure
blue and the parrot red. Notice the isolated blue patches of color similarity to determine which colors should be af-
among the parrot’s feathers. Our global color editing tech- fected by which color gradient mappings. In future work,
nique can not distinguish foreground from background when we look forward to addressing how our technique may be
they share the same color. Of course, within an image pro- integrated with other interactive methods for image selec-
cessing environment, the user could select the image region tion [RB02,LLW04] and texturing [HJO*01,BD02,BCD03].
that should receive the color changes. In Figure 3, interactive
contrast enhancement is performed by raising the luminance References
contrast of the chosen color gradients while leaving their
hues unaffected. Figure 4 shows interactive color transfer, [BCD03] B ROOKS S., C ARDLE M., D ODGSON N. A.:
where the source colors are selected from the input image Enhanced texture editing using self-similarity. In Pro-
and the matching target colors are chosen from a reference ceedings of Vision, Video and Graphics (2003), 231-238.
image. This operation could be useful in harmonizing the [BD02] B ROOKS S., D ODGSON N.A.: Self-similarity
colors of film footage taken under different lighting condi- based texture editing. In Proceedings of SIGGRAPH
tions. There are other possible applications of our technique. (2002), 653-656.

c The Eurographics Association 2005.
 M. Grundland & N. Dodgson / Color Search and Replace

[BM91] B ERRY S., M ARTIN J.: Designing with Colour, [Kri92] K RIESBERG I.: Working with Color, Van Nos-
Batsford, 1991. trand Reinhold, 1992.
[BN92] B EIER T., N EELY S.: Feature-based image meta- [Kue03] K UEHNI R. G.: Color Space and Its Divisions,
morphosis. In Proceedings of SIGGRAPH (1992), 35-42. Wiley-Interscience, 2003.
[CSN03] C HANG Y., S AITO S., NAKAJIMA M.: A [LLW04] L EVIN A., L ISCHINSKI D., W EISS Y.: Col-
framework for transfer colors based on the basic color orization using optimization. In Proceedings of SIG-
categories. In Proceedings of Computer Graphics Inter- GRAPH (2004), 689-694.
national (2003), 176-181.
[MS03] M OROVIC J., S UN P.-L.: Accurate 3D im-
[DK99] D OUGLAS S. A., K IRKPATRICK A. E.: Model age colour histogram transformation. Pattern Recognition
and representation: The effect of visual feedback on hu- Letters 24, 11 (2003), 1725-1735.
man performance in a color picker interface. ACM Trans-
[MSK04] M EIER B. J., S PALTER A. M., K ARELITZ
actions on Graphics 18, 2 (1999), 96-127.
D. B.: Interactive color palette tools. IEEE Computer
[FH02] FARUP I., H ARDEBERG J. Y.: Interactive color Graphics & Applications 24, 3 (2004), 64-72.
gamut mapping. In International Printing and Graphics
[PNS03] P ICHON E., N IETHAMMER M., S APIRO G.:
Arts Conference (2002).
Color histogram equalization through mesh deformation.
[GD05] G RUNDLAND M., D ODGSON N. A.: Color his- In Proceedings of the International Conference on Image
togram specification by histogram warping. In Proceed- Processing (2003), vol. 2, 117-120.
ings of SPIE (2005), vol. 5667, 610-621.
[RAG*01] R EINHARD E., A DHIKHMIN M., G OOCH B.,
[GH03] G REENFIELD G. R., H OUSE D. H.: Image re- S HIRLEY P.: Color transfer between images. IEEE Com-
coloring induced by palette color associations. Journal of puter Graphics & Applications 21, 5 (2001), 34-41.
WSCG 11, 1 (2003), 189-196.
[RB02] R EESE L. J., BARRETT W. A.: Image editing
[GL99] G UAN S.-S., L UO M. R.: A colour-difference with intelligent paint. In Proceedings of EUROGRAPH-
formula for assessing large colour differences. Color Re- ICS: Short Presentations (2002), 714-723.
search & Application 24, 5 (1999), 344-355.
[RHO97] RUDAZ N., H ERSCH R. D., O STROMOUKHOV
[Gla03] G LASSNER A.: Image search and replace. IEEE V.: Specifying color differences in a linear color space
Computer Graphics & Applications 23, 3 (2003), 80-88. (LEF). In Proceedings of the Fifth Color Imaging Confer-
[Gol03] G OLDMAN R.: Deriving linear transformations ence (1997), 197-202.
in three dimensions. IEEE Computer Graphics & Appli- [RT01] RUZON M. A., T OMASI C.: Edge, junction, and
cations 23, 3 (2003), 66-71. corner detection using color distributions. IEEE Transac-
[HFK*02] H ARDEBERG J. Y., FARUP I., KOLAS O., tions on Pattern Analysis & Machine Intelligence 23, 11
S TJERNVANG G.: Color management for digital video: (2001), 1281-1295.
Color correction in the editing phase. In International [SES95] S PAULDING K. E., E LLSON R. N., S ULLIVAN
IARIGAI Research Conference (2002). J. R.: Ultracolor: A new gamut-mapping strategy. In
[HJO*01] H ERTZMANN A., JACOBS C. E., O LIVER N., Proceedings of SPIE (1995), vol. 2414, 61-68.
C URLESS B., S ALESIN D. H.: Image analogies. In Pro- [She87] S HEPARD R. N.: Toward a universal law of gen-
ceedings of SIGGRAPH (2001), 327-340. eralization for psychological science. Science 237, 4820
[IT97] I NOUE A., TAJIMA J.: Selective color correction (1987), 1317-1323.
for arbitrary hues. In Proceedings of the International [Sut63] S UTHERLAND I. E.: Sketchpad: A man-machine
Conference on Image Processing (1997), vol. 3, 38-41. graphical communication system. In AFIPS Conference
[KB88] K URLANDER D., B IER E. A.: Graphical search Proceedings (1963), vol. 23, 329-346.
and replace. In Proceedings of SIGGRAPH (1988), 113- [TR98] TASTL I., R AIDL G.: Transforming an ana-
120. lytically defined color space to match psychophysically
[KIO99] K ATOH N., I TO M., O HNO S.: Three- gained color distances. In Proceedings of SPIE (1998),
dimensional gamut mapping using various color differ- vol. 3300, 98-106.
ence formulae and color spaces. Journal of Electronic
[Wol01] W OLF R. R.: Keys to Painting: Color & Tonal
Imaging 8, 4 (1999), 365-379.
Value, North Light, 2001.
[KKK*01] K IM K.-M., K IM J.-Y., K IM H.-S., H A Y.-
[WSM99] W EEKS A. R., S ARTOR L. J., M YLER H. R.:
H.: Natural color reproduction of an image with high-
Histogram specification of 24-bit color images in the color
lights by vector transformation and nonlinear mapping.
difference (C-Y) color space. In Proceedings of SPIE
Journal of Imaging Science & Technology 45, 2 (2001),
(1999), vol. 3646, 319-329.
100-106.

c The Eurographics Association 2005.