1120 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 15, NO.

5, MAY 2006

Fast Image and Video Colorization

Using Chrominance Blending
Liron Yatziv and Guillermo Sapiro

Abstract—Colorization, the task of coloring a grayscale image Chen et al. [5] used manual segmentation to divide the
or video, involves assigning from the single dimension of inten- grayscale image into a set of layers. Then an alpha channel
sity or luminance a quantity that varies in three dimensions, such was estimated using Bayesian image matting. This decompo-
as red, green, and blue channels. Mapping between intensity and
color is, therefore, not unique, and colorization is ambiguous in na- sition allows to apply colorization using Welsh’s approach.
ture and requires some amount of human interaction or external The final image is constructed using alpha-blending. Recently,
information. A computationally simple, yet effective, approach of Sykora et al. [26] have similarly used a segmentation method
colorization is presented in this paper. The method is fast and it optimized for the colorization of black and white cartoons.
can be conveniently used “on the fly,” permitting the user to in- Other approaches, including our own, assume that homo-
teractively get the desired results promptly after providing a re-
duced set of chrominance scribbles. Based on the concepts of lumi- geneity of the grayscale image indicates homogeneity in the
nance-weighted chrominance blending and fast intrinsic distance color. In other words, as detailed in [23], the geometry of the
computations, high-quality colorization results for still images and image is provided by the geometry of the grayscale information
video are obtained at a fraction of the complexity and computa- (see also [4], [6], and [15]). Often, in these methods, in addition
tional cost of previously reported techniques. Possible extensions of to the grayscale data, color hints are provided by the user via
the algorithm introduced here included the capability of changing
the colors of an existing color image or video, as well as changing scribbles. Horiuchi [12] used a probabilistic relaxation method
the underlying luminance, and many other special effects demon- while Levin et al. [17] solved an optimization problem that
strated here. minimizes a quadratic cost function derived from the color
Index Terms—Chrominance blending, colorization, gradient, in- differences between a pixel and its weighted average neigh-
terpolation, intrinsic distance, recolorization, special effects. borhood colors. Sapiro [23] proposed to inpaint the colors
constrained by the grayscale gradients and the color scribbles
that serve as boundary conditions. The method reduces to
I. INTRODUCTION solving linear or nonlinear Poisson equations.
The main shortcoming of these previous approaches is their
C OLORIZATION is the the art of adding color to a mono-
chrome image or movie. The idea of “coloring” photos
and films is not new. Ironically, hand coloring of photographs is
intensive computational cost, needed to obtain good-quality
results. Horiuchi and Hirano addressed this issue in [13], where
as old as photography itself. There exists such examples from they presented a faster algorithm that propagates colored seed
1842 and possibly earlier [19]. It was practiced in motion pic- pixels in all directions and the coloring is done by choosing
tures in the early 1900s by the French Company Pathe, where from a preselected list of color candidates. However, the method
many films were colored by hand. It was also widely practiced produces visible artifacts of block distortion since no color
for filmstrips into the 1930s. The computer-assisted process was blending is performed. While Horiuchi’s method colorizes a
first introduced by Markle in 1970 for adding colors to black and still image within a few seconds, we present in this paper a
white movies [3]. propagation method that colorizes a still image within a second
As neatly presented by Sykora et al. [26] (their work also in- or less, achieving even higher quality results. In contrast with
cludes an outstanding overview of the literature on the subject), works such as those in [17], the technique proposed here is
various early computer-based colorization techniques include easily extended to video without the optical flow computation,
straightforward approaches such as luminance keying [9]. This further improving in the computational cost, at no sacrifice in
method is based on a user-defined lookup table which trans- the image quality. Other special effects are also easily obtained
forms grayscale into color. Welsh et al. [28], inspired by work of using our framework, as demonstrated here and further exem-
Reinhard et al. [21] and Hertzmann et al. [11], extended this idea plified at http://mountains.ece.umn.edu/~liron/colorization/.
by matching luminance and texture rather than just the grayscale The scheme proposed here is based on the concept of color
values. blending. This blending is derived from a weighted distance
function efficiently computed from the luminance channel. The
Manuscript received December 1, 2004; revised May 24, 2005. This work
underlying approach can be generalized to produce other effects
was supported in part by the Office of Naval Research, in part by the National such as recolorization. In the remainder of this paper, we de-
Science Foundation, and in part by the National Geospatial-Intelligence Agency. scribe the algorithm and present a number of examples.
The associate editor coordinating the review of this manuscript and approving
it for publication was Prof. Stanley J. Reeves.
The authors are with the Electrical and Computer Engineering Depart- II. FAST COLORIZATION FRAMEWORK
ment, University of Minnesota, Minneapolis, MN 55455 USA (e-mail:
[email protected]; [email protected]). Similar to other colorization methods, e.g., [17] and [23],
Digital Object Identifier 10.1109/TIP.2005.864231 we use luminance/chrominance color systems. We present our
1057-7149/$20.00 © 2006 IEEE
YATZIV AND SAPIRO: FAST IMAGE AND VIDEO COLORIZATION 1121

method in the YCbCr color space, although other color spaces Since the chrominance data is often given in whole regions
such as YIQ or YUV could be used as well. Moreover, work and not necessarily in single isolated points, we would like to
can also be done directly on the RGB space. Let get an idea of the distance from a certain known chrominance
be the given monochromatic image (“scribbles” with a given uniform color) to any point in .
or video defined on a region . Our goal is to com- We then define the intrinsic distance from a point (to be
plete the Cb and Cr channels colored) to a certain chrominance , as the minimum distance
and , respectively. For clarity from to any point of the same chrominance in the
of the exposition, we refer to both channels as the chrominance. set of given colors
The proposed technique also uses as input observed values of
the chrominance channels in a region which is signifi- (2)
cantly smaller than (see [17]). These values are often provided
by the user or borrowed from other data. This gives the distance from a point to be colored to scrib-
Let and be two points in and let be bles (from the provided set ) with the same color , or, in other
a curve in . Also, let represent a curve connecting and words, the distance from a point to a provided color.
such that and . We define the intrinsic Our idea for colorization is to compute the Cb and Cr com-
(geodesic) distance between and by ponents (chrominance) of a point in the region where they are
missing by blending the different chrominance in
according to their intrinsic distance to
(1)

This intrinsic distance gives a measurement of how “flat” is the (3)

flattest curve between any two points in the luminance channel.
The integral in the equation above is basically integrating the where stands for all the different unique
luminance gradient in the direction of the curve (as chrominance in the set , and is a function of the in-
given by projecting the gradient into the tangent to this trinsic distance that translates it into a blending weight. In other
curve). When considering the minimum over all paths , we words, the above blending expression assigns to any point to be
then keep the one with the smallest overall gradient in this di- colored, a color that is a weighted average of the different colors
rection, thereby the flattest path connecting the two points and in the provided set of scribbles . For every distinct color in
(the path that goes from to with minimal overall gradient). the set , the distance to it from is computed following (2)
Note that the minimal path needs not to be unique, but we only [which uses (1)], and this distance is used to define the weight of
care about the intrinsic length of this path, so this does the color at the point (the blending proportion of this color).
not affect the algorithm. The denominator in the above equation is just a normalization
Geodesic distances of this type can be efficiently and accu- factor.
rately computed using recently developed fast numerical tech- The function that provides the weighting factor from
niques [10], [14], [24], [25], [27], [29], [31]. We found that the distance should hold some basic properties:
for the application at hand, even simpler techniques such as a 1) ;
best first one (in particular, Dijkstra [8]), integrated in the pseu- 2) ;
docode below, are sufficient. This will be further detailed in the 3) a constant.
Section III. The first two requirements are obvious. Requirement 3) is nec-
Even though a mapping between luminance and chrominance essary when there are two or more chrominance sources close
is not unique, a close relationship between the basic geometry of by, but the blending is done relatively far from all sources. The
these channels is frequently observed in natural images (see, for desired visual result would be the even blending of all chromi-
example, [4], [6], and [15]). Sharp luminance changes are likely nance. For the experiments reported below, we used
to indicate an edge in the chrominance, and a gradual change in
luminance often indicates that the chrominance is also not likely (4)
to have an edge but rather a moderate change. In other words, as
reported in the aforementioned works, often, there is a close re- where is the blending factor, typically . This factor
lationship between the geometries of the luminance and chromi- defines how smooth is the chrominance transition.
nance channels. Exploiting this assumption, a change in lumi- Note that, in theory, following the equations above, a point
nance causes a related change in chrominance. This has been to be colored will be influenced by all distinct colors in ,
used in different fashions in [17] and [23], as well as in [2] for since .
super resolution. From this, for the proposed colorization ap- A. Implementation Details
proach we assume that the smaller the intrinsic distance
between two points , the more similar chrominance they A key component of our algorithm is the computation of
would have.1 the intrinsic, gradient weighted, distance provided by (1). We
should then first concentrate on how to compute this distance.
1It is important to notice that the goal of colorization is not to restore the
Geodesic distances of this type have been widely studied in the
original color of the image or scene, which is, in general, not available, but as in
image inpainting, to produce visually pleasant and compelling colored images scientific computing community, in general, as well as in the
(see Section IV for more on this). control and image-processing communities, in particular. For
1122 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 15, NO. 5, MAY 2006

example, expressions of this type have been used for edge de- d)
tection in [16]. They constitute a modification of the popular 4) while
geodesic active contours for object detection (see [22] and refer- a) the link with smallest distance in (see Footnote (4)).
ences therein for details). The popularity of these expressions in b)
other disciplines means that their efficient computation has been c) the blend structure of
widely studied, e.g., [10], [14], [24], [25], [27], [29], and [31], d) if
where the reader is referred for more details, including exact i) add to
numerical computations of the gradient in (1) using up-wind ii) add to
schemes. These algorithms are inspired by the classical Dijk- iii) for all pixels neighboring
stra graph algorithm [8],2 fast sweeping techniques [7], and spe- A) create link
cial data structures such as buckets and priority queues [29]. B) 3

A straightforward and very fast and accurate implementation C)

of our algorithm consists in computing for every point to be 5) for all pixels in ; generate the chrominance channels by applying (3) with the
colored in the distance to every distinct chromi- pixels’ blend structures.
nance in the given , using these fast techniques, thereby
obtaining the set of weights needed for the blending (3). We
found that there is no real need to implement the computa- B. Performance and Relaxation
tion of the weighted distance using the aforementioned accu- The described colorization algorithm has average time and
rate techniques, and simply using classical Dijkstra is sufficient. space complexity of .4 The algo-
Thereby, in principle, we create a graph where each image pixel rithm passes over the image/video for each different chromi-
is a vertex connected to its neighbors with edges that, following nance observed in and needs a memory in the order of the
(1), have weights equal to the absolute gray value derivatives number of different chrominances observed in , times the
between the neighbors, and work with Dijkstra’s algorithm on input image/video size. If there are a large number scribbles of
this graph. Following the results in [29] (see also the references different chrominances, the algorithm could be relatively slow
therein), we could use more accurate distance computations at and pricey in memory (although still more efficient that those
minimal additional cost, but we found this not to be necessary previously reported in the literature).
to obtain the high quality results reported here. Fortunately, since human perception of color blending is lim-
To complete the description of the proposed colorization tech- ited, high blending accuracy is not fully necessary to obtain sat-
nique, we provide a pseudocode that exemplifies one of the isfactory results. Experimental results show that it is enough
possible ways to efficiently implement the proposed algorithm just to blend the most significant chrominance (the chrominance
[(1)–(3)] and further emphasizes the simplicity of the proposed with the closest intrinsic distance to their observed source). We
technique. This is just a standard form of implementing the clas- found that, in natural images, it is enough to blend just the two
sical Dijkstra-based ideas mentioned above. or three most significant chrominance to get satisfactory results.
Such a relaxation reduces both the time and space complexity
to , thereby linear in the amount of data. Therefore, we
• Input:
do not include in the blend chrominances that their weight in
— Grayscale video/image.
the blending equation is small relatively to the total weight. Ad-
— List of observed pixels and their chrominance (user provided scribbles).
ditional quality improvements could be achieved if an adaptive
• Output: Colored video/image.
threshold, following results such as those from the MacAdam el-
• Definitions:
lipses [18], is used. Any color lying just outside an ellipse is at
— For each pixel in a blend structure is created that contains:
the “just noticeable difference” (jnd) level, which is the smallest
1) —list of chrominances (the chrominances that reached this blend)
perceivable color difference under the experiment conditions. A
2) —list of distances (the corresponding intrinsic distance to each chrominance in
possible use of this is to define an adaptive threshold that would
)
filter out chrominance that if added to the blend would not cause
— A link structure contains:
a jnd.
1) ptr—pointer to a pixel (the destination pixel of the link)
This proposed algorithm relaxation of limiting the number of
2) —a chrominance (the chrominance the link attempts to transfer)
contributors to the blending equation gives a tight restriction on
3) —distance (the intrinsic distance from the chrominance origin in )
how far the chrominance will propagate to be included in the
• The basic algorithm:
blend. The restriction can be easily implemented adding condi-
1) create an empty blend structure for each pixel in
tions to d) in 4) in the pseudocode.
2)
3) for each pixel in
a) let be the chrominance of the scribble in
3The function distance(p; q ) is the distance between p and q . Following (1)
b) create link
and the Dijkstra algorithm, this is obtained by a numerical approximation of
c)
the absolute value of the corresponding luminance derivative; e.g., just absolute
luminance difference for a simple approximation.
4Using a priority queue with O (1) average complexity per operation, as done
2The Dijkstra algorithm is not consistent and has a bias when measuring dis- in [1], [27], and [29]; a heap sort data structure as used in the original Dijkstra
tances on a grid. The aforementioned techniques are correct for this bias. algorithm would slightly increase the run-time complexity.
YATZIV AND SAPIRO: FAST IMAGE AND VIDEO COLORIZATION 1123

Fig. 2. Comparison of visual quality with the technique proposed in [17].

(a) Given grayscale image; (b) the user marks chrominance scribbles (the
selected scribbles are obtained from the work in [17]); (c) our algorithm results
with CPU run-time of 0.76 s; (d) Levin et al. approach with CPU run-time of
24 s using their supplied fast implementation based on multigrid solver (611 s
are needed for their standard Matlab implementation). We observe the same
quality at a significantly reduced computational cost.

Fig. 1. Still-image colorization examples. Given a grayscale image, (left)

the user marks chrominance scribbles, and (right) our algorithm provides a
2
colorized image. The image size/run time from top to bottom are 230 345/less
2 2
than 0.42 s, 256 256/less than 0.36 s, and 600 450/less than 1.73 s.

III. COLORIZATION RESULTS

We now present examples of our image and video coloriza-
tion technique. Additional examples, comparisons, and movies,
as well as software for testing our approach, can be found at Fig. 3. Comparison of visual quality with the technique proposed in [17].
http://mountains.ece.umn.edu/~liron/colorization/. This site
2
(a) Given grayscale image (400 240); (b) the user marks chrominance
scribbles; (c) our algorithm results with CPU run time of 1.20 s; (d) Levin et al.
also contains examples beyond colorization, such as video re- approach with CPU run time of 22.8 s using their supplied fast implementation
coloring, depth effects, decolorizing, and depth-of-field effects. of multigrid solver; (e) Levin et al. approach using a slower exact Matlab least
squares solver also provided by the authors.
The proposed algorithm has been implemented in C++ as
a stand alone win32 application so it could be tested for
speed and quality. For timing we used an Intel Pentium III Fig. 2, we observe that we achieve the same visual quality at a
with 512-MB RAM running under Windows 2000. Fig. 1 fraction of the computational cost (more comparisons are pro-
shows examples of still image colorization using our proposed vided in the aforementioned web site, all supporting the same
algorithm. The algorithm run time for all the examples in finding). Overall, the method proposed in [17] performs very
Fig. 1, measured once the images were loaded into memory, well on many images, yet Fig. 3 demonstrates that it can per-
is less than 7 s per pixel. form poorly when colorizing relatively far from the provided
Figs. 2 and 3 compare our method with the one recently pro- color constrains. Following the equations in the previous section
posed by Levin et al. [17] (this work partially inspired our own). [see (3)], even far away pixels will receive color from the scrib-
The method minimizes the difference between a pixel’s color bles with our approach (in particular from the closest ones, in
and the weighted average color of its neighboring pixels. The weighted distance, see section on relaxation above). In order to
weights are provided by the luminance channel. The minimiza- match visual quality with our technique, the method proposed in
tion is an optimization problem, subject to constraints supplied [17] needs more user input, meaning additional color scribbles.
by the user as chrominance scribbles. Solving this is computa- We also found that the inspiring technique developed in [17]
tionally costly and slower than our proposed technique. First, in has a sensible scale parameter and often fails at strong edges,
1124 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 15, NO. 5, MAY 2006

Fig. 4. Testing the robustness of the proposed algorithm. The first row shows an example where different sets of scribbles are used, obtaining visually identical
results. The original monochromatic image is shown first, followed by the first set of scribbles and the resulting colorized result. This is followed by a second set
of scribbles and the corresponding colorized results. Note how the two results are virtually identical. The next row repeats the same experiment for a different
image. Next, we show (third row) how the result is evolving with the addition of new scribbles by the user. The original image is presented first, followed by the
set of scribbles labeled by their order of application. The third figure shows the result when only the two scribbles labeled 1 are used. Then, scribbled 2 is added
to improve the results on the top of the ball, obtaining the fourth figure. Finally, the scribble labeled 3 was added to keep the shadow gray instead of green, and
the result is provided in the last figure.

Fig. 5. Colorization with palettes. After the scribble is propagated, the gray value is used to key the palette. The original image is followed by the colored one in
the first row. The second row shows the palettes for the mountain and the sky, respectively.

since these provide zero or very limited weight/influence in their we place it at a different location? If the ideal scribble and the
formulation. placed one are both inside the same object, and the region be-
In Fig. 4, we study the robustness of our proposed approach tween them is relatively homogenous, then using our gradient
with respect to the scribbles placement (location of the set ). weighted metric, the distance between the two scribbles will
Before describing these examples, let us provide some basic be relatively small. From the triangle inequality, we can then
comments that will help to understand the observed robustness bound the distance from the placed scribble to a given pixel
of this technique. Assume that we know the “ideal” position to to be colored, simply using the sum of the distance from the
place a scribble (see Section IV for more on this). What hap- ideal scribble to such pixel and the distance between the scrib-
pens if, instead of placing this scribble at the optimal place, bles. Since the latter is small, then the distance from the ideal
YATZIV AND SAPIRO: FAST IMAGE AND VIDEO COLORIZATION 1125

Fig. 6. Animated video colorization example. (a) Given the ten-frame grayscale sequence, (b) our algorithm provides a colorized video. (c) The first and last
frame of the colorized video are enlarged to show the color content. (d) To colorize the ten frames, all that was needed were a few chrominance scribbles on a
2
single frame. The user marked chrominance scribbles on the seventh frame of this sequence. The size of each frame is 312 264, and the algorithm total run time
for the whole sequence is 6.8 s. The movie can be found on the web site for this project.

scribble and the one from the placed one to the pixel to be col- tion compared to [17] and [26], where not only is each frame
ored are very similar, and, as such, the result of the colorization computed significantly faster, but there is also no need for op-
algorithm. If the placed scribble is located “on the wrong side tical flow. Fig. 6 demonstrates the colorization of an animated
of the edge” (or with high gradients between it and the ideal scene from the movie Finding Nemo. Fig. 7 shows a coloriza-
scribble), then, of course, the distance between the ideal and the tion example of an old film. In both examples, we obtained very
placed scribbles will be large and as such the algorithm will pro- good results just by marking a few chrominance scribbles in a
duce very different results. Of course, this will almost mean an single frame.
intentional mistake by the user. Moreover, interactive coloriza- The blending factor is the only free parameter of the algo-
tion is possible thanks to the fast speed of the proposed tech- rithm as currently implemented. We set it to be for all
nique, thereby permitting the user to correct errors and to add or examples in this paper and the additional ones in the afore-
move scribbles as needed (this can be easily experimented with mentioned web page. Better results may have been archived se-
the public domain software mentioned above). The first row of lecting a different value per image and video, although we did
Fig. 4 shows an example of an image colored with different sets not find this necessary to obtain the high quality results pre-
of scribbles. Notice how the results are visually indistinguish- sented here.
able. This is repeated for a second example in the next row. The
last row of the figure shows the evolution of the result as the A. Recolorization and Extensions
user adds scribbles, a common working scenario for this type Recolorization is the art of replacing the colors of an image
of application, which is allowed by the speed and robustness of by new colors. Fig. 8 shows that it is possible to change colors
our proposed technique (this is also part of the public domain of an existing image or video just by slightly modifying our
implementation mentioned above). original colorization algorithm. When colorizing grayscale im-
Fig. 5 shows a simple example of special effects that can be ages, we based the process on the assumption that homogeneity
easily obtained with our proposed framework, colorization with of the grayscale indicates homogeneity in the chrominance. In
palettes. After the scribble is propagated, the gray value is used recolorization, we have more clues about how the chrominance
to key the palette. The original image is followed by the colored should vary. We assume that homogeneity in the original
one on the first row. The second row shows the palettes for the chrominance indicates homogeneity in the new chrominance.
mountain and the sky, respectively. The chrominance propagation can be based on the grayscale
Figs. 6 and 7 show how our technique can be applied to video. assumption or the original color assumption or both, as demon-
Given a grayscale video and some chrominance scribbles any- strated in Fig. 9. The intrinsic distance can also be measured on
where in the video, our algorithm colors the whole video within the Cb and Cr channels rather than just on the intensity channel
seconds. This is a significant computational complexity reduc- as done in (1). This is done simply by replacing by or
1126 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 15, NO. 5, MAY 2006

Fig. 7. Video colorization example. (a) Given a 21 frame grayscale sequence, (b) our algorithm provides a colorized video. (c) The first and last frame of the
colorized video are enlarged to show the color content. (d) To colorize the 21 frames, all that was needed are a few chrominance scribbles on a single frame. The
2
user marked chrominance scribbles on the eleventh frame of this sequence. The size of each frame is 320 240, and the algorithm total run time for the whole
sequence is 14 s. The movie can be found in the web site for this project.

in this equation, thereby using the gradients of these (now Fig. 10 follows this idea and gives an example of object bright-
available) chroma channels to control the blending weight of ness change.
the new color. Video recolorization examples can be seen in the
aforementioned web site.
Recolorization is just one possible extensions of our method. IV. CONCLUDING REMARKS
It is also possible to further generalize it by defining the mea-
surement medium on which the in- In this paper, we have introduced a fast colorization algorithm
trinsic distance is measured. can be any channel of the input for still images and video. While keeping the quality at least
image, a mix of the channels, or any other data that will make as good as previously reported algorithms, the introduced tech-
sense for weighting the intrinsic distance. The blending medium nique manages to colorize images and movies within a second,
is then the data that is actually compared to other techniques that may reach several minutes.
blended. Both in colorization and recolorization, we selected The proposed approach needs less manual effort than techniques
to be the chrominance. Yet, can be any image processing ef- such as those introduced in [17] and [23] and can be used in-
fect or any data that makes sense to blend for a given application. teractively due to its high speed. We also showed that simple
YATZIV AND SAPIRO: FAST IMAGE AND VIDEO COLORIZATION 1127

Fig. 8. Recolorization example. (a) Original color image, (b) scribbles, and (c) image after recolorization (note the pants, glasses, and window). The white
scribbles represent a “mask,” used to keep the original colors.

M
Fig. 9. Recolorization example using the Cr channel for measurement ( ) rather than the intensity channel. (a) Original color image; (b) intensity channel of the
image. It can be clearly seen that the intensity image does not contain significant structural information on the red rainbow strip (this is quite a unique example of
this effect). On the other hand, both the Cb and Cr do change significantly between the stripes and can, therefore, be used for recolorization. (c) Recolored image,
the red stripe of the rainbow was replaced by a purple one.

modifications in the algorithm lead to recolorization and other be efficiently reconstructed by the decoder. In order to do this, we
special effects [30]. need to understand what is the simplest set of color scribbles that
A number of directions can be pursued as a result of the frame- when combined with our algorithm, manages to reconstruct the
work introduced here. First, as mentioned before, other special color without any visual artifacts. In the same spirit, for the col-
effects can be obtained following this distance-based blending orization of originally mono-chromatic data, or the recoloring of
of image attributes approach. We are working on including in color data, it is important to understand the best position to place
this framework a crude (fast) computation of optical flow, to use the scribbles so that the user’s effort is minimized. For example,
mainly when the motion is too fast for the weighted distance used an edge map on the luminance data might help to guide the user.
in the examples presented here. Another interesting problem is to Although, as demonstrated with the examples provided in this
investigate the use of this colorization approach for image com- paper and the additional ones in our web site, the robustness and
pression. As shown in the examples presented here, a few color speed of the algorithm make it possible to work in an interactive
samples are often enough to produce visually pleasant results. fashion, reducing/minimizing user intervention is always an im-
This can be exploited for compression, in the same spirit as done portant goal. Results in these research directions will be reported
in [20], where the encoder voluntarily drops information that can elsewhere.
1128 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 15, NO. 5, MAY 2006

Fig. 10. Example of the generalization of our method to other image processing effects. (a) The original color image. Our goal is to change the color of the yellow
car into a darker color. (b) We define the blending medium by roughly marking areas we do not wish to change in white and areas we do want to change in black;
we do so by placing scribbles or by just marking whole areas. (c) Using our colorization method, we propagate the markings (white and black colors) and get a
grayscale matte (we only keep the blending channel). With the matte, it is possible to apply an effect to the original image with a magnitude proportional to the
gray level of the matte. In this case, we chose to change the brightness. (d) Image after applying the darkening. Note that the intensity only changed in the desired
parts of the image. The darkening is done simply by subtracting the gray-level matte from the intensity channel, where white means no change and black is the
maximum preselected change. (e) It is possible to further process the image using the same matte and (f) demonstrate this by similarly adding the gray-level matte
to the Cb and Cr channels, respectively.

ACKNOWLEDGMENT [13] T. Horiuchi and S. Hirano, “Colorization algorithm for grayscale image
by propagating seed pixels,” in Proc. IEEE Int. Conf. Pattern Recogni-
The authors would like to thank G. Brown, A. Bartesaghi, and tion, 2003, pp. 457–460.
[14] (2003, Dec.) Univ. California, Los Angeles, CAM Rep.. [Online]. Avail-
K. Patwardhan for their inspiring and useful remarks. able: http://www.math.ucla.edu/applied/cam/index.html
[15] R. Kimmel, “A natural norm for color processing,” presented at the 3rd
Asian Conf. Computer Vision, Hong Kong, Jan. 8–11, 1998.
REFERENCES [16] R. Kimmel and A. M. Bruckstein, “On regularized Laplacian zero cross-
ings and other optimal edge integrators,” Int. J. Comput. Vis., vol. 53, no.
[1] R. k. Ahuja, K. Melhrhorn, J. B. Orlin, and R. E. Tarjan, “Faster algo- 3, pp. 225–243, 2003.
rithms for the shortest path problem,” J. AMC, vol. 37, pp. 213–223, [17] A. Levin, D. Lischinski, and Y. Weiss, “Colorization using optimiza-
1990. tion,” in Proc. ACM SIGGRAPH Conf., 2004, pp. 689–694.
[2] C. Ballester, V. Caselles, J. Verdera, and B. Rouge. A varia- [18] D. MacAdam, Sources of Color Science. Cambridge, MA: MIT Press,
tional model for P + XS image fusion. presented at Work- 1970.
shop on Variational and Level Set Methods. [Online]. Available: [19] P. Marshall, “Any Color You Like,” [Online]. Available: http://photog-
http://www.iua.upf.es/vcaselles/ raphy.about.com/library/weekly/aa061 002a.htm, Jun. 11, 2002.
[3] Museum of Broadcast Communication: Encyclopedia of Television, [On- [20] S. Rane, G. Sapiro, and M. Bertalmio, “Structure and texture filling-in of
line]. Available: http://www.museum.tv/archives/etv/C/htmlC/coloriza- missing image blocks in wireless transmission and compression appli-
tion/colorization.htm. cations,” IEEE Trans. Image Process., vol. 12, no. 2, pp. 296–303, Feb.
[4] V. Caselles, B. Coll, and J.-M. Morel, “Geometry and color in natural 2003.
images,” J. Math. Imag. Vis., vol. 16, pp. 89–105, 2002. [21] E. Reinhard, M. Ashikhmin, B. Gooch, and P. Shirley, “Color transfer
[5] T. Chen, Y. Wang, V. Schillings, and C. Meinel, “Gray-scale image mat- between images,” IEEE Comput. Graph. Appl., vol. 21, no. 5, pp. 34–41,
ting and colorization,” in Proc. Asian Conf. Computer Vision, 2004, pp. May 2001.
1164–1169. [22] G. Sapiro, Geometric Partial Differential Equations and Image Pro-
[6] D. H. Chung and G. Sapiro, “On the level-lines and geometry of vector- cessing. Cambridge, U.K.: Cambridge Univ. Press, Jan. 2001.
valued images,” IEEE Signal Process. Lett., vol. 7, no. 9, pp. 241–243, [23] , “Inpainting the colors,” in IMA Preprint Series 1979: Inst. Math.
Sep. 2000. Appl., Univ. Minnesota, Minneapolis, May 2004.
[7] P. Danielsson, “Euclidean distance mapping,” Comput. Graph. Image [24] J. Sethian, “Fast marching level set methods for three-dimensional
Process., vol. 14, pp. 227–248, 1980. photolithography development,” presented at the SPIE Int. Symp.
[8] E. Dijkstra, “A note on two problems in connection with graphs,” Numer. Microlithography, Santa Clara, CA, Mar. 1996.
Math, vol. 1, pp. 269–271, 1959. [25] J. A. Sethian, “A fast marching level-set method for monotonically ad-
[9] R. C. Gonzalez and R. E. Woods, Digital Image Processing, 2nd vancing fronts,” Proc. Nat. Acad. Sci., vol. 93, no. 4, pp. 1591–1595,
ed. Reading, MA: Addison-Wesley, 1987. 1996.
[10] J. Helmsen, E. G. Puckett, P. Collela, and M. Dorr, “Two new methods [26] D. Sýkora, J. Buriánek, and J. Zára, “Unsupervised colorization of black-
for simulating photolithography development in 3D,” in Proc. SPIE Mi- and-white cartoons,” in Proc. 3rd Int. Symp. NPAR, Annecy, France, Jun.
crolithography, vol. IX, 1996, p. 253. 2004, pp. 121–127.
[11] A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. H. Salesin, [27] J. N. Tsitsiklis, “Efficient algorithms for globally optimal trajectories,”
“Image analogies,” in Proc. ACM SIGGRAPH Conf., 2001, pp. 327–340. IEEE Trans. Autom. Control, vol. 40, no. 10, pp. 1528–1538, Oct. 1995.
[12] T. Horiuchi, “Estimation of color for gray-level image by probabilistic [28] T. Welsh, M. Ashikhmin, and K. Mueller, “Transferring color to
relaxation,” in Proc. IEEE Int. Conf. Pattern Recognition, 2002, pp. grayscale images,” in Proc. ACM SIGGRAPH Conf., 2002, pp.
867–870. 277–280.
YATZIV AND SAPIRO: FAST IMAGE AND VIDEO COLORIZATION 1129

[29] L. Yatziv, A. Bartesaghi, and G. Sapiro. (2005, Feb.) O(N) Imple- Guillermo Sapiro was born in Montevideo,
mentation of the Fast Marching Algorithm. [Online]. Available: Uruguay, on April 3, 1966. He received the B.Sc.
http://www.ima.umn.edu/preprints/feb2005/feb2005.html (summa cum laude), M.Sc., and Ph.D. degrees
[30] L. Yatziv and G. Sapiro, “Image and Video Data Belnding Using In- from the Department of Electrical Engineering,
trinsic Distances,” Patent Pending, 2005. Technion–Israel Institute of Technology, Haifa, in
[31] H. K. Zhao, “Fast sweeping method for Eikonal equations,” Math. 1989, 1991, and 1993, respectively.
Comput., vol. 74, pp. 603–627, 2004. After postdoctoral research at the Massachusetts
Institute of Technology, Cambridge, he became
a Member of the Technical Staff at the research
facilities of HP Labs, Palo Alto, CA. He is currently
with the Department of Electrical and Computer
Engineering, University of Minnesota, Minneapolis. He works on differential
Liron Yatziv was born in Haifa, Israel, on January geometry and geometric partial differential equations, both in theory and
21, 1975. He received the B.Sc. degree (summa applications in computer vision, computer graphics, medical imaging, and
cum laude) from the Department of Electrical En- image analysis. He recently coedited a special issue on image analysis in the
gineering, Technion–Israel Institute of Technology, Journal of Visual Communication and Image Representation. He has authored
Haifa, in 2001, and the M.Sc. degree from the and coauthored numerous papers in image anaylsis and he has written a book.
University of Minnesota, Minneapolis, in 2005. Dr. Sapiro is a member of SIAM. He recently co-edited a special issue of
He is currently an Associate Member of Technical IEEE TRANSACTIONS ON IMAGE PROCESSING. He was awarded the Gutwirth
Staff, Siemens Corporate Research, Princeton, NJ. In Scholarship for Special Excellence in Graduate Studies in 1991, the Ollendorff
his current position at Siemens, he is responsible for Fellowship for Excellence in Vision and Image Understanding Work in 1992, the
developing image processing algorithms that will be Rothschild Fellowship for Postdoctoral Studies in 1993, the Office of Naval Re-
used in advanced medical equipment. His current in- search Young Investigator Award in 1998, the Presidential Early Career Awards
terests are in the fields of image processing and algorithms, about which he for Scientist and Engineers (PECASE) in 1998, and the National Science Foun-
published several papers. dation Career Award in 1999.