Preserving Color in Neural Artistic Style Transfer

Leon A. Gatys Matthias Bethge Aaron Hertzmann Eli Shechtman

U. Tübingen U. Tübingen Adobe Research Adobe Research
arXiv:1606.05897v1 [cs.CV] 19 Jun 2016

June 2016

(a) Content (b) Style (c) Output using [2]

(d) Output with color preservation (color transfer) (e) Output with colour preservation (luminance matching)

Figure 1: Example using a style dominated by brushstrokes. (a) Input photograph. (b) Painting Starry
night over the Rhone by Vincent van Gogh. (c) Transformed content image, using original neural style
transfer algorithm [2]. The color scheme is copied from the painting. (d) Transformed content image,
using color transfer to preserve colors. (e) Transformed content image, using style transfer in luminance
domain to preserve colors.

Abstract
This note presents an extension to the neural artistic style transfer algorithm [2]. The original
algorithm transforms an image to have the style of another given image. For example, a photograph
can be transformed to have the style of a famous painting. Here we address a potential shortcoming
of the original method: the algorithm transfers the colors of the original painting, which can alter
the appearance of the scene in undesirable ways. We describe simple linear methods for transferring
style while preserving colors.

1
1 Introduction
The recent neural artistic style algorithm [2] takes as input two images — a style image and a content
image — and outputs a new image depicting the objects of the content image in the style of the other
image. For example, Figure 1 shows a photograph of a farmhouse garden as the content image, and
a painting by Vincent van Gogh as the style image. Applying the algorithm produces a new painting
of the farmhouse in the style of the van Gogh painting. This method works by matching statistics on
the feature responses in a Convolutional Neural Network trained on object recognition; for a detailed
description of the algorithm see [2]. As illustrated in Figures 1 and 2, the output reproduces the style of
brushstrokes, geometric shapes, and painterly structures exhibited in the style image. However, it also
copies the color distribution of the style image which might be undesirable in many cases (Fig. 1(c), 2(c)).
For example, the painting of the farmhouse has the colors of the original van Gogh painting (Fig. 1(c)),
whereas one might prefer the output painting to preserve the colors of the farmhouse photograph. In
particular, one might also imagine that the artist would have used the colors of the scene if they were
to paint the farmhouse.
This note presents two simple methods to preserve the colors of the source image during neural style
transfer—in other words, to transfer the style without transferring the colors. We compare two different
approaches to color preservation: color histogram matching and luminance-only transfer (Fig. 1(d,e),
2(d,e)). We compare these methods and discuss their advantages and disadvantages.

(a) Content (b) Style (c) Output using [2]

(d) Output with color preservation (color transfer) (d) Output with colour preservation (luminance matching)

Figure 2: Example using a style dominated by geometric shapes. (a) Input photograph. (b) Painting
Femme nue asisse by Pablo Picasso. (c) Transformed content image, using original neural style transfer
algorithm [2]. The color scheme is copied from the painting. (d) Transformed content image, using color
transfer to preserve colors. (e) Transformed content image, using style transfer in luminance domain to
preserve colors.

2
2 Approach #1: Color histogram matching
The first method we present works as follows. Given the style image (S), and the content image (C),
the style image’s colors are tranformed to match the colors of the content image. This produces a new
style image S 0 that replaces S as input to the neural style transfer algorithm. The algorithm is otherwise
unchanged.
The one choice to be made is the color transfer procedure. There are many color transformation
algorithms to choose from; see [1] for a survey. Here we use linear methods, which are simple and very
effective for color style transfer.
In particular, let xi = (R, G, B)T be a pixel of an image. Each pixel is transformed as:

xS 0 ← AxS + b (1)

where A is a 3 × 3 matrix and b is a 3-vector.

We choose this transformation so that the mean and covariance of the RGB values in the new style
image S 0 match those of C 0 . In other words, let µS and µC be the mean colors of the style and content
images, P
and ΣS and ΣC bePthe pixel covariances. The mean and covariance of the color pixels is given
by µ = i xi /N and Σ = i (xi − µ)(xi − µ)T /N . We want to choose A and b to be satisfy µS 0 = µC
and ΣS 0 = ΣC . These are satisfied by the constraints:

b = µC − AµS (2)
AΣS AT = ΣC (3)

There remains a family of solutions for A that satisfies these constraints.

We consider two variants. The first variant uses the Cholesky decomposition:

Achol = LC L−1
S (4)

where Σ = LLT is the Cholesky decomposition of Σ.

The second variant is a 3D color matching formulations explored in Image Analogies [3] (Appendix
B). First, let the eigenvalue decomposition of a covariance matrix be Σ = UΛUT . Then, we define a
matrix square-root as: Σ1/2 = UΛ1/2 UT . Then, the transformation is given by
1/2 −1/2
AIA = ΣC ΣS (5)

In general, we find that results using the Image Analogies color transfer (Fig. 3(b)) look better than
those using the Cholesky transfer (Fig. 3(a)). Furthermore, the Cholesky transform has the conceptually-
undesirable property that it depends on the channel ordering, i.e., using RGB images will give different
results from BGR. We also experimented with the Monge-Kantorovitch linear transform [5] that mini-
mizes the pixel-wise L2 -distance to the image before color transfer, and found the results to be essentially
indistinguishable from the Image Analogies approach.
In general, we find that the color matching method works reasonably well with neural style transfer.
This is in contrast to Image Analogies, where it gave poor synthesis results [3].
We also test transferring the color distribution to the output of neural style transfer instead of to
the style image. In other words, neural style transfer is computed from the original inputs S and C,
and then the output T is color-matched to C, producing a new output T 0 . We find that transferring
the color histogram before style transfer leads to better results (Fig. 4). This is particularly apparent
for the second example in Figure 4, where the cubist texture is not transferred as completely to the sky.
It appears to be easier for the algorithm to transfer the style between two images with similar color
distribution. This can be explained by the reduced competition between reconstructing the content
image and simultaneously matching the texture information from the style image.

3
(a) (b)

Figure 3: Comparison of linear color transfer onto the style image from Figure 1. (a) Cholesky color
transfer (b) Image Analogies color transfer. The latter better preserves the colors of the source image.

(a) (b)

(c) (d)

Figure 4: Comparison between matching the color distribution before or after style transfer. The Image
Analogies color transfer is used. (a) Color transfer before synthesis (same as Figure 1(d)). (b) Color
transfer after synthesis. There are subtle differences, including purple plants in the middle-top of (b).
(c) Color transfer before synthesis (same as Figure 2(d)). (d) Color transfer after synthesis. Here, the
sky loses the “cubist texture” of the style image.

4
3 Approach #2: Luminance-only transfer
The second method we consider is to perform style transfer only in the luminance channel, as used in
Image Analogies [4]. This is motivated by the observation that visual perception is far more sensitive to
changes in luminance than in color [6].
The modification is simple. The luminance channels LS and LC are first extracted from the style
and content images. Then the neural style transfer algorithm is applied to these images to produce an
output luminance image LT . Using the YIQ color space, the color information of the content image
are represented by the I and Q channels; these are combined with LT to produce the final color output
image (Fig. 5(c,d)).
If there is a substantial mismatch between the luminance histogram of the style and the content
image, it can be helpful to match the histogram of the style luminance channel LS to that of the content
image LC before transferring the style. As in the previous section, we use a linear map that matches
the second-order statistics of the content image (Fig. 5(e,f)). Let µS and µC be the mean luminances
of the two images, and σS and σC be their standard deviations. Then each luminance pixel in the style
image is updated as:
σC
LS 0 = (LS − µS ) + µC (6)
σS

4 Comparison and discussion

Here we present two simple methods to preserve the color of the content image in the neural style transfer
algorithm [2]:

1. Linear color transfer onto the style image, before style transfer.
2. Style transfer only in the luminance channel.
Both methods give perceptually-interesting results but have different advantages and disadvantages.
The first method is naturally limited by how well the color transfer from the content image onto
the style image works. The color distribution often cannot be matched perfectly, leading to a mismatch
between the colors of the output image and that of the content image (Fig. 6(e)). The synthesis also
replicates “content” structures from the van Gogh style scene, i.e., the pattern of reflections on the river
appear as vertical yellow stripes of brushstrokes in the output.
In contrast, the second method preserves the colors of the content image perfectly. However, depen-
dencies between the luminance and the color channels are lost in the output image (Fig. 6 (d)). This
is particularly apparent for styles with prominent brushstrokes. In Figure 6(d), colors are no longer
aligned to strokes. That means a single brushstroke can have multiple colors, which does not happen
in real paintings. In comparison, when using full style transfer and color matching, the output image
really consists of strokes which are blotches of paint, not just variations of light and dark.
One potential advantage of the luminance-based method is that it reduces the dimensionality of the
optimization problem for the neural synthesis. The neural synthesis algorithm performs numerical opti-
mization of the output image, and luminance-only synthesis has one-third fewer parameters. However,
it is unclear that there is any practical advantage in typical GPU implementations.
In future work, it would be interesting to explore how the two statistical models in here (color
statistics vs. CNN activations) might be unified, and to explore more sophisticated color transfer and
adjustment procedures [1].

5
(a) (b)

(c) (d)

(e) (f)

Figure 5: Luminance-only synthesis, on the images of Figure 1. (a) Luminance channel (Y ) of input
photograph. (b) Color channels (I, Q) of input photograph. (c) Style transfer result in luminance
channel. (d) Combination of synthesized luminance and source color channels. (e) Synthesis, using
luminance-histogram matching before synthesis. (f ) Combination of color and luminance channels,
using luminance-histogram matching before synthesis.

6
(a) (b)

(c) (d) (e)

(f) (g)

Figure 6: Advantages and disadvantages of the two methods presented. (a) Content image from Fig.
1 with marked region. (b) Style image. This example contains prominent brushstrokes. (c) Detail of
a region in the content image (marked in (a)) (d) Luminance-only synthesis for this region. Color is
preserved perfectly, but dependencies between luminance and color are lost: colors do not align well with
strokes. (e) Color-matching synthesis for this region. Dependencies between luminance structure and
colors are preserved but the colors are not preserved exactly. (f ) Content image of a second example
(same as Figure 2 (c) and (d)). Content image has a marked region in the bottom left corner. (g) Same
as (c),(d) and (e) for the second example. Even without brushstrokes, there is an effect. Dependencies
between color and luminance structure are only preserved with the color transfer method.

7
References
[1] Faridul, H. S., Pouli, T., Chamaret, C., Stauder, J., Reinhard, E., Kuzovkin, D., and
Tremeau, A. Colour mapping: A review of recent methods, extensions and applications. Computer
Graphics Forum 35, 1 (2016), 59–88.
[2] Gatys, L. A., Ecker, A. S., and Bethge, M. Image Style Transfer Using Convolutional Neural
Networks. In Proc. CVPR (2016).

[3] Hertzmann, A. Algorithms for Rendering in Artistic Styles. PhD thesis, New York University,
2001.
[4] Hertzmann, A., Jacobs, C. E., Oliver, N., Curless, B., and Salesin, D. H. Image Analogies.
In Proc. SIGGRAPH (2001).

[5] Pitié, F., and Kokaram, A. The Linear Monge-Kantorovitch Linear Colour Mapping for Example-
Based Colour Transfer. In Proc. IETCVMP (2007).
[6] Wandell, B. Foundations of Vision. Sinauer Associates Inc., 1995.