Example-Based Expressive Animation of 2D Rigid

Bodies
Marek Dvorožňák, Pierre Bénard, Pascal Barla, Oliver Wang, Daniel Sýkora

To cite this version:

Marek Dvorožňák, Pierre Bénard, Pascal Barla, Oliver Wang, Daniel Sýkora. Example-Based Ex-
pressive Animation of 2D Rigid Bodies. ACM Transactions on Graphics, 2017, 36 (4), pp.10.
�10.1145/3072959.3073611�. �hal-01514619�

HAL Id: hal-01514619

https://inria.hal.science/hal-01514619
Submitted on 26 Apr 2017

HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est

archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents
entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non,
lished or not. The documents may come from émanant des établissements d’enseignement et de
teaching and research institutions in France or recherche français ou étrangers, des laboratoires
abroad, or from public or private research centers. publics ou privés.
Example-Based Expressive Animation of 2D Rigid Bodies
MAREK DVOROŽŇÁK, Czech Technical University in Prague, Faculty of Electrical Engineering
PIERRE BÉNARD, LaBRI (UMR 5800, CNRS, Univ. Bordeaux), France
PASCAL BARLA, Inria Bordeaux Sud-Ouest, France
OLIVER WANG, Adobe Research
DANIEL SÝKORA, Czech Technical University in Prague, Faculty of Electrical Engineering

(a) style pairs (b) target sequence (c) stylized sequence (d) stroke appearance transfer

Fig. 1. Given a small set of exemplars consisting of computer-generated and hand-drawn 2D animation pairs (a), our method transfers to a new target
sequence produced by physical simulation (b) both the high-level deformations and fine-scale appearance variations (c) present in the example animations.
Optionally, the final appearance of the drawings can be modified by re-synthesizing different stroke textures (d).

We present a novel approach to facilitate the creation of stylized 2D rigid process that requires years of training by an artist and countless
body animations. Our approach can handle multiple rigid objects following hours of labor. Furthermore, style is a highly personalized concept,
complex physically-simulated trajectories with collisions, while retaining a and two different artists never animate exactly in the same way. As
unique artistic style directly specified by the user. Starting with an existing a result, example-based stylization has been a long-standing goal in
target animation (e.g., produced by a physical simulation engine) an artist computer graphics.
interactively draws over a sparse set of frames, and the desired appearance
In this work we focus on rigid bodies, which are particularly
and motion stylization is automatically propagated to the rest of the sequence.
The stylization process may also be performed in an off-line batch process challenging to animate by hand, since multiple objects may collide
from a small set of drawn sequences. To achieve these goals, we combine and rebound in ways that are difficult to plan in advance. Conversely,
parametric deformation synthesis that generalizes and reuses hand-drawn using physical simulation, computer-based methods can quickly
exemplars, with non-parametric techniques that enhance the hand-drawn give rise to rigid body animations with realistic trajectories, but
appearance of the synthesized sequence. We demonstrate the potential of ones that lack expressiveness. Our main goal is therefore to combine
our method on various complex rigid body animations which are created the ease of use of computer-simulated 2D rigid body animations
with an expressive hand-drawn look using notably less manual interventions with the expressive qualities of hand-drawn techniques.
as compared to traditional techniques. To accomplish this goal, we have a number of added requirements.
CCS Concepts: • Computing methodologies → Motion processing; Non- First, editability is of paramount importance to animators, and an
photorealistic rendering; ideal solution should work iteratively, always providing the artist
Additional Key Words and Phrases: 2D animation, example-based synthesis
with the ability to refine the current solution. Second, producing
each hand-drawn frame is time consuming, so a practical example-
ACM Reference format: based 2D animation system should be able to generalize from a
Marek Dvorožňák, Pierre Bénard, Pascal Barla, Oliver Wang, and Daniel
very limited set of artistic inputs, while being able to apply these
Sýkora. 2017. Example-Based Expressive Animation of 2D Rigid Bodies. ACM
edits seamlessly into the dense set of final target frames. These
Trans. Graph. 36, 4, Article 127 (July 2017), 10 pages.
DOI: http://dx.doi.org/10.1145/3072959.3073611 two requirements make example-based 2D animation out of reach
of current data-driven machine learning techniques, due to the
scarcity of data (tens of exemplars, rather than tens of thousands),
1 INTRODUCTION and uniqueness of each style.
Despite the recent success of computer-generated animations, tra- Instead, our approach is inspired by a workflow that is widespread
ditional hand-drawn approaches often yield more expressive and among both traditional and digital animators. A 2D animation is
stylized looks than those produced with the currently available dig- successively produced and refined at three different temporal scales
ital tools. However, creating hand-drawn animations is a tedious (see Figure 3): the full animation scale at which timing, contacts
and trajectories are planed; the pose-to-pose scale, at which the
© 2017 ACM. This is the author’s version of the work. It is posted here for your personal
use. Not for redistribution. The definitive Version of Record was published in ACM overall dynamics and deformations between contacts are considered;
Transactions on Graphics, https://doi.org/http://dx.doi.org/10.1145/3072959.3073611.

ACM Transactions on Graphics, Vol. 36, No. 4, Article 127. Publication date: July 2017.
127:2 • Dvorožňák, M. et al.

and the frame-to-frame scale at which the actual drawings with artistically controllable side, Li et al. [2003] present a sketching in-
secondary deformations and precise collisions are produced. terface that allows a user to guide the deformation of both the input
At the full animation scale, we split the input computer-generated animated skeleton and surface mesh to improve the expressiveness
sequences based on collision events, and we independently analyze of MoCap data. Recently, several 2D sketching systems [Kazi et al.
and stylize each sub-sequence around a hit point, which we call 2014a,b, 2016; Xing et al. 2016] have been developed to simplify the
a key pose. Next, for every sub-sequence, we estimate the spatial production of dynamic illustrations, reproducing most principles
deformations of the hand-drawn exemplars before and after the of character or special effects animation. However these principles
key pose; these are then transferred using a parametric synthesis are essentially encoded as scripted deformations or animated loops
algorithm. Correspondences between sub-sequences are estimated triggered by events, and unlike our work are not tailored to the
by leveraging the physical properties of each frame, which ensures specific style of a given artist.
preservation of appropriate stylistic effects given the forces applied
at each frame. The final frame-by-frame drawings are then synthe- Physics-driven approaches. Physical simulation is a convenient
sized from the artist drawings using a non-parametric technique, way to automatically animate a large number of 2D or 3D bodies,
that captures residual deformations and appearance details. but expressiveness of the resulting motion is restricted by the degree
We show that this organization is necessary to capture both the of complexity modeled by the physical system. For instance, it is
long- and short-range stylistic choices made by the artist, while pro- common to restrict the simulation to rigid bodies for computational
ducing results that have the desired hand-drawn look and feel. When efficiency, while traditional hand-drawn animated objects more
taken as a whole, our method considerably reduces the amount of often resemble deformable bodies governed by exaggerated physical
work needed to create the entire sequence by hand. laws. To enhance basic 3D simulations, multiple works [Chenney
In summary, we present the following contributions: et al. 2002; Garcia et al. 2007; Haller et al. 2004] derive automatic
• a careful analysis of traditional hand-drawn animations, procedural rules to generate squash-and-stretch and temporal effects
especially focusing on the correlation between physical based on motion parameters (velocity, acceleration, etc.) but such
parameters and deformations, methods have limited art-directability.
• a parametric motion synthesis algorithm capable of trans- To allow artistic control, the spacetime [Witkin and Kass 1988]
ferring deformations from exemplars, and dynamic [Barzel and Barr 1988] constraint formulations cast
• an example-based non-parametric stylization technique physical simulation as a constrained optimization problem. Through
capturing the fine-scale drawing appearance. those constraints, the artist can direct the simulation to act as a
physically-plausible interpolation mechanism between key-poses.
Bai et al. [2016] leverage this idea to build a 2D animation system
2 PREVIOUS WORK that combines keyframing of local deformations with physical sim-
We present prior work related to general stylized computer anima- ulation for powerful inbetweening. Although this approach allows
tion, followed by those driven by physical simulations, and finally an unprecedented level of artistic control and manages to reproduce
example-based solutions. many of the principles of animation, it requires the user to specify
control handles, which are constrained and unnatural to use when
Techniques inspired by the principles of animation. From its begin- compared to simply drawing frames, in particular when the artist
ning, one of the goals of computer graphics has been to reproduce desires a precise control over shape outlines.
the expressiveness of traditional hand-drawn 2D animations, while
reducing the cost and effort of producing it. The fundamental prin-
ciples of animation, developed from the late 1920’s to the 1930’s Example-based methods. This family of techniques provides a
at Walt Disney Studio [Thomas and Johnston 1981], play a crucial natural and intuitive interface, where examples are used to capture
role in this expressiveness, and many works have tried to adapt the style and intent of an artist. Such approaches have already
them to digital animation tools. Lasseter [1987] describes these 2D produced impressive results for static images and videos, either
principles – including squash and stretch, timing and spacing, an- using non-parametric texture synthesis [Bénard et al. 2013; Fišer
ticipation and follow-through, arc trajectories and lines of action et al. 2016; Hertzmann et al. 2001; Lu et al. 2012] or more recently
– and how they can be manually applied by an artist to produce with neural networks [Gatys et al. 2016]. Yet these methods are
expressive 3D keyframe animations. mostly restricted to appearance stylization, leaving motion largely
Subsequent work has aimed at fully or partially automatizing untouched.
those effects. Wang et al [2006] describe a simple temporal filter There are some exceptions, such as Bregler et al. [2002] who
that produces anticipation and follow-through as well as squash- propose to capture and re-target motion from existing cartoon ani-
and-stretch deformations by delaying parts of an existing animation, mations by combining a global affine deformation with drawings
represented by 2D polygonal shapes or motion captured (MoCap) interpolation using a key-shape model. Jones et al. [2015] follows
data. Lee et al. [2012] obtain similar effects on segmented objects in a similar approach, connecting the navigation in a simplicial com-
a video by relating a set of representative 2D deformations to the plex [Ngo et al. 2000] with events of a 2D physical simulation. Pose-
modal analysis of the object motion. Focusing on 3D skeletal ani- space interpolation can produce impressive results, but the quality
mation, Noble and Tang [2006] present a tool to automatically bend of the output is highly dependent on a good choice of the key-shapes
the limbs of a character following lines of actions or arcs. On the which an artist has to select and organize manually beforehand.

ACM Transactions on Graphics, Vol. 36, No. 4, Article 127. Publication date: July 2017.
 Example-Based Expressive Animation of 2D Rigid Bodies • 127:3

Source sequence F S Style exemplars F E Target sequence F T Stylized sequence F O

Fig. 2. Stylization analogy setup — given a set of frames F S coming from reference 2D rigid body source animations, corresponding hand-animated exemplars
F E , and a new target animation F T , the synthesis algorithm relates physical parameters in F S and F T to produce the output stylized sequence F O that
resembles F E . (Animations are depicted with onionskins, colored from green to blue according to frame numbers.)

To guide or enrich 3D simulations, example-based approaches 1. Full animation Keys

augment the simulated objects with examples of desirable deforma-
tions [Coros et al. 2012; Jones et al. 2016; Koyama et al. 2012; Martin
2. Pose-to-pose
et al. 2011]. In those approaches, however, exact correspondences
between deformation exemplars are known beforehand and only a
simple parametric deformation with limited number of degrees of 3. Frame-to-frame
freedom is used. Even though this setting may be natural for digital
3D artists, it is again limited and constraining for traditional 2D
animators.
Closest to the traditional animation pipeline, Xing et al. [2015]
present an interactive system for computer-assisted 2D animation.
It combines discrete texture synthesis with an as-rigid-as-possible
deformation model to predict and interpolate drawings based on Fig. 3. Three scales hierarchical decomposition of the animation process,
the current and previous frames. This approach is convincing for based on [Williams 2001, p.67].
frame-by-frame animation, but the spatio-temporal locality of the
analysis makes it unsuited for pose-to-pose planning. Since the
interpolations are solely based on the past drawings using local
affine transformations, the predicted motion and deformations tend
to be unnatural and cannot easily be edited, unless the artist draws stages or temporal scales: (1) the full animation scale, at which tim-
most intermediate frames. ing and spacing are planned by choosing the key events, (2) the
pose-to-pose stage, at which the main dynamics and deformations
are defined between two key poses by drawing “extremes” and
3 OVERVIEW “breakdowns”, and (3) the frame-to-frame scale, corresponding to
Similar in spirit to Image Analogies [Hertzmann et al. 2001], our final straight-ahead “runs” (or drawing passes) during which subtle
algorithm transforms a target 2D rigid body animation F T into an variations and details are added.
output stylized version F O using an example-based transformation Each of the three stages need to be analyzed for transferring
defined by a set of source sequences F S and a corresponding set the style of a hand-drawn animation and our method thus follows
of hand-drawn exemplars F E (Fig. 2). Sequences F S and F T can this organization. First, timing and spacing are specified by the
be computer-generated using, e.g., physical simulation. The style input sequences F S and all animations are subdivided into overlap-
exemplars F E are created by an artist digitally or physically, by ping sub-sequences around key events (Section 4). The style pairs
redrawing a small subset of the source frames F S . In one application, F S : F E are then decomposed into a coarse geometric deformation
the untouched frames of F S can be added to F T , in which case our D and a fine-scale “residual” stylization R (Section 5.1). Our aim is
method can be seen as an interactive style propagation tool. This is to transfer both D and R to the target sequence F T . For each target
shown in the accompanying video where the artist first draws over a sub-sequence independently, a new parametric deformation is syn-
few frames, sees the intermediate result, identifies parts which have thesized by selecting and blending together multiple deformations
not been successfully stylized, provides additional examples, and D coming from similar sub-sequences of the style pairs (Section 5.2).
iterates this procedure until she is satisfied by the stylized result. Finally, sub-sequences are blended together, the fine-scale details are
The key challenge here comes from the fact that F T will typically reintroduced on a frame-by-frame basis by morphing the residual
not contain sub-sequences exactly like those in F S , and thus stylized changes R, and the appearance of individual strokes is changed to
frames from F E cannot simply replace original rigid frames in F T . have a desired look of particular artistic media (Section 6). In the
To tackle this problem, we take inspiration from guidelines in tradi- following, we use the classical “bouncing ball” animation to illus-
tional 2D animation books [Thomas and Johnston 1981; Williams trate the various steps of our algorithm; results on more complex
2001], especially from Richard Williams’ hierarchical decomposi- sequences (collisions between objects, bouncing square, textured
tion of the animation process (see Figure 3). We identify three main objects) are shown in Section 7 and the supplemental material.

ACM Transactions on Graphics, Vol. 36, No. 4, Article 127. Publication date: July 2017.
127:4 • Dvorožňák, M. et al.

fs fe fr
F N1 F2 D D −1
e2 =
e1 M 1,2
F1 (a) (b)

Fig. 5. Deformation analysis — (a) The parametric deformation D is esti-

mated using as-rigid-as-possible registration between the source f s and
Fig. 4. Decomposition into sub-sequences — an input sequence F is sub- exemplar f e frames. (b) The residual frame f r is then computed by applying
divided into sub-sequences (F i ) of N i frames around key events (e i ∈ E) the inverse deformation D −1 on f e .
with an overlap of M i,i +1 frames with the following sub-sequence.
deformable grid matching, we use a single quadratic transformation
as in Müller et al. [2005]. The main advantage of this quadratic
4 TIMING AND SPACING model is that besides shear and stretch, it also captures twist and
The source F S and target F T input sequences consist in 2D rigid bending modes (see Figure 6) which better represent the larger scope
body animations produced using physical simulation. In practice, we of deformations used in traditional animation.
use Box2D [Catto 2007]. Similar to prior work [Kazi et al. 2016], an The output of the image registration phase consists of 12 parame-
arbitrary target rigid body is represented by its proxy geometry, e.g., ters describing the corresponding quadratic deformation that warps
bounding circle, square, or any other closed polygon. In addition pixels p = (x, y)> from the source frame f s ∈ FiS to match pixels
to images, the simulation generates, at each frame and for each p 0 = (x 0, y 0 )> of the stylized frame f e ∈ FiE :
object, a set of physical parameters including the object velocity v
x 0 = a 11x + a 12y + q 11x 2 + q 12y 2 + m 1xy + t 1
and the rotation angle α around the object centroid with respect to (1)
the local trajectory frame. The timing and spacing are dictated by y 0 = a 21x + a 22y + q 21x 2 + q 22y 2 + m 2xy + t 2
the simulation; the artist draws over existing frames, establishing
Written in matrix form:
one-to-one temporal correspondences between F S and F E .
p0 = A Q m
 
The simulation also identifies frames at which semantically impor- t p̃ (2)
tant events E occur, such as contact points or collisions. Following | {z }
the guidelines of Richard Williams [2001], these frames represent D
key poses. Analyzing the hand-drawn exemplars F E , we also ob- where p̃ = (x, y, x 2 , y 2 , xy, 1)>
is p expressed in extended homo-
served that those frames, and their immediate neighbors in time, geneous coordinates, and D is a quadratic transformation matrix
are the ones most stylized by the artist, whereas distant frames are composed of affine A, purely quadratic Q, mixed m, and translation
less modified. In addition, we noticed that the physical parameters t parts: 
a 11 a 12 q 11 q 12 m1 t1
      
along the simulated trajectories before and after these key events
largely influence the artist’s stylization choices, e.g., the magnitude A= Q= m= t= .
a 21 a 22 q 21 q 22 m2 t2
of the deformation.
These observations motivate us to subdivide F S , F T and F E into 5.2 Parametric deformation synthesis
a set of smaller overlapping sub-sequences FiS , FiT and FiE around
Based on traditional hand-drawn animation resources as well as
every key event ei of E. Each sub-sequence Fi contains Ni consecu-
our own observations and discussions with 2D animators, we make
tive animation frames and overlap with the next sub-sequence on
the key hypothesis that deformation is closely tied to motion. As
Mi,i+1 frames. As shown in Figure 4, the overlapping part between
a result, to perform the deformation transfer, we search for corre-
two events resides at frames where there are no abrupt changes in
spondences between source FiS and target F jT sub-sequences using
physical parameters and moderate artistic stylization, making them
physical parameters that describe the frame’s motion (velocity, tra-
most suitable for stitching.
jectory orientation, and the object’s rotation), and we assume that
the matching sub-sequences should undergo similar deformations
5 POSE-CENTERED DEFORMATIONS
D as the source ones.
At this stage, we consider each sub-sequence independently, and fo-
cus on the coarse deformations used by artists when hand-animating
rigid bodies to reproduce effects described in the principles of anima-
tion (squash-and-stretch, arc trajectories, lines of action). Residual
deformations and appearance variation that are not captured by this
coarse deformation will be reintroduced in Section 6.

5.1 Parametric deformation analysis

For each frame of a style pair FiS : FiE , we first estimate a coarse affine A quadratic Q mixed m
parametric deformation D (see Figure 5(a)) using the registration
algorithm of Sýkora et al. [2009] which aligns bitmap images with Fig. 6. Visualization of the 10 modes defined by the quadratic deformation
an as-rigid-as-possible grid deformation. However, instead of the of Müller et al. [2005].

ACM Transactions on Graphics, Vol. 36, No. 4, Article 127. Publication date: July 2017.
 Example-Based Expressive Animation of 2D Rigid Bodies • 127:5

f 2e
Source-target sub-sequence matching. Practically, we define the f 1e f 3e
f 1s f 2s f 3s
following difference metric between a source sub-sequence FiS and
a target sub-sequence F jT :

Diff(FiS , F jT ) = λ vel Vel(FiS , F jT )

D1 D2 D3
+ λ dir Dir(FiS , F jT ) (3) w2
w1 w3
+ λ rot Rot(FiS , F jT ),
D̂
where weights λ vel , λ dir , λ rot are used to balance the influence of
individual terms:
• Vel(FiS , F jT ) measures the difference between rigid body centroid Fig. 7. Blending quadratic deformations D — individual quadratic deforma-
velocities v: tions D1, D2, D3 estimated from the source frames f 1s , f 2s , f 3s and their
N corresponding stylized counterparts f 1e , f 2e , f 3e are blended together using
Vel(FiS , F jT ) = ||vn (FiS ) − vn (F jT )|| 2 , the weights w 1, w 2, w 3 to obtain the resulting quadratic deformation D̂.
Õ
(4)
n=1

• Dir(FiS , F jT ) penalizes discrepancy of the trajectory orientation δ : where ⊕ computes a weighted average of circular quantities. The
N remaining coefficients of rotation-free quadratic transformations
Dir(FiS , F jT ) = ||δn (FiS ) δn (F jT )|| 2 , D10 · · · DK0 are similarly computed:
Õ
(5)
n=1
D̂ 0 = w 1 D10 + . . . + w K DK0 (8)
where computes the smallest difference between two angles,
• Rot(FiS , F jT ) accounts for differences in the rotation α of the rigid
where D 0 = R -α D. Finally, the blended quadratic transformation
body around its centroid: matrix D̂ is constructed from α̂ and D̂ 0 :
N
Rot(FiS , F jT ) = ||α n (FiS ) α n (F jT )|| 2 .
Õ
(6) D̂ = R α̂ D̂ 0 (9)
n=1
When computing the metric we assume that both sub-sequences Data augmentation. To generate plausible results even when the
are centered at a key event and have the same number of frames global orientation or scale of the target trajectory departs consid-
N . This can be done by resampling the original trajectories to have erably from the available exemplars, we enrich the set of source
equidistant samples according to their arc length. The longest sub- analogies by scaling, rotating, and flipping the input sequences. We
sequence is trimmed to have the same length as the shortest one. can directly extract the set of required rotation angles γ by ana-
lyzing the target simulation. Based on our experiments, we also
Deformation blending. Since it is unlikely that any source sub-
use 5 scaling factors ρ between 0.2 to 1 and allow symmetries with
sequence perfectly matches a given target sub-sequence F jT , we
respect to the vertical axis only (to preserve gravity effects). For
retrieve K nearest neighbor sub-sequences F 1S . . . F KS instead of a rotationally symmetric objects of order n, we modify the operator
single one. For each frame in F jT , we then compute its stylized in a way that it outputs zero difference for angles k 360
◦
n where the
version as a combination of K quadratic transformations D1 . . . DK apperance of the rotated object is the same. For the circle the order
from the K best corresponding frames in source sub-sequences using of rotational symmetry is infinite so instead we set λ rot = 0.
weights w 1 . . . w K proportional to their similarity: The drawback of the augmentation is that it may lead to incorrect
1/Diff(FkS , F jT ) stylizations when the source exemplars are far from the target mo-
w k = ÍK S T
, k ∈ [1 . . . K] tion. For example, a source exemplar corresponding to a small jump
κ=1 1/Diff(Fκ , F j ) will not be equivalent to a source exemplar with a higher jump. To
account for this, we dampen the resulting quadratic transformation
where normalization is used to obtain the partition of unity. See Fig-
D̂ by computing a weighted blend of D̂ with the identity matrix I
ure 7 for an overview of this blending scheme (with K = 3) which
using weight ξ , proportional to the ratio of the average source and
adds robustness to the matching process and gives more stable
target velocities:
results than simply using the single best match (K = 1).
ÍN
To perform a meaningful interpolation of the rotational part of the vn (FiS )
transformation, the affine part A of the matrix D is factorized using D̂ = ξ D̂ + (1 − ξ ) I with ξ = Ín=1N
(10)
T
polar decomposition [Higham and Schreiber 1990] into a linear n=1 vn (F j )
stretch U (not used directly) and a rotation matrix R α , from which
However, if rotational invariance is not (even approximately) satis-
the rotation angle α = arctan(r 11 /r 21 ) is extracted. A weighted
fied, orientation augmentation cannot be used, and the artist will
blend is computed on α 1 . . . α K :
need to prepare a set of additional exemplars corresponding to the
α̂ = w 1 α 1 ⊕ . . . ⊕ w K α K (7) correct rotational motion.

ACM Transactions on Graphics, Vol. 36, No. 4, Article 127. Publication date: July 2017.
127:6 • Dvorožňák, M. et al.

f 1r f 2r f 3r
Contact points adjustment. Our synthesis process does not guar-
antee that the resulting animation preserves the alignment with
obstacles at contacts. This issue bears resemblance to the foot step
detection/correction mechanism used when blending motions in
D̂ D̂ D̂
fˆ1r fˆ3r fˆ2r skeletal animation. Yet our problem is simpler since we know the
position of contact points; we can easily verify whether the spatial
alignment with obstacles is preserved. If not, we simply shift or
w2 slightly rotate the synthesis result so that it aligns perfectly with
w1 w3 the obstacle at collision time. To estimate the corresponding trans-
n-way morph lation and rotation we use again image registration algorithm of
Sýkora et al. [2009]. To avoid ambiguity in translation along the
fˆr obstacle during the registration, we restrict the centroid of the syn-
thesized drawing to move perpendicularly to (along the normal of)
the nearest obstacle.

Fig. 8. Synthesis of fine-scale details R — the synthesized quadratic de- Texturing. We support two options to apply a texture inside the
formation D̂ is applied to individual residual frames f 1r , f 2r , f 3r producing deformed drawings. The first one takes as input a static image
their deformed counterparts fˆ1r , fˆ2r , fˆ3r . Those are then blended together whose content is constant during the full sequence. This image is
using n-way morphing [Lee et al. 1998] to produce the resulting frame fˆr . first rotated according to the target sequence orientation, then it is
registered with every residual frame fir using [Glocker et al. 2008],
and finally replaces those during the subsequent fine-scale synthesis
6 FRAME-BY-FRAME STYLIZATION steps (see Figure 13(c)). If the content of the texture varies in time,
the artist needs to provide two versions of the style exemplar: one
Although the parametric transformations D capture most of the only showing the outline of the drawing and another with the full
dominant global deformations, there are still small residual deforma- texture (see Figure 17). The former is used for quadratic registration
tions and appearance variations R (e.g., sketch lines of the drawings) whereas the latter is copied during the frame-by-frame synthesis.
which cannot be simply described by the quadratic deformation
model. These residual changes represent a very important part of Stroke appearance transfer. To offer additional artistic controls,
the stylization, as they provide much of the uniqueness of traditional we optionally allow the stroke appearance to be re-synthesized by
hand-drawn, as opposed to computer-generated animation. exemplar using StyLit [Fišer et al. 2016]. This can also help suppress
resampling artifacts that may occur when applying the quadratic
Extraction of the residual. Due to the parametric deformation and free-form deformations. In practice we replace the complex
model, extracting R from the source exemplars is straightforward. illumination-based guidance with a simple gray-scale guiding chan-
We compute and store the residual frames in FiR by “rectifying” nel G that softly demarcates positions of strokes in the source and
the example frames in FiE using the inverse transformation to the in the target image (see Figure 9).
deformation D estimated in Section 5.1 (see Figure 5(b)).
(c)
Synthesis of fine-scale details. For a given target frame in F jT , we
now want to re-introduce the residual variations to the synthesized (d)
parametric transformation D̂. As illustrated in Figure 8, we deform
the corresponding residual frames in f 1r , . . . , f Kr using D̂ to produce
(e)
a set of deformed example frames fˆ1r . . . fˆKr . We then compute a set
(a) (b)
of pairwise pixel-level warping fields ϕκ,κ 0 : fˆκr → fˆκr0 ∀(κ, κ 0 ) ∈
{1 . . . K }2 using deformable image registration [Glocker et al. 2008]. Fig. 9. Stroke appearance transfer based on StyLit algorithm [Fišer et al.
Finally we apply a weighted n-way morph [Lee et al. 1998] to pro- 2016] — it takes as input an exemplar drawing containing few strokes created
duce a single output frame fˆr by displacing and blending pixels in using the desired drawing media, e.g., color pencil (a), its corresponding
fˆ1r . . . fˆKr according to the same weights w 1 . . . w K as in Section 5.2 guiding channel (b), the target frame synthesized using our method (c) and
and the warping fields ϕκ,κ 0 . its guiding channel (d), and produces the resulting appearance transfer (e).

Sub-sequence stitching. Since the style transfer is applied indepen- To generate G we filter the input stroke style exemplar as well
dently on each animation sub-sequence, stylized overlapping sub- as the output frame using the FDoG filter [Kang et al. 2007] which
sequences need to be stitched together to avoid abrupt changes. We suppresses local discrepancies caused by physical properties of the
apply the same approach as described in previous paragraphs, but artistic medium (in the stroke style exemplar) or by errors caused
now only with two frames and with blending weights proportional by warping and blending pixels (in the target stylized frame) and
to the temporal distance τ : to stitch overlapping sub-sequences i produces clean stroke outlines. Then a Gaussian blur is applied on
and i + 1, we use w i = τ /Mi,i+1 and w i+1 = 1 − τ /Mi,i+1 , where this clean outline mask and the resulting bitmap image is normalized
Mi,i+1 is the number of frames in the transition. and used as a guiding channel for the StyLit algorithm.

ACM Transactions on Graphics, Vol. 36, No. 4, Article 127. Publication date: July 2017.
 Example-Based Expressive Animation of 2D Rigid Bodies • 127:7

Fig. 10. Style pair database — overview of the source animations and their corresponding stylizations of a bouncing ball drawn by a professional artist.

target sequence synthesis (one example trajectory) synthesis (all example trajectories) ground truth stylization

Fig. 11. Results of our synthesis algorithm — (from left to right) target animation computed using physical simulation, synthesis using only one exemplar
trajectory, synthesis using all available exemplar trajectories shown in Figure 10, ground truth stylization drawn by a professional artist.

7 RESULTS
We implemented our approach using a combination of Matlab/C++
code, and integrated it as a plug-in into “TVPaint Animation”, a pro-
fessional 2D animation software. In all our experiments, the input se-
quences were split into sub-sequences of Ni = 23 #(ei−1 , ei+1 ) frames,
where #(ei , e j ) is the number of frames between the key events ei
and e j . The overlap Mi,i+1 was set to 23 #(ei , ei+1 ). For source-target
sub-sequence matching, N was set to 20 and the weights were set
to λ vel = 1, λ dir = 1, λ rot = 50 to encourage consistent rotation.
To validate our method we designed two practical usage scenarios: target synthesis
(1) an off-line scenario where an existing database of pre-made style
exemplars is used to perform stylization of new more complex Fig. 12. Result of our synthesis algorithm for two bouncing balls — target
sequences and (2) an interactive scenario where an animator specifies sequence computed using physical simulation (left); resulting synthesis
a set of sparse style exemplars in the target sequence and those are computed using all available exemplar trajectories shown in Figure 10 (right).
then used as a source for the synthesis of the entire sequence.
In the case of the off-line scenario, to create the database, we matching step of Sýkora et al. [2009] and the graph-cut computa-
asked a professional animator to stylize a few physical simulations tion in Glocker et al. [2008]. In Figure 14 we also demonstrate the
with a single ball (see Figure 10) and square bouncing on the ground synthesized sequences with additional stroke appearance transfer
(see Figure 1). This database of exemplars was then used to stylize which further enhances the hand-drawn look and feel.
more complex physical simulations containing multiple interacting For the interactive scenario, we let the artist work in TVPaint
objects in different environment (see Figures 1, 11, 12, 16, 13, 17 Animation, a professional animation software, using our plug-in to
and the supplementary video). For this scenario, we precompute iteratively improve the stylization of a target sequence. The artist
parametric deformations for each exemplar in the database. Our first selects a few frames where a more pronounced stylization
implementation can precompute 10 frames in about 13 seconds on is required and draws over them. The target sequence is then re-
average. Using the precomputed deformations, synthesis of an ani- synthesized taking into account those modified frames, and the artist
mation containing 100 frames lasts approximately 56 seconds. The can inspect the result, tweak already modified frames, or provide
most time consuming parts of the pipeline are the image registration stylization for additional frames (see the supplementary video for
phases. Parallel processing could be used to accelerate the block the recording of a live editing session). In this scenario, we compute

ACM Transactions on Graphics, Vol. 36, No. 4, Article 127. Publication date: July 2017.
127:8 • Dvorožňák, M. et al.

(a) target sequence (b) synthesis (c) textured result

Fig. 13. Result of our synthesis algorithm for more colliding balls — (a) target sequence computed using physical simulation; (b) resulting synthesis computed
using all available exemplar trajectories shown in Figure 10; (c) “stone wheel” texture applied during the frame-by-frame synthesis.

our method ground truth stylization D only D plus R

Fig. 14. Stroke appearance transfer applied to our stylized result (left), and Fig. 15. Comparison — synthesis using only the quadratic deformation
on the same sequence drawn by a professional artist (right). model D (left), and with the full processing pipeline which includes both
the quadratic deformation D and the residual changes R (right). Note
how the addition of the residual changes R significantly improves the
parametric deformations only for newly stylized frames and then expressiveness of the resulting animation.
the synthesis is executed with comparable computational overhead
as in the off-line scenario. In the captured session, the artist stylized
only 5 percent of the frames until she was satisfied with the result. 8 LIMITATIONS AND FUTURE WORK
The method could even work with a single stylized frame, but it We have shown that our approach can produce good results using
will obviously produce visually pleasing results only when a few only a small subset of exemplar frames, and as a result can notably
frames before and after a collision event are provided. lower the amount of manual work required to create a variety of
To confirm that our approach reproduces the hand-drawn style 2D animations. However, those are currently restricted to simple
well, we asked the professional artist to stylize the entire sequence rigid bodies, and some practical – rather than conceptual – limita-
frame-by-frame (ground truth), and we also let our algorithm syn- tions need to be addressed to allow the stylization of more complex
thesize the same sequence using exemplars (1) from the database sequences.
and (2) from a fraction of the stylized frames in the ground truth se- First, as mentioned in Section 4, for simplicity, our method does
quence (see Figure 11). Then, we asked two professional animators not explicitly model the spacing variations that an artist may intro-
and three casual observers to compare the three results side-by- duce compared to the physical simulation (e.g., ease-in / ease-out
side and assess the overall motion stylization quality. In most cases effects). This is especially noticeable in the dynamics of collisions.
the results were almost indistinguishable from ground truth for To a certain extent, those variations are implicitly captured by the
non-professional observers. Professional artists were able to see translational part of the parametric transformation, but only as a
some discrepancies namely in frames of which assumed stylization linear frame-by-frame approximation when it should maybe be mod-
diverted significantly from the available exemplars, however, the eled as a continuous function along the rigid body trajectory. This
overall response was very positive and they highly appreciate the can be seen in instances where a shape deforms along the contact
advantage of having a trade-off between the final visual quality and plane just before or right after the collision (levitation effect).
the time required to produce the exemplars. Besides, our approach takes advantage of rigid body symmetries
In addition to the comparison with a ground truth sequence, we which are helpful namely for the synthesis of rotations. In particular,
also provide an additional comparison showing the necessity of exemplars of symmetric counterparts can be reused in situations
the decomposition into a quadratic deformation D and residual where there is no corresponding sequence available in the original
changes R. We synthesize sequences using only our parametric orientation. However, for asymmetric objects this simplification
model and then compare with those synthesized using the full pro- cannot be used and thus more exemplars need to be provided in
cessing pipeline (see Figure 15). order to be able synthesize consistent orientations. To further reduce

ACM Transactions on Graphics, Vol. 36, No. 4, Article 127. Publication date: July 2017.
 Example-Based Expressive Animation of 2D Rigid Bodies • 127:9

target sequence synthesis

Fig. 16. Result of our synthesis algorithm for two interacting squares — the style pairs of Figure 1 are used, these only contain collisions with the ground plane.

the burden on the artist in such cases, it would be required to model 9 CONCLUSION
the rotation of the example object separately and allow some amount We have presented a method allowing to perform style transfer from
of re-targeting during the transfer to the novel sequence. an existing 2D hand-drawn animation exemplar to a more complex
The success of our method also strongly depends on the result of rigid body animation. To the best of our knowledge this is the first
the automatic image registration phase, during which the quadratic attempt to provide a solution for such a challenging task. Despite
deformation model is fit to the stylized exemplar. For elongated the recent success of example-based stylization techniques focusing
objects such as the car in Figure 17, it would be interesting to explore on appearance transfer, animation is still a mostly unexplored area
the use of multiple quadratic deformation clusters such that the front and we believe that the results provided in this paper will motivate
of the object may be stylized earlier than its rear along the motion other researchers to explore and solve the large variety of challenges
trajectory. More complex or excessive stylization may not fit well that emerges when considering transfer of motion style to more
to the available degrees of freedom and consequently the resulting complex articulated objects such as cartoon characters.
residual changes may become overly large. This can cause difficulties
during the deformable image registration and the subsequent n-way ACKNOWLEDGEMENTS
morphing phase. As future work, we would like to explore different
We would like to thank Zuzana Studená for preparation of hand-
parametric models which would be able to describe a larger variety
drawn animations used in the paper, Petra Hapalová for integrating
of deformations occurring in hand-drawn animations. Those more
our algorithm into TVPaint, Vojtěch Votýpka for the recording of
complex models will probably require a better image registration
interactive sessions, and all anonymous reviewers for insightful
algorithm with a hierarchical processing that is able to adaptively
comments and suggestions. This research was funded by Adobe
add individual degrees of freedom to avoid getting stuck in some
and has been supported by the Technology Agency of the Czech
erroneous configurations. It would also require a more advanced
Republic under research program TE01020415 (V3C – Visual Com-
blending scheme.
puting Competence Center) and by the Grant Agency of the Czech
When the exemplar animations differ considerably as compared
Technical University in Prague, grant No. SGS16/237/OHK3/3T/13
to the target motion our method just picks a set of most similar ones
(Research of Modern Computer Graphics Methods).
which likely would not be appropriate. Moreover, when there is a
large difference in velocities, the dampening phase tends to remove
the stylization effect. Both effects are undesirable and serve as a
REFERENCES
Yunfei Bai, Danny M Kaufman, C. Karen Liu, and Jovan Popović. 2016. Artist-directed
signal for the artist to provide some more appropriate exemplars. dynamics for 2D animation. ACM Transactions on Graphics 35, 4 (2016), 145.
A last drawback of our technique is that it requires multiple pixel- Ronen Barzel and Alan H. Barr. 1988. A Modeling System Based on Dynamic Constraints.
based warping operations to produce the final shape. Although In Proceedings of the 15th Annual Conference on Computer Graphics and Interactive
Techniques (SIGGRAPH ’88). ACM, 179–188.
their number can be minimized by applying compound quadratic Pierre Bénard, Forrester Cole, Michael Kass, Igor Mordatch, James Hegarty, Martin Se-
transformations, subsequent image re-sampling may still introduce bastian Senn, Kurt Fleischer, Davide Pesare, and Katherine Breeden. 2013. Stylizing
animation by example. ACM Transactions on Graphics 32, 4 (2013), 119.
artifacts that can alter the genuine appearance of the used artistic Christoph Bregler, Lorie Loeb, Erika Chuang, and Hrishi Deshpande. 2002. Turning
media. This limitation can be partially alleviated by additional post- to the Masters: Motion Capturing Cartoons. ACM Transactions on Graphics 21, 3
processing steps such as the proposed appearance re-synthesis or (2002), 399–407.
Erin Catto. 2007. Box2d – a 2D physics engine for games. http://www.box2d.org.
vectorization. (2007).
Combining all these improvements, we plan to extend our method Stephen Chenney, Mark Pingel, Rob Iverson, and Marcin Szymanski. 2002. Simulat-
to articulated objects such as cartoon characters. In this case a more ing Cartoon Style Animation. In Proceedings of International Symposium on Non-
photorealistic Animation and Rendering. 133–138.
complex, potentially hierarchical parametric model will be needed. Stelian Coros, Sebastian Martin, Bernhard Thomaszewski, Christian Schumacher,
Robert Sumner, and Markus Gross. 2012. Deformable Objects Alive! ACM Transac-
tions on Graphics 31, 4 (2012), 69.
Jakub Fišer, Ondřej Jamriška, Michal Lukáč, Eli Shechtman, Paul Asente, Jingwan Lu,
and Daniel Sýkora. 2016. StyLit: Illumination-guided Example-based Stylization of
3D Renderings. ACM Transactions on Graphics 35, 4 (2016), 92.

ACM Transactions on Graphics, Vol. 36, No. 4, Article 127. Publication date: July 2017.
127:10 • Dvorožňák, M. et al.

target sequence

style exemplar synthesis

Fig. 17. Result of our synthesis algorithm for a cartoon car – a more complex polygonal proxy is used for the simulation (top left) and registered to the outline
of the stylized exemplar (bottom left). During synthesis, the exemplar with texture varying in time (middle left) is used instead.

Marcos Garcia, John Dingliana, and Carol O’Sullivan. 2007. A Physically Based Defor- Seungyong Lee, George Wolberg, and Sung Yong Shin. 1998. Polymorph: Morphing
mation Model for Interactive Cartoon Animation. In Workshop in Virtual Reality Among Multiple Images. IEEE Computer Graphics and Applications 18, 1 (1998),
Interactions and Physical Simulation "VRIPHYS" (2007). Eurographics Association. 58–71.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. 2016. Image style transfer Sun-Young Lee, Jong-Chul Yoon, Ji-Yong Kwon, and In-Kwon Lee. 2012. CartoonModes:
using convolutional neural networks. In Proceedings of IEEE Conference on Computer Cartoon stylization of video objects through modal analysis. Graphical Models 74, 2
Vision and Pattern Recognition. 2414–2423. (2012), 51–60.
Ben Glocker, Nikos Komodakis, Georgios Tziritas, Nassir Navab, and Nikos Paragios. Yin Li, Michael Gleicher, Ying-Qing Xu, and Heung-Yeung Shum. 2003. Stylizing Motion
2008. Dense Image Registration Through MRFs And Efficient Linear Programming. with Drawings. In Proceedings of the 2003 ACM SIGGRAPH/Eurographics Symposium
Medical Image Analysis 12, 6 (2008), 731–741. on Computer Animation (SCA ’03). Eurographics Association, 309–319.
Michael Haller, Christian Hanl, and Jeremiah Diephuis. 2004. Non-photorealistic Jingwan Lu, Fisher Yu, Adam Finkelstein, and Stephen DiVerdi. 2012. HelpingHand:
Rendering Techniques for Motion in Computer Games. Computers in Entertainment example-based stroke stylization. ACM Transactions on Graphics 31, 4 (2012), 46.
2, 4 (2004). Sebastian Martin, Bernhard Thomaszewski, Eitan Grinspun, and Markus Gross. 2011.
Aaron Hertzmann, Charles E Jacobs, Nuria Oliver, Brian Curless, and David H Salesin. Example-based Elastic Materials. ACM Transactions on Graphics 30, 4 (2011), 72:1–
2001. Image analogies. In Proceedings of the 28th annual conference on Computer 72:8.
graphics and interactive techniques. ACM, 327–340. Matthias Müller, Bruno Heidelberger, Matthias Teschner, and Markus Gross. 2005.
Nicholas J. Higham and Robert S. Schreiber. 1990. Fast polar decomposition of an Meshless Deformations Based on Shape Matching. ACM Transactions on Graphics
arbitrary matrix. SIAM J. Sci. Statist. Comput. 11, 4 (1990), 648–655. 24, 3 (2005), 471–478.
Ben Jones, Jovan Popovic, James McCann, Wilmot Li, and Adam W. Bargteil. 2015. Dy- Tom Ngo, Doug Cutrell, Jenny Dana, Bruce Donald, Lorie Loeb, and Shunhui Zhu. 2000.
namic sprites: Artistic authoring of interactive animations. Journal of Visualization Accessible Animation and Customizable Graphics via Simplicial Configuration
and Computer Animation 26, 2 (2015), 97–108. Modeling. In Proceedings of the 27th Annual Conference on Computer Graphics and
Ben Jones, Nils Thuerey, Tamar Shinar, and Adam W. Bargteil. 2016. Example-based Interactive Techniques (SIGGRAPH ’00). ACM, 403–410.
Plastic Deformation of Rigid Bodies. ACM Transactions on Graphics 35, 4 (2016), Paul Noble and Wen Tang. 2006. Automatic Expressive Deformations for Stylizing
34:1–34:11. Motion. In Proceedings of the 4th International Conference on Computer Graphics
Henry Kang, Seungyong Lee, and Charles K. Chui. 2007. Coherent Line Drawing. and Interactive Techniques in Australasia and Southeast Asia (GRAPHITE ’06). ACM,
In Proceedings of International Symposium on Non-photorealistic Animation and 57–63.
Rendering. 43–50. Daniel Sýkora, John Dingliana, and Steven Collins. 2009. As-Rigid-As-Possible Image
Rubaiat Habib Kazi, Fanny Chevalier, Tovi Grossman, and George W. Fitzmaurice. Registration for Hand-Drawn Cartoon Animations. In Proceedings of International
2014a. Kitty: Sketching dynamic and interactive illustrations. In Proceedings of ACM Symposium on Non-photorealistic Animation and Rendering. 25–33.
Symposium on User Interface Software and Technology. 395–405. Frank Thomas and Ollie Johnston. 1981. The illusion of life : Disney animation. Disney
Rubaiat Habib Kazi, Fanny Chevalier, Tovi Grossman, Shengdong Zhao, and George W. Editions, New York.
Fitzmaurice. 2014b. Draco: Bringing life to illustrations with kinetic textures. In Jue Wang, Steven M. Drucker, Maneesh Agrawala, and Michael F. Cohen. 2006. The
Proceedings of ACM Conference on Human Factors in Computing Systems. 351–360. Cartoon Animation Filter. ACM Transactions on Graphics 25, 3 (2006), 1169–1173.
Rubaiat Habib Kazi, Tovi Grossman, Nobuyuki Umetani, and George Fitzmaurice. 2016. Richard Williams. 2001. The animator’s survival kit. Faber and Faber, London, New
Motion Amplifiers: Sketching Dynamic Illustrations Using the Principles of 2D York.
Animation. In Proceedings of ACM Conference on Human Factors in Computing Andrew Witkin and Michael Kass. 1988. Spacetime Constraints. In Proceedings of the
Systems. 4599–4609. 15th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH
Yuki Koyama, Kenshi Takayama, Nobuyuki Umetani, and Takeo Igarashi. 2012. ’88). ACM, 159–168.
Real-time Example-based Elastic Deformation. In Proceedings of the ACM SIG- Jun Xing, Rubaiat Habib Kazi, Tovi Grossman, Li-Yi Wei, Jos Stam, and George Fitz-
GRAPH/Eurographics Symposium on Computer Animation (SCA ’12). Eurographics maurice. 2016. Energy-Brushes: Interactive Tools for Illustrating Stylized Elemental
Association, 19–24. Dynamics. In Proceedings of the 29th Annual Symposium on User Interface Software
John Lasseter. 1987. Principles of Traditional Animation Applied to 3D Computer and Technology (UIST ’16). ACM, 755–766.
Animation. In Proceedings of the 14th Annual Conference on Computer Graphics and Jun Xing, Li-Yi Wei, Takaaki Shiratori, and Koji Yatani. 2015. Autocomplete Hand-drawn
Interactive Techniques (SIGGRAPH ’87). ACM, 35–44. Animations. ACM Transactions on Graphics 34, 6 (2015), 169:1–169:11.

ACM Transactions on Graphics, Vol. 36, No. 4, Article 127. Publication date: July 2017.