Author manuscript, published in "Workshop on Intelligent Cinematography and Editing (2012)"

A Review of Film Editing Techniques for Digital Games

Remi Ronfard
INRIA, LJK, Université de Grenoble, France
[email protected]

1. INTRODUCTION the editor is left with fewer or no options. As a result, the

Automated film editing involves the generation of the po- scene may have to be shot again from another angle. This is
sition, orientation, motion and selection of virtual cameras usually not a problem because it is easy (and cheap) to do
in interactive 3D graphics applications. There is a pressing so. When implementing automated systems, it is important
demand for techniques to assist and automate the control of to take the rules of editing into account in the early stages
virtual cameras in the computer games industry where the of planning and controlling the camera. Otherwise, a lot of
rapid development of personal computers and high perfor- effort will be wasted on attempting to edit shots that ”do
mance consoles has led to substantial improvements in the not cut together”. This will be examined in depth in Section
visual fidelity of games. The goal of this survey is to charac- 3.
terize the spectrum of applications that require automated
film editing, present a summary of state-of-the-art models In traditional cinematography, cutting can be taken into ac-
hal-00694444, version 1 - 4 May 2012

and techniques, and identify both promising avenues and count by following one of several working practises. We men-
hot topics for future research tion three of them.

2. CINEMATOGRAPHY AND EDITING 1. Cutting in the head means that the director has al-
One fundamental part of cinematography, as outlined in ready decided very precisely every single shot, usually
Maschielli’s 5C’s of cinematography [14] is to provide shots in the form of a storyboard. In that case, it suffices to
that can easily be edited together. In the early days of cin- shoot each action or beat in the screenplay from a sin-
ema, the interplay between cinematography and editing was gle viewpoint. Textbooks in film-making warn against
a matter of trial and error. As noted by Barry Salt [20], it the dangers of the method because it cannot recover
took several years before cinematographers and editors un- easily from errors in planning.
derstood the ”exit left enter right” editing rule. Before that,
the rule was usually obeyed because it appeared to work bet- This approach is very suitable for real-time applica-
ter in most cases. But the ”wrong” solution was still used tions. It consists in planning the editing first, resulting
from time to time. When it finally became clear what the in a list of shots that can then be rendered exactly as
”right” solution was, cinematographers stopped shooting the planned following the timeline of the final movie.
alternate solution because they knew it was useless. After One drawback of that approach is that the animation
more than a century of cinema, good professional cinematog- itself cannot always be predicted in all its actual de-
raphers have thus ”internalized” the rules of editing in such tails. As a result, it may be difficult to plan exactly
a way that they can avoid shots that will not cut together. when to cut from shot to shot.

In games, we are probably still at an earlier stage because it 2. Three-take technique A common variant of ”cutting
is not yet quite clear how the rules of cinematography should in the head” consists in shooting a little more of the
translate for an interactive game, which is a very different action from each planned camera position. As a result,
situation from a movie. each action is shot from three camera positions - one
according to the shot list, one from the immediately
In computer graphics, the camera is controlled by animators. previous viewpoint and one from the next viewpoint.
A good professional animator should have a similar sense of This has the advantage that the exact cutting point
which shots will cut together. When this is not the case, can be resolved at a later stage.
3. Master-shot technique Another common practice
consists in planning all the camera works for shooting
the scene in one continuous take - the ”master shot”
- and then adding shots of various sizes to show the
details of the action in various sizes (close-ups and
medium shots). Editing can then more carefully pre-
pared by insuring that all those shots will cut nicely
with the master shot, resulting in a typical sequence
of ”Master-Closeup-Master-Closeup”, etc.
 Note that those techniques are very useful in practice be- preventing editing errors of all orders. But that is of course
cause they are more general than ”film idioms” where the not the entire story because there are still infinitely many
camera positions are prescribed once and for all. ”correct” camera pairs that can be cut together at any given
time. A second part of automated editing is therefore to
3. AUTOMATIC CAMERA EDITING evaluate when to cut to which shot.
This section covers the approaches that draw on a theory
of film editing for planning and performing camera place- The classical Hollywood concept of editing [14] recommends
ment and composition. Here scenes are described in terms that successive shots should minimize perceptually disrup-
of actions and communicative goals that must be translated tive transitions. The modern viewpoint [9] stresses the con-
into successive shots. Cutting between cameras adds con- sistency of the narrative structure which overrule disturbing
siderable freedom in the focalization and order of presenta- transitions, as attention will primarily be directed to grasp-
tion of the visual material. Cutting between cameras also ing the succession of significant events in the story. A good
introduces constraints. We review the most important con- computational theory of film editing should probably stand
straints and corresponding rules (180 degree rule, 60 degree in the middleground between those two viewpoints. On the
rule) and explain how they can be expressed and solved al- one hand, it is difficult to get a good model of ”perceptu-
gorithmically. Then, we review the principles that can be ally disruptive transitions”. At best, a computational model
used to evaluate the quality of a shot sequences and the al- may be expected to avoid the most obvious mistakes, still
gorithmic strategies that can be used to solve for the best leaving a large number of possibilities. On the other hand,
sequence. Finally, we review the strengths and limitations the narrative structure of an animated scene may not always
for some of the existing systems proposed for real-time, live- be easily uncovered, again leaving multiple choices.
editing [He96,Funge98] as well as offline, post-production
editing [Elson07] and sketch promising future directions for Few editors have written about their art with more depth
research in this area. than Walter Murch [16]. In his book, he introduces a Rule of
Six with six layers of increasing complexity and importance
Automatic film editing has a long history, dating back at in the choice of how and when to cut between shots:
hal-00694444, version 1 - 4 May 2012

least to Gilles Bloch’s PhD thesis in 1986 [1]. In this sec-

tion, we present both procedural and declarative approaches.
Three-dimensional space of action. Respect of 3-D con-
A procedural approach to movie editing builds an explicit
tinuity in the real world: where people are in the room
solution. A good example of that is the Virtual Cinematog-
and their relations to each other (accounts for only 4
rapher system (VC) where each idiom is implemented as
% of what makes a good cut)
finite state machine. A reactive approach is essentially a
procedural approach where multiple courses of events can Two-dimensional space of screen. Respect of 2D conti-
be taken into account. A declarative approach states the nuity. Where people appear on the screen. Where the
constraints and rules and lets a separate solver find a so- lines of action, look, movement project on the screen.
lution that meets all the constraints, and/or maximizes a (5 %)
measure of quality.
Eye-trace. Respect of the audience’s focus of interest be-
3.1 Editing rules and constraints fore and after the cut. (7 %)
It is important to understand the motivation between the
Rhythm. Cut at a moment which is both right and inter-
so-called ”rules of editing”. Most of them are in fact con-
esting. (10 %)
straints. What that means is that it may not be possible to
cut from any two arbitrary cameras. Why not ? Because Story. Cut in a way that advances the story. (23 %)
some transitions may provoke false inferences. For a cut be-
tween two shots to work, it is fundamental that it does not Emotion. Cut in a way that is true to the emotion of the
break the logic of human perception. moment. (accounts for 51 % of what makes a good
cut).
Psychologists dŠYdewalle and Vanderbeeken offer a use-
ful classification of editing errors [8]. Editing errors of the
”first order” are small displacements of the camera or image In 3-D animation, the three-dimensional space of action is
size, disturbing the perception of apparent movement and always in continuity as long as we perform live editing. So we
leading to the impression of jumping. Editing errors of the only really need to be concerned with the other five criteria.
”second order” are violations of the spatial-cognitive repre- We can attempt to build a computational theory of film
sentation of the 3-D scene. One example is the 180-rule editing based on this reduced rule of five if we know how
violation, where the camera crosses the line between two to evaluate each of the five criteria AND find a consistent
actors and as a result the actors appear to swap positions. way to rank possible cuts and shots using a combination of
Another example is the motion continuity violation, when them.
the camera crosses the line of an actor’s movement and as
a result the actor appears to change directions. Editing er- 3.1.1 Two-dimensional continuity.
rors of the ”third-order” are when successive shots have too Two-dimensional continuity is easiest to evaluate by com-
little in common to be integrated into a single chronological puter. All the programmer has to do is project the various
sequence of events. lines (of action, of looks, of movements, etc) to the camera
plane and check that they remain consistent. This is a direct
An important part of automated movie editing consists in application of projective geometry.
 Two-dimensional continuity can be insured by adhering to editors. But predicting where the audience is looking re-
the following rules of the so-called classical continuity style: mains hard even for editors. In the context of stereoscopic
3-D Film director James Cameron (who also edits his own
movies) phrased it as follows: ”You can only converge to one
Line of action The relative ordering of characters must re- image plane at a time – make sure it is the place the audi-
main the same in the two shots. ence (or the majority of the audience) is looking. If it’s Tom
This is the basis for the 180 degree rule, which forbids Cruise smiling, you know with 99% certainty where they’re
cuts between cameras situated across a line between looking. If it’s a wide shot with a lot of characters on dif-
the two characters - the line of action. ferent depth-planes doing interesting things, your prediction
rate goes down.”
Screen continuity Characters who appear in both shots
must not appear to jump around too much. Current research in vision science attempts to predict the
focus of attention in an image, based on the computation of
Motion continuity Moving characters who appear in both
local image features. The most established theory is the
shots must appear to move in the same screen direc-
”saliency-based” model of Itti and Koch at Caltech [11].
tion.
Their model was used by Santella et al. for the purpose
This is the basis for another variant of the 180 de- of evaluating the composition while cropping and reframing
gree rule, which forbids cuts between cameras situated images [22]. Their conclusion was that better predictions
across a line along the actor’s trajectory - the line of were obtained by considering the eyes and gaze of people in
action in that case. the image.
Motion continuity also requires that the screen posi-
tion of the actor in the second shot should be ”ahead”,
rather than ”behind” 3.1.3 Rhythm.
Jump cut Characters who appear in both shots must not Rhythm refers to the tempo of the scene (how fast the film
appear to jump around too little. is cut). But we should be aware that the perceived duration
hal-00694444, version 1 - 4 May 2012

of a shot depends on its content. Thus a shot that we have

Small changes in screen coordinates are interpreted as already seen many times will seem to last longer than it
actor movements, rather than camera changes, as an really is. A close-up will also seem to last longer than it
effect of human perception. They should be avoided, really is. We should cut from any given shot only after the
or used systematically to obtain a stylistic effect (Go- audience has been able to fully see what we intend them to
dard). see. We should also cut before the shot becomes redundant
Look The gaze directions of characters seen in separation or boring.
should match. If they are looking at each other, their
images should also be looking at each other. If the One further complication is that the perceived length of a
two characters are NOT looking at each other, their shot depends on its size, its novelty and the intensity of the
images should NOT be looking at each other action. Thus, a close-up will be perceived as taking longer
than a long shot. A recurring shot will be perceived as
Distance The sum of apparent distances to two characters taking longer than a new shot. And a shot of a static scene
shown in separation should be at least twice the actual will be perceived as taking (much) longer than a shot of a
distance between them (as if the two images were taken fast action. A reasonable approximation may be to set the
from the same camera position). This prevents the use average shot length as a function of shot size, so that close-
of close-ups for two characters very far apart. ups are cut faster and long shots are cut slower. This is a
reasonable first approximation.
Size The shot size relative to a character should change
smoothly, rather that abruptly.
Another important factor is to choose a natural distribution
Cutting from a long shot directly to a close-up makes of shot durations. Automated editing should not ”get in the
it harder for the viewer to understand the relation be- way”. As a very simple illustrative example, cutting at reg-
tween the two shots. Instead, the editor should prefer ular intervals (as with a metronome) can be very annoying
to first cut to a medium-shot, then to a close-shot. because it distracts the viewer from the experience of the
movie. Cutting shots with randomized durations is usually
a better idea. Even better editing can be computed by fol-
It is important to realize that the rules can be checked by di- lowing the distribution of shot durations in real movies.
rect computations in screen space (through-the-lens). They
typically do not require knowledge of world coordinates. Film scholars Barry Salt [20] and David Bordwell [3] (among
others) have extensively studied shot durations in cinema
3.1.2 Eye-trace. and found it to be an important parameter of film style. An
Eye-trace refers to the expected trajectories of the eyes of empirical finding by Barry Salt is that the distribution of
the audience. Where on the screen is the audience looking shot durations in a movie sequence is correctly represented
in the first shot ? What happens there during the cut ? by a log-normal distribution. This is also the distribution of
Where will the audience look in the second shot ? sentence lengths in a book chapter. This is non-symmetric
distribution with a smaller probability for very short du-
A popular heuristic is to use the actors’ eyes in the image. rations and a relatively larger probability for longer shot
This is a well established principle confirmed by many film durations.
 What is important is to set the editing rhythm by choosing The emotional content of a shot or a transition between shots
an average shot length or ASL for the sequence, and cut has been little explored [21, 28] and is a promising avenue
according to a log-normal distribution. We can fine-tune for future research in cinematography.
the rhythm by also choosing the variance σ 2 of the shot
lengths. After having explained the theory of editing, we now turn
to actual implementations of working systems. We review
3.1.4 Story advancement. procedural and declarative approaches separately.
Story advancement can be measured by checking that all
changes in the story line are correctly presented in the image. 3.2 Procedural approaches
Thus, actors should only change places on-screen (not off- The Virtual Cinematographer by He et al. [10] relies on
screen). We should see (or hear) their reactions. We should the use of film idioms, which are recipes for obtaining good
see entrances and exits of all characters. We should see them framing and editing in a given situation. The general ap-
when they sit down or stand up, when they dress or undress, proach is similar to the old-fashioned AI principle of case-
when then they put on or take off their hats, etc. Of course, based reasoning - if a conversation starts in a game, use the
real directors and editors break this rule all the times, with conversation idiom; if a fight start, use the fight idiom; etc.
interesting effects. But it seems to be a safe bet to adopt
the rule that the best editing is the one that presents the Each idiom has two components - a set-up (blocking) of
entire action in the scene from the best angle at all times. the cameras relative to the actors; and a state machine
for switching automatically between cameras in that setup.
An even stronger principle was proposed by Hitchcock in an This is a powerful paradigm, that easily allows for gradually
interview with Truffaut [23]. ”Screen size and visibility of building up a complex cinematography system from simple
actors and objects should be proportional to their impor- building blocks.
tance in the plot at any given time (Hitchcock principle).
This is useful principle to keep in mind because it allows the Each idiom is very easy to program - the set-up of the cam-
programmer to define mathematically what makes a good eras is defined in terms of world coordinates - relative to
hal-00694444, version 1 - 4 May 2012

editing. Computing the screen size and visibility of actors the actors. The VC takes as input strings of simple sen-
and objects in a shot is the easy part. Computing their tences : SUBJECT+VERB+OBJECT representing the ac-
importance in the plot is the really difficult part. tion taking place in the scene. The VC also takes as input a
continuous stream of bounding boxes and orientation, rep-
In a scripted sequence, it seems reasonable to assume that resenting the relative geometric positions and orientations
the scripted actions are all equally important. Thus at any of the virtual actors, objects and scene elements.
given time, the importance of actors and objects can be
approximated as the number of actions in which they are Idioms are usually chosen based on the next action string.
taking part, divided by the total number of actions being More complex editing patterns can also be achieved by defin-
executed in the scene at that time. Other approximations ing hierarchical state machines, encoding the transitions be-
are of course possible. For instance, it may be preferable to tween idioms.
assign all the attention to a single action at all times. This
may be implemented with a ”winner takes all” strategy. While powerful, this scheme has yet to demonstrate that
it can be used in practical situations. One reason may be
3.1.5 Emotion. that there is a heavy burden on the application program-
Emotion is hardest to evaluate. There is a large body of mer, who must encode all idioms for all narrative situations.
research being done in neuroscience on emotion. They dis- Another reason may be that the resulting editing may be
tinguish between primitive emotions, such as surprise, fear, too predictable.
laughter, etc. whose action is very fast; primitive moods,
such as sadness or joy, whose action is much slower; and In a finite state machine, the switching of a camera is trig-
learned, cognitive affects such as love, guilt, shame, etc. gered by the next action string. This may have the unde-
sirable effect that the switching becomes too predictable. A
For the purpose of editing, evaluating the emotional impact good example is the ”dragnet” style of editing [16] where the
of any given shot or cut appears to be very difficult. Emo- camera consistently switches to a close-up of the speaker on
tional cues can be received from the screenplay or from the each speaker change; then back to a reaction shot of the
director’s notes. They assert which emotions should be con- other actors being spoken to. This can become especially
veyed at any given point in time. Given such emotional cues, annoying when the speakers alternate very quickly.
we can then apply simple recipes such as separating actors
or showing them closer together; changing editing rhythm While it is possible to use the dragnet style of editing as
to show increasing or decreasing tension; changing shot sizes a separate film idiom, this causes the number of idiom to
to show increasing or decreasing tension; using lower cam- explode since every configuration can be filmed in dragnet
era angles to show ceilings and feel oppression; using higher style. A better solution separates the camera set-ups from
camera angles to hide ceilings and feel freedom; using longer the state machines - for each set-up, different styles can then
lenses to slow down actor movements and isolate them from be encoded with different state machines. But the same
the background; using wider lenses to accelerate actor move- ”style” must still be separately re-encoded for each set-up.
ments and put them in perspective, etc. How simple should
those strategies be ? Too simple a solution may look foolish. It is not obvious how to ”generalize” film idioms. This is an
Too complicated solution may be out of reach. open problem for procedural approaches.
 3.3 Declarative approaches 3.4 Optimization approaches
In the beginning, automatic editing was attempted with tra- To overcome the problems of procedural and AI-based declar-
ditional, rule-based systems. ative approaches, it seems natural to rephrase the editing
problem as an optimization problem. In this section, we re-
IDIC by Sack and Davis [19] was one of the first systems to visit the editing constraints listed above and illustrate how
attempt automatic film editing from annotated movie shots. they can be used to build a quality function.
Mostly a sketch of what is possible, it was based on the
general problem solver (GPS), a very simple forward planner Let us review the common case of a dialog scene between two
[18]. actors. We are given a sequence of time intervals, each of
which may include actions performed by the two characters
”Declarative Camera Control for Automatic Cinematogra- A and B.
phy” is a much more elaborate attempt at formalizing the
editing of an animated movie, this time using modern plan- (a1 , b1 , t1 ), (a2 , b2 , t2 ), ...., (an , bn , tn )
ning techniques [4]. In that paper, idioms are not described
in terms of cameras in world coordinates but in terms of A solution to the automatic editing problem is a sequence
shots in screen coordinates, through the use of the DCCL of shots
language. DCCL is compiled into a film tree, which con-
tains all the possible editings of the input actions. Actions (c1 , s1 , t1 ), (c2 , s2 , t2 ), ...., (cn , sn , tn )
are represented as subject-verb-object triples. As in the Vir- where each shot si is taken by camera ci starting at time ti .
tual Cinematographer companion paper, the programming Cuts occur whenever the camera changes between succes-
effort for implementing an idiom is important. sives intervals. Reframing actions occur when the camera
remains the same but the shot descriptions change. Transi-
Jhala and Young have used text generation techniques to au- tions between the same shots result in longer shots and we
tomatically edit shots together using ”plan operators” [12]. can write the duration of the shot ∆i .
In another paper, Jhala and Young have used examples from
the movie ”The Rope” by Alfred Hitchcock to emphasize
hal-00694444, version 1 - 4 May 2012

Most optimization approaches compute the cost of a se-

stronger requirements on how the story line AND the di- quence from three types of preferences. They can use stylis-
rector’s goal should be represented to an automatic edit- tic preferences on shots; preference on which shot to use for
ing system [13]. They use Crossbow, a partial order causal each action; and preferences on how to cut from one shot
link planner, to solve for the best editing, according to a to the next one. This can be include stylistic preferences
variety of strategies, including maintaining tempo and de- as well. For example, we can prefer shots whose durations
picting emotion. They do not attempt to combine those are modeled by a log-normal distribution with average shot
strategies and instead prefer to demonstrate the capability length (ASL) m and standard deviation σ. The cost associ-
of their solver to present the same sequence in different edit- ated with a shot of length ∆t can then be expressed as
ing styles.
(log ∆ − log m)2
C(∆) = − log p(∆) = log ∆ +
Miyazaki et al. describe a complete film-making production 2σ 2
system [25, 26]. They model the scene graph in CLIPS/COOL
and define rules for choosing cameras and editing them. But Other style parameters are the desired ratios of long shots,
they are restricted to common idioms. medium shots and close shots; the relative importance of
the two characters, measured by their screen-time; and the
Kennedy and Mercer use the LOOM knowledge represen- relative importance between verbal and non-verbal action.
tation language to encode different communicative acts in
the rhetorical structure theory. By mapping the story-line As stated previously, an important goal in camera control
into communicative goals, stated in terms of themes and and editing is to show actions of the characters appropri-
moods, they are able to plan the choice of camera and edit- ately. In each segment, the editor reads the actions ai and bi
ing. They discuss ”inductive” and ”deductive” approaches of performed by the two actors and the corresponding tables for
characters, using Zettl as a reference [30]. shot preferences. Those tables can easily be pre-computed
for general action categoriessuch as speech actions, facial ex-
Friedman and Feldman present another knowledge-rich ap- pressions, hand gestures and pointing gestures. THis gives
proach for editing sitcoms [7]. a ranking of shot choices for each category.
AI-based approaches present an excellent overview of many The editor must also able to express preferences for the tran-
important aspects of automated film editing, but the results sitions between a shot si and a shot sj based on film gram-
are not always convincing for lack of a sufficient integration mar. Examples of preferences are - keep each character on
with advanced camera control techniques. Another draw- the same side of the screen across a cut. Avoid cutting be-
back of the AI-based approach is that it requires an in-depth tween two-shots. Avoid cutting from a long-shot to a close-
semantic analysis of the storyline, which is not always read- shot. Prefer cutting between shots of the same size (long,
ily available in practical applications, especially in real-time medium of close). Etc. A general method for building such
games. More importantly, those methods usually return a as table is by counting editing errors of all orders. Again,
(usually large) list of possible solutions, even in simple cases. this can be captured in an n × n table.
As a result, they usually do not scale very well with larger
vocabularies of plot actions, films idioms and shot categories. The problem is then of finding the sequence with the best
 ranking. With a Markov assumption, finding a sequence camera can pan, tilt or roll around the camera axis in all di-
of shot transitions that maximizes the quality function can rections. A ”sticky” camera is attached to an actor’s eye
be very efficiently implemented by dynamic programming. line. The camera can pan, tilt or roll ”around the actor’s
That approach is taken by Elson and Riedl [6]. Note that eye axis” which is much more powerful. In principle, it is
this precludes the use of shot durations. A semi-Markov possible to implement sticky cameras on other targets, and
or ”segment” model is needed for enforcing shot durations. multiple targets. This can be extended to compute dolly
That approach was introduced in Xtranormal’s ”magicam” paths as well. For two-shots, the camera can move along a
system [17]. Higher-order Markov models would be useful circle while keeping the two actors in any given screen po-
to implement repetition, which is a very powerful cinematic sition. Few machinima systems include support for editing.
technique [24], not easily taken into account with a Markov One may expect that this will change in the near future.
model. This requires the ability to record an entire 4-D scene (3-
D + time). Such replay functions are not usually offered
A promising approach that combines AI-based planning with in games. Dedicated applications such as Xtranormal State
optimization is hierarchical task network (HTN) planning have it. Support for editing also requires ”higher level” ac-
with preferences [29], which has been used in game AI to tion descriptions of some sort. For machinima in a game
solve military team planning. While we are not aware of it engine, such high-level descriptions can in principle be in-
being used for planning cinematography, it appears to be a ferred from the player’s or puppeteer’s actions. But player’s
likely candidate for future work in this area. intentions cannot easily be inferred from their movements.
Non-player characters (NPC) have a more formalized vo-
3.5 Applications cabulary of intentions and actions. This can be used to mo-
tivate the cinematography. Cinematography is intimately
3.5.1 Interactive storytelling related to game AI in that respect. The camera and ed-
Interactive storytelling promises to become a hybrid between itor ”agents” must infer the players intentions and actions
film and game, with a very strong need for fully automated in order to correctly react to them. The main difference
real-time cinematography and editing, so that all possible with other NPCs is that they pursue different goals. In
navigation paths through the ”story graph” generate movies essence, a camera is an NPC whose goal is to ”follow and
hal-00694444, version 1 - 4 May 2012

that are aesthetically pleasing. FACADE by Mateas and watch” other actors. In dedicated applications such as Xtra-
Stern is a good example, although with a very simple cine- normal, The Sims or MovieStorm, the actors’s movements
matic look [15]. are labeled with higher-level commands, including ”looking”,
”speaking”, ”pointing”, ”sitting” or ’standing”, etc. This is
3.5.2 Automated movie production sufficient in principle to motivate the cinematography and
Some systems go even beyond camera control and editing , editing. In addition, Movie Storm outputs a ”movie script”
towards fully automated movie production. In 1966, Alfred inferred from the choice of actions. On the other hand, text-
Hitchcock dreamed of ªa machine in which heŠd insert to-scene systems such as Xtranormal instead use the movie
the screenplay at one end and the film would emerge at the script as an input, and infer the sequence of actions to be
other end, complete and in colorÂŤ (Truffaut/Hitchcock, performed by the virtual actors from the script.
p. 330). In limited ways, this dream can be achieved by
combining the staging of virtual actors with the techniques
of camera control and editing described in this course. An 4. DISCUSSION AND OPEN ISSUES
example is the text-to-scene system by Xtranormal. This This final section discusses the problems related to the ac-
includes various declarative shots - one-shots and two-shots tual deployment of these techniques and directions for fu-
with a variety of camera angles and compositions. Cam- ture research, including augmenting the expressiveness of
era placement is automated for declarative shots. Editing is camera control and switching techniques by considering cog-
fully automated and makes use of both declarative shots (id- nitively well-founded perceptual and aesthetic properties of
ioms) and free cameras. This overcomes the traditional lim- the shots, including framing and lighting ; extending cam-
itations associated with a purely idiom-based system. Visi- era models to include the control of other important cine-
bility is taken into account through the use of ”stages”, i.e. matographic properties such as focus, depth-of-field (DOF)
empty spaces with unlimited visibility, similar to Elson and and stereoscopic 3-D depth (interaxial distance and conver-
Riedl. Both systems use a simple algebra of ”stages”, i.e. gence); and learning more general and varied camera control
intersections and unions of stages, allowing for very fast vis- and editing idioms directly from real movies using a variety
ibility computation against the static elements of the scene. of data mining and machine learning techniques.
Occlusion between actors is handled separately by taking
pictures through the eyes of the actors. The text-to-scene 4.1 Perception and aesthetics
system by Xtranormal is currently limited to short dialogue The state of the art in automatic framing (composition) and
scenes, although with a rich vocabulary of gestures, facial editing relies on a symbolic description of the view seen by
expressions and movements. But we can expect future im- the virtual camera. This is powerful, but important aspects
provements and extensions to other scene categories, includ- of the image are not well taken into account. A possible
ing action and mood scenes. avenue for future research lies in the possibility to perform
image analysis directly from the virtual camera, to recover
3.5.3 Machinima other important perceptual and/or aesthetics attributes of
Most machinima systems include some sort of camera con- the image. This is especially important for lighting [5].
trol. For instance, MovieStorm by Short Fuze includes ”through- Other image attributes, such as the contrast between fig-
the-lens” camera control with two types of cameras. A ”free” ure and background may be equally important [2].
 s Proceedings of AAAI ’96 (Portland, OR), pages
148–155, 1996.
4.2 Level of details [5] M. S. El-Nasr. A user-centric adaptive story
architecture: borrowing from acting theories. In ACE
One as yet unexplored area for future research is the relation
’04: Proceedings of the 2004 ACM SIGCHI
between cinematography and level-of-details modeling. Dur-
International Conference on Advances in computer
ing the early phases of pre-production and previz, a rough
entertainment technology, pages 109–116, 2004.
version of the scene with little details may be sufficient to
do the blocking of the actors and the cameras, and even [6] D. K. Elson and M. O. Riedl. A lightweight intelligent
to generate an early version of the editing (a rough cut). virtual cinematography system for machinima
The choices made in this stage result in a list of a few shots generation. In AI and Interactive Digital
which need to be rendered at full resolution. Thus, only Entertainment, 2007.
those parts of the scene that appear in the shot list really [7] D. Friedman and Y. A. Feldman. Automated
need to be modeled and rendered in full details. In practice, cinematic reasoning about camera behavior. Expert
it is not easy to implement this because the animation must Systems with Applications, 30(4):694–704, May 2006.
still appear realistic. Levels-of-details are still a problem for [8] F. Germeys and G. dı̈£¡Ydewalle. The psychology of
physics-based animation and AI. film: perceiving beyond the cut. Psychological
Research, 71(4):458–466, 2007.
[9] J.-L. Godard. Montage, mon beau souci. Les cahiers
4.3 Cinematic knowledge du cinéma, 11(65), décembre 1956.
Much of the current research remains limited to simple toy
[10] L. He, M. Cohen, and D. Salesin. The virtual
problems such as two-actor dialogues and fights. At this
cinematographer: a paradigm for automatic real-time
point, there has never been a convincing demonstration of a
camera control and directing. In SIGGRAPH ’96,
touching machine-generated love scene. Or a funny machine-
pages 217–224, 1996.
generated comic scene. Or a frightening machine-generated
[11] L. Itti, C. Koch, and E. Niebur. A model of
horror scene.
saliency-based visual attention for rapid scene
hal-00694444, version 1 - 4 May 2012

analysis. IEEE Transactions on Pattern Analysis and

This is the main challenge for this field. In the future, we
Machine Intelligence, 20(11):1254–1259, 1998.
expect that strategies will be taken to better reproduce the
specific cinematography and editing styles needed for soc- [12] A. Jhala and R. M. Young. A discourse planning
cer and car-racing replays; machinima remakes of film noir, approach to cinematic camera control for narratives in
drama, soap and sitcom. Cinematography and editing styles virtual environments. In AAAI’05: Proceedings of the
can be hand-crafted through the use of idioms in a proce- 20th national conference on Artificial intelligence,
dural system; or by choosing preferences in an optimization pages 307–312. AAAI Press, 2005.
approach. One promising avenue of research for imitating [13] A. Jhala and R. M. Young. Representational
movie styles is through the use of video data-mining, i.e. requirements for a plan based approach to automated
searching large annotated databases of movie examples [27]. camera control. In AIIDE, pages 36–41, 2006.
[14] J. Mascelli. The Five C’s of Cinematography: Motion
Picture Filming Techniques. Cine/Grafic Publications,
4.4 Evaluation Hollywood, 1965.
Evaluation of automatic camera control and editing has been [15] M. Mateas and A. Stern. Facade: An experiment in
attempted by only a few researchers. The result seems to be building a fully-realized interactive drama. In Game
that it is relatively easy to emulate an ”amateur” cameraman Developers Conference, 2003.
or film editor, but very hard to emulate even a ”modest” pro-
[16] W. Murch. In the blink of an eye. 1986.
fessional. In other words, empirical evaluations show that
[17] R. Ronfard. Automated cinematographic editing tool.
a professionally cinematographer and edited scene is always
Technical report, Xtranormal Technologies, May 7,
preferred to a machine-generated scene. But a machine-
2009.
generated scene can be preferred (or found comparable) to
an amateur-generated scene. Another possible evaluation [18] S. Russell and P. Norvig. Artificial Intelligence: A
criteria is ease of use. For example, it would be useful to Modern Approach. Prentice Hall, 2002.
compare the time needed for generating a satisfactory movie [19] W. Sack and M. Davis. Idic: assembling video
scene with different camera control and editing systems. sequences from story plans and contentannotations. In
Multimedia Computing and Systems, pages 30 – 36,
1994.
5. REFERENCES [20] B. Salt. Film Style and Technology: History and
[1] G. Bloch. Eléments d’une machine de montage pour Analysis (2nd edition). Starword, 2003.
l’audio-visuel. PhD thesis, T’elécom Paris, 1986. [21] A. Salway and M. Graham. Extracting information
[2] B. Block. The Visual Story: Seeing the Structure of about emotions in films. In ACM Conference on
Film, Tv, and New Media. Focal Press, 2001. Multimedia, pages 299–302, 2003.
[3] D. Bordwell. The Way Hollywood Tells It: Story and [22] A. Santella, M. Agrawala, D. DeCarlo, D. Salesin, and
Style in Modern Movies. University of California M. Cohen. Gaze-based interaction for semi-automatic
Press, 2006. photo cropping. In CHI ’06: Proceedings of the
[4] D. B. Christianson, S. E. Anderson, L.-W. He, D. S. SIGCHI conference on Human Factors in computing
Weld, M. F. Cohen, and D. H. Salesin. Declarative systems, pages 771–780, New York, NY, USA, 2006.
camera control for automatic cinematography. In
 ACM.
[23] H. G. Scott and F. Truffaut. Hitchcock-Truffaut
(Revised Edition). Simon and Schuster, 1985.
[24] S. Sharff. The Elements of Cinema: Toward a Theory
of Cinesthetic Impact. Columbia Press, 1982.
[25] J. Shen, S. Miyazaki, T. Aoki, and H. Yasuda.
Intelligent digital filmmaker dmp. In Computational
Intelligence and Multimedia Applications, pages 272 –
277, 2003.
[26] J. Shen, S. Miyazaki, T. Aoki, and H. Yasuda.
Representing digital filmmaking techniques for
practical application. In Information and Knowledge
Sharing, 2003.
[27] M. K. Shirahama and K. Uehara. Video data mining:
Extracting cinematic rules from movie. In Int’l
Workshop Multimedia Data Management
(MDM-KDD), 2003.
[28] T. J. Smith. An Attentional Theory of Continuity
Editing. PhD thesis, University of Edinburgh, 2005.
[29] S. Sohrabi, J. A. Baier, and S. A. McIlraith. Htn
planning with preferences. In IJCAI’09: Proceedings
of the 21st international jont conference on Artifical
intelligence, pages 1790–1797, San Francisco, CA,
USA, 2009. Morgan Kaufmann Publishers Inc.
[30] H. Zettl. Sight, Sound, Motion: Applied Media
hal-00694444, version 1 - 4 May 2012

Aesthetics. Wadsworth Publishing Company, USA,

1999.