Games: Sound and music for

Interactivity and Immersion

Anja Volk

Sound and Music Technology,

Nov 21, 2017

1
 Today

g Main modules
A. Sound and music for games
• Different functions of sound and music in games (recap)
• Challenges of interactivity and immersion
• Music and emotion

2
 Recapitulation last lecture

g Functions of sound and music in film and games

g Annabel Cohen (1999) for film: e.g.

• Provides continuity between shots
• Induces mood
• …

g additional function for games

• support interactive aspects of the game play

3
 Recapitulation last lecture

g Dual nature of games:

g Virtual world
• Sound and music: support reality status of the game world (such
as supporting sense of presence)
g User system
• Sound and music: important part of usability system, needs to be
informational and communicative

g Challenge for music and sound design for games:

combination of the two functions

4
 Game audio typologies

g Recapitulation film music: diegetic vs. ?

g What was it about?

5
 Game audio typologies

g Recapitulation film music: diegetic vs. non- (extra-) diegetic

g Diegetic sound: originates from a source existing within the film
or game world
g Extradiegetic: no connection to an actual source within the film
or game world

g Application to games (Jørgensen, 2009): diegetic vs. ?

g difference to extradiegetic in film?

6
 Game audio typologies

g Recapitulation film music: diegetic vs. non- (extra-) diegetic

g Diegetic sound: originates from a source existing with the film
or game world
g Extradiegetic: no connection to an actual source within the film
or game world

g Application to games (Jørgensen, 2009): diegetic vs.

transdiegetic
g Extradiegetic: can provide game player with information that
changes players’ actions, which in turn can change the game
world (therefore “transdiegetic”)

7
 Recapitulation: IEZA model

g Huiberts & van Tol (2008)

g Conceptual design tool for game andGMG!%:3!&Q_#!;>23=!<>4!0627>!71!@0;39!
audio design students at
HKU
g Applies to game audio during the interactive game play

8 !
 GHH[m!">==719E!GHH\.DM!%:3!2/10;7971@!4>=3!A01!:0F3!0!J>9787F3!71<=631A3!>1!7;;3497>1!
09!78!79!07;32!08!;0S71@!8:3!@0;3!;>43!987;6=0871@M!!
IEZA model: two functionalities
W:31! 8:393! 8X>! J349J3A87F39! 043! 43=0832! 8>! &Q_#E! 78! 9:>X9! 8:08E! 6960==/E! 8:3! &1834<0A3!
2>;071! 79! ;071=/! >4731832! 8>! 8:3! >J87;79087>1! ><! 8:3! @0;3! <>==>X32! ./! Q<<3A8! 2>;071!
g two main perspectives on the expression of in-game audio aimed
012!=098=/!8:3!2>;0719!8:08!<>4;!8:3!*38871@M!&1!>8:34!X>429E!78!79!@31340==/!;>43!27<<7A6=8!
at the player's experience:
8>!A>;;617A083!<0A860=!71<>4;087>1!X78:!8:3!*38871@!9723M!%:3!2/10;79087>1!><!8:3!@0;3!
g Audio used for optimizing game play: helping the player to play the
79!J471A7J0==/!2>13!./!8:3!#<<3A8E!<>==>X32!./!8:3!_>13M!%:3!#A87F78/!;>98=/!A>1847.6839!
game by providing necessary gameplay information; serves usability
=399!0A87F3=/!8>!8:79!<61A87>10=78/M!&1!<7@643!YE!8:393!8X>!J349J3A87F39!043!43J43931832!71!
g Audio used for dynamizing game play: making the gameplay
experience more intense and thrilling
8:3!&Q_#!;>23=M!!

! !
9
(69;G=$ UH$ !I=$ 5R:$ 4=G>4=@567=>$ :8$ 5I=$ D;8@56:83C65J$ :D$ 93K=$ 3;<6:V$ :456K6>356:8$ 38<$
 Today

g Main modules
A. Sound and music for games
• Different functions of sound and music in games
• Challenges of interactivity and immersion
• Music and emotion

10
 Important historic milestones
(according to Jørgensen, 2009)

g First Nintendo console

g storage: primary concern for computer games
g sound: only small amount of storage space
g only a few sounds could be played simultaneously, usability
functions became prioritized
g Nintendo’s classic Super Mario Bros. games: use of sound for
response and urgency purposes

g Stereo introduced in 1980s

g new functionalities: sound and music with high precision
compared to real world sounds

g 1990s: sound cards could produce CD quality sound

g auditory focus on creating believable virtual environments
g new uses of game audio beside the usability function,
connected to orientation and the sense of presence.

11
 Important historic milestones
(according to Jørgensen, 2009)

g Towards modern times:

g inclusion of CD-ROM allowed heavier applications with
more sound and graphics
g functionality, fidelity, sense of presence, atmosphere still
important, but came to merge with the use of sound for
usability purposes
g E.g. games utilized stereo speakers to create the sense
of three-dimensionality in gamy by making the sound
move from one speaker to the other
g E.g. true surround systems which allowed the
positioning of different channel speakers around the
player in real space

12
 An important drawback

musical example compare compar structur convert convert

content image e text e to to below
above
Digital level 1: speech none - hard
audio primitive
(MP3, features
Wav)
Time- level 2: text little easy fairly
stamped objects hard (OK
events job)
(MIDI)
Music level 2: text + much easy -
notation compoun markup (OK job)
(Finale, d objects
Sibelius,
MusicXML
)

13
 An important drawback

g Greater level of interactivity requires more flexibility of

music; a more granular (note) level control over the music
necessary

g Composers work with MIDI sequencers and a sampling

engine; by rendering out to a wav file, we are essentially
creating the “one-track-orchestra” (every possible
variation requires to render out a separate file).
Stevens and Reybold (2013): Game audio tutorial

14
 Challenges for Interactivity

g In a surprising discussion, Sony's Jason Page admitted that the

future of video game music may be based in technology that's
considered antiquated by most: MIDI. Many developers have
been moving away from MIDI to pre-recorded, orchestrated
background music, but in the process, the interactivity of game
music has been sacrificed.

g The incredible processing power of the next-generation consoles

has changed the rules of MIDI … MIDI samples used by
consoles can be just as good, if not better, than the samples
used on dedicated synthesizers. Because MIDI loads in real-
time, it retains the interactivity that composers like Koji Kondo
would need, and it would allow games to load more quickly.

http://www.engadget.com/2007/03/08/midi-is-the-future-of-game-audio/#comments

15
 Challenges for Interactivity

g adaptive music (Brandon 2004, Whitmore 2003)

g technique to make music change according to the
player’s behavior and actions in game.

16
 Forms of Interactivity
(Michael Liebe, 2013)

g Linear music
g firmly coupled to certain game-immanent elements
g cannot be influenced by players
g Examples: soundtracks
g Reactive music
g music is directly connected to actions of the players
g triggered by specific actions (e.g. location change, begin a
fight …)
g Often in role-playing
g Example of entirely driven by player: Elektroplankton
https://www.youtube.com/watch?v=ttFoK8BTXM4

g Proactive music
g prompts the players to undertake a specific action when it is
played
g Dance Dance Revolution: players follow rhythmic patterns
g https://www.youtube.com/watch?v=NqD-B8t_6us#t=100s

17
 How to achieve interactive music?

g Michael Sweet (Composer): Typical techniques

g Method 1: Vertical Remixing (Layering)

g Method 2: Horizontal Re-sequencing

based on pre-composed stems of music to create interactivity

18
 How to achieve interactive music?

g Michael Sweet (Composer): Typical techniques

g Method 1: Vertical Remixing (Layering)

g Method 2: Horizontal Re-sequencing

19
 How to achieve interactive music?

g Michael Sweet (Composer): Typical techniques

g Method 1: Vertical Remixing (Layering)

g Method 2: Horizontal Re-sequencing

Height (Pitch

Time

20
 How to achieve interactive music?

g Michael Sweet (Composer): Typical techniques

g Method 1: Vertical Remixing (Layering)

g Method 2: Horizontal Re-sequencing
g Cross-fading
g Phrase-branching
g Musical demarcation branching
g Bridge transition
g Stinger-based sequencing

21
 How to achieve interactive music?

Michael Sweet, Composer

g Method 1: Vertical Remixing (Layering)

g break a music cue into two or more musical layers (e.g.

instruments)

g control-inputs: which game events trigger the layers to enter

and exit?

g the more layers, the greater the diversity

g often just two layers used

g length, tempo, and harmonic framework the same

g Example: Fall out New Vegas

https://www.youtube.com/watch?v=JR38Yn9qxkQ

22
 How to achieve interactive music?

Michael Sweet, Composer

g Method 1: Vertical Remixing (Layering)

g Advantages:
g Immediate changes to music based on a game event
g Less impactful than switching to an entirely new music cue,
(change is more subtle)
g Disadvantages:
g Musical phrases easily interrupted (e.g. melody is faded in or
out in the middle of a phrase)
g fading in or out layers can sound non-musical
g No change in tempo or harmonic structure possible

23
 How to achieve interactive music?

Michael Sweet, Composer

g Method 1: Vertical Remixing (Layering)

g Advantages:
g Immediate changes to music based on a game event
g Less impactful than switching to an entirely new music cue,
(change is more subtle)
g Disadvantages:
g Musical phrases easily interrupted (e.g. melody is faded in or
out in the middle of a phrase)
g fading in or out layers can sound non-musical
g No change in tempo or harmonic structure possible
g When used:
g when state changes from one state to another shorter than 30
seconds
g puzzles within game: Completion of each phase of the puzzle
might bring in a new layer of the music indicating to the
player that he’s progressing

24
 How to achieve interactive music?

Michael Sweet, Composer

g Method 2: Horizontal Re-sequencing
g Cross-fading
• one music cue fades out while another music cue fades up
• Example: World of Warcraft https://www.youtube.com/watch?v=9-
JKJMnH8wM
• Advantages
– Easy to compose and implement into a game.
– Immediate changes to the music based on a game event.
– Ability to completely change the tempo, harmony, instrumentation, or
melody instantly based on a game event
• Disadvantages
– The least musical of all adaptive techniques (changes abrupt)
– Musical phrases are often interrupted in the middle of a phrase.
– No accounting for the tempo or key changes when switching from one
musical cue to another.
• When used
– Very often (is simple)
– bad method to use if the cues switch more often than 30 seconds
because of the constant interruption to the player

25
 How to achieve interactive music?
Michael Sweet, Composer

g Method 2: Horizontal Re-sequencing

g Phrase-branching
• waits for the current musical phrase to end before playing the next
musical cue; https://www.youtube.com/watch?v=C58TuhQPHNc
g Advantages
• Most musical of all the horizontal re-sequencing techniques, never
interrupts a musical phrase.
• Ability to change tempo, harmony, instrumentation or melody in
the next phrase based on a game event.
g Disadvantages
• Non-immediate musical change because the music change will wait
until the end of the current phrase which is dependent on the
length of the phrases.
• Can be more disruptive to the player in terms of musical changes
than vertical remixing.
g When used
• suited for styles/genres whith shorter phrase lengths such as rock
and techno; longer phrase lengths will delay the entrance of the
next music cue.

26
 How to achieve interactive music?

Michael Sweet, Composer

g Method 2: Horizontal Re-sequencing
g Musical demarcation branching
• Allows music cue to switch at a musical demarcation point such as
a beat, or measure

27
 How to achieve interactive music?

Michael Sweet, Composer

g Method 2: Horizontal Re-sequencing
g Musical demarcation branching
• Allows music cue to switch at a musical demarcation point such as
a beat, or measure
• Advantages
– More musical than cross-fading.
– Faster changes than phrase branching.
– Ability to change tempo, harmony, instrumentation or melody in the
musical demarcation point based on a game event.
• Disadvantages
– Non-immediate musical change because the music change will wait until
the next demarcation point.
– Musical phrases can be interrupted.
– Can be more disruptive to the player in terms of musical changes than
vertical remixing.
g Example: https://www.youtube.com/watch?v=oMVedBq6H24

28
 How to achieve interactive music?

Michael Sweet, Composer

g Method 2: Horizontal Re-sequencing
g Bridge transition
• short musical cues used to connect one musical cue with another
for more seamless transitions

Musical Bridge:
http://www.youtube.com/watch?v=hE_qOY5GkH0&t=1m22s

29
 How to achieve interactive music?

Michael Sweet, Composer

g Method 2: Horizontal Re-sequencing
g Bridge transition
• short musical cues used to connect one musical cue with another
for more seamless transitions
• Advantages
– Ability to link to disparate music cues in terms of tempo, harmony, and
instrumentation.
– Ability to change tempo, harmony, instrumentation or melody based on
a game event.
• Disadvantages
– Musical phrases can be interrupted.
– The length of the bridge transition pushes the beginning of the next cue
later making it more difficult to do another change until after the next
cue begins.
– Can be more disruptive to the player in terms of musical changes than
vertical remixing.
– If the same bridge transition is heard frequently, it can be repetitive to
the player.
g Example: https://www.youtube.com/watch?v=jZqaEZjxqLo

30
 How to achieve interactive music?

Michael Sweet, Composer

g Method 2: Horizontal Re-sequencing
g Stinger-based sequencing
• a series of stingers which are played back based on game events
• player triggers these stingers individually based on game events
• Stingers may overlap
• do not have a connecting rhythmic framework, are composed
primarily of crescendos and accents with silence in-between
• Advantages
– Stingers are usually separated by silence, so they tend to work well
musically together.
– Immediate punctuation of a game event.
– Ability to link to disparate music cues in terms of tempo, harmony, and
instrumentation.
• Disadvantages
– No tempo map or rhythmic framework linking the music together, e.g.
can feel like disparate elements.
– Can feel close to the film scoring cliche ‘Mickey-Mousing’.
– Phrase lengths are heavily dependent on how the game is dramatically
scripted. Example:
g Example: https://www.youtube.com/watch?v=djI3FFM6oVM

31
 How to achieve interactive music?

Michael Sweet, Composer

g Writing interactive music

g http://interactivemusicbook.com/wp/

Other example

Stevens & Reybold (2013)

The Game Audio Tutorial: A Practical Guide to Sound and Music for
Interactive Games.

32
 Another example for interactivity

g adaptive music (Brandon 2004, Whitmore 2003)

g technique to make music change according to the player’s
behavior and actions in game.

g One technique to achieve that: leitmotifs

33
 Leitmotifs

g Music is extremely repetitive

g Famous example

http://philomel.com/asa156th/mp3/Sound_Demo_1.mp3

http://philomel.com/asa156th/mp3/Sound_Demo_2.mp3

34
 Leitmotifs

g Music is extremely repetitive

g Famous example: Diana Deutsch’s “Speech to Song Elusion”;

for more information check out:

http://deutsch.ucsd.edu/psychology/pages.php?i=212

35
 Leitmotifs

g Music is extremely repetitive

g Convenient for automatic composition, example

http://tonematrix.audiotool.com

g Used for loops in games, but no interactivity

36
 Leitmotifs

g Music makes use of non-literal repetition, called variation

http://www.youtube.com/watch?v=SKMVLKmgxAw

http://www.youtube.com/watch?v=QYrZZ68zhSs

37
 Leitmotifs

g Leitmotif:

g attached to a person, object, place, idea, state of mind,

supernatural force or any other ingredient in a dramatic work

g a theme, or other coherent musical idea, clearly defined so as

to retain its identity if modified on subsequent appearances

Grove Music Online

38
 Leitmotifs

39
 Leitmotifs

The inventor: Richard Wagner

40
 Leitmotifs

http://ringcycle.metoperafamily.org/leitmotifs

41
 Leitmotifs

g widely used in film music, e.g. LOTR

g Leitmotif of “The Fellowship” in different variations:

https://www.youtube.com/watch=qxc0Z6zZcto&list=PLHmmG49wjXiCFYH5lmCtmBHDwB6REejl9&index=
3

42
 Leitmotifs

g Example Game: Final Fantasy 7

http://videogamemusicnerd.blogspot.nl/2012/11/analysis-final-fantasy-vii-
leitmotifs.html

43
 Leitmotifs

g Leitmotif challenge: how to do this automatically such that

music adapts to player’s actions?
g Experimentation project
g Automatic leitmotif generation
g Challenging: not every motif is suitable for each algorithm
g Challenging: aesthetic value

45
 Today

g Main modules
A. Sound and music for games
• Different functions of sound and music in games (recap)
• Challenges of interactivity and immersion
• Music and Emotion

46
 Immersion

g no generally accepted definition of term immersion,

instead different descriptions in literature, e.g.

g the experience of losing a sense of embodiment in the

present whilst concentrating on a mediated environment
g losing track of immediate physical surroundings
g being transported into the game world
g being absorbed in the activity
g being identified with the situation or a character of the game

Huiberts (2010), p. 36

47
 Immersion:
some systematic aspects by Sander Huiberts

g Sander Huiberts, PhD

g Three basic aspects of game immersion

g 1- Being transported into the game world
g 2- Absorption in the activity
g 3- Identification with the situation or a character of the game

48
 Immersion:
some systematic aspects by Sander Huiberts

g Sander Huiberts, PhD

g Three basic aspects of game immersion

g 1- Being transported into the game world
• e.g. sensory features stimulate the feeling of being there

g 2- Absorption in the activity

• e.g. challenge-based immersion

g 3- Identification with the situation or a character of the game

• e.g. imaginative immersion
• e.g. emotional responses: increasing the empathy through specific
mood

49
 Immersion:
some systematic aspects by Sander Huiberts

g Sander Huiberts, PhD

g Three basic aspects of game immersion

g 1- Being transported into the game world
• e.g. sensory features stimulate the feeling of being there

g 2- Absorption in the activity

• e.g. challenge-based immersion

g 3- Identification with the situation or a character of the game

• e.g. imaginative immersion
• e.g. emotional responses: increasing the empathy through specific
mood

50
 Strong experiences related to music

g Empirical study by Gabrielsson & Wik (2003)

g 900 people reporting on strong experiences of music

Strong experiences related to music: A descriptive system

ALF GABRlELSSON AND SlY LINDSTROM WIK

But it was not until the second half of the performance mat me mystery and the
power (4.7) really gripped me (4.3). I was filled by an enormous warmth and heat (2.1).
I really swallowed all me notes (2.3) mat were streaming out in me air, nor a single note,
effect or sequence missed my hungry ears (3.7). The music became so distinct (3.7). I was
captivated by each of me instruments and what they had to offer me (3.7). Nothing else
existed (4.1)!
I was dancing, whirling (2.2) and really gave myself up to the music and the
Someone listening to a band playing Finnish tango in a pub
rhythms (4.4), overjoyed (5.2) -laughing (2.2). Tears came into my eyes (2.1) -
however suange that may seem-and it was as a further sign, some kind of liberation. The
music set me free from my sober everyday life (7.1). Now I could let my body parts
dance as freely as they wanted (2.2) - just let them follow me rhythms and totally lose
control (4.3).

51
The music danced around like a whirlwind (4.7) in me narrow room and all the
dramatic sentiment mat it conveyed (4.7) reflected my own situation in life (7.3) - but
 the music and the effects from every side, frombehind, to the right, to the left (3.1).
They played the good old pieces mixed with new ones (3.8). I recognised them (4.5) and
Strong experiences related to music
could stand there rocking and joining in singing (2.2). I felt one with me music (4.4). I
shivered (2.1).
When they began playing again after the intermission darkness had fallen over me arena
and they made great use of light effects, laser beams that 'went out into me universe' (3.2).
Yes, it actually felt so, everything became so infinite and I was one with it (6.2).
This is me first time mat I am so involved in me music (4.4). I feel how me base comes
in from me ground via me sales of my feet, continues up through my calves, thighs, me

spine (3.3) and I am filled by the music (2.3). Boundaries are wiped out. I am one with
'67
rhe universe (6.2). The music dissolves all boundaries, just like what I have understood it
means to be psychotic (4.5, 6.2). When they play my favourite tune "Learning to fly" tears
begin trickling (2.1). I have always thought that it is ridiculous when girls howled at the
Beatles (4.5). Now I am in the same situation myself, albeit not so hysterical- No, I am
standing there very calm (5.2), rocking in time with the music (2.2), feeling whole (4.2,
7.1) and just letting tears trickle down my cheeks (2.1).
The concert approaches the end, and I feel dislike (5.3) as I always do when everything
is well. I want to keep it (7.1). TheSomeone
end became the climax
listening when
to Pink theyinleta off
Floyd one of the
concert
biggest fireworks that I have ever seen (3.2). Like a child on Christmas Eve I stood there
with open mouth, laughed and clapped my hands (2.2). What an experience (Ll)!

Repor: C One piece of music which has given me a suong experience is the Adagimo in
Gustav Mahler's Fifth Symphony. The day this experience occurred I was alone at home,
52 after school in the afternoon. I was gloomy and broody (5.3) as I usually am. Mahler's
 Strong experiences related to music

g Empirical study by Gabrielsson & Wik (2003)

g 900 people reporting on strong experiences of music

g complete absorption
g one did not think about where one was or for how long this
would last
g I was totally caught in the experience
g everything around me did not exist
g time and space ceased to exist
g I dreamed myself away

53
 How can we create music for games…

g … that induces these strong experiences and contributes to

immersive experience of games?

54
 Challenge

g How is music inducing these experiences and emotions?

g What do we know from the scientific perspective?

• Juslin & Vjästfäll (2008); Juslin, Harmat, Eerola (2013)

55
 How is emotion induced?
g (1) brain stem reflex

g (2) rhythmic entrainment

g (3) evaluative conditioning

g (4) contagion
BRECVEMA
g (5) visual imagery

g (6) episodic memory

g (7) musical expectancy

g (8) aesthetic judgment

56
 BRECVEMA

g (1) brain stem reflex

g hard-wired attention response
g acoustic features (e.g. extrem loudness or speed)
g Prepares body to react
g quick and automatic

57
 BRECVEMA

g (2) rhythmic entrainment

g Adjustment of internal body rhythm to external rhythm in
music

58
 BRECVEMA

g (3) evaluative conditioning

g Certain music associated with certain events
g Memory plays important role
g conditioned association

59
 BRECVEMA

g (4) Contagion
g listener perceives the emotional expression of the music, and
then “mimics” this expression internally
g Induction of the same emotion

60
 BRECVEMA

g (6) Episodic memory

g an emotion is induced in a listener because the music evokes
a memory of a particular event in the listener’s life
g “darling, they are playing our tune”

61
 BRECVEMA

g (7) Musical expectancy

g emotion is induced because a specific feature of the music
violates, delays, or confirms the listener’s expectations about
the continuation of the music

62
 BRECVEMA

g (8) Aesthetic judgment

g subjective evaluation of the aesthetic value of the music
based on an individual set of weighted criteria

63
 Examples in game music
g (1) brain stem reflex

g (2) rhythmic entrainment

g (3) evaluative conditioning

g (4) contagion
BRECVEMA
g (5) visual imagery

g (6) episodic memory

g (7) musical expectancy

g (8) aesthetic judgment

64
 Summary of important points

g Challenges for interactivity and immersion

g How to achieve interactive music?

g Leitmotifs

g Emotion and immersion

g BRECVEMA-model on emotion

73
 Literature

g Gabrielsson & Wik (2003), Strong experiences related to

music, Musicae Scientiae, Vol. 7, No. 2.

g Juslin, P. N., Harmat, L. & Eerola, T. (2013). What Makes

Music Emotionally Significant? Exploring the Underlying
Mechanisms. Psychology of Music 42(4): 599-623.

g New Grove Online: Leitmotif

g Michael Sweet: Writing interactive music

g http://interactivemusicbook.com/wp/

g Michael Liebe (2013): Interactivity and Music in Computer

Games. In: Music and Game, (ed. P. Mohrmann), Springer.

74