Generating Music from Literature

Hannah Davis Saif M. Mohammad

New York University National Research Council Canada
[email protected] [email protected]

Abstract • Mapping pieces of literature to musical

pieces according to compatibility of the flow
We present a system, TransProse, that au- of emotions in text with the audio character-
arXiv:1403.2124v1 [cs.CL] 10 Mar 2014

tomatically generates musical pieces from istics of the musical piece.

text. TransProse uses known relations be- • Finding songs that capture the emotions in
tween elements of music such as tempo different parts of a novel. This could be use-
and scale, and the emotions they evoke. ful, for example, to allow an app to find and
Further, it uses a novel mechanism to de- play songs that are compatible with the mood
termine sequences of notes that capture the of the chapter being read.
emotional activity in text. The work has
• Generating music for movie scripts.
applications in information visualization,
• Appropriate music can add to good visualiza-
in creating audio-visual e-books, and in
tions to communicate information effectively,
developing music apps.
quickly, and artfully.
1 Introduction Example 1: A tweet stream that is accom-
panied by music that captures the aggregated
Music and literature have an intertwined past. sentiment towards an entity.
It is believed that they originated together Example 2: Displaying the world map where
(Brown, 1970), but in time, the two have devel- clicking on a particular region plays music
oped into separate art forms that continue to influ- that captures the emotions of the tweets ema-
ence each other.1 Music, just as prose, drama, and nating from there.
poetry, is often used to tell stories.2 Opera and bal-
let tell stories through music and words, but even Given a novel (in an electronically readable
instrumental music, which is devoid of words, form), our system, which we call TransProse, gen-
can have a powerful narrative form (Hatten, 1991). erates simple piano pieces whose notes are depen-
Mahler’s and Beethoven’s symphonies, for exam- dent on the emotion words in the text. The chal-
ple, are regarded as particularly good examples of lenge in composing new music, just as in creat-
narrative and evocative music (Micznik, 2001). ing a new story, is the infinite number of choices
In this paper, for the first time, we present a and possibilities. We present a number of map-
method to automatically generate music from liter- ping rules to determine various elements of music,
ature. Specifically, we focus on novels and gener- such as tempo, major/minor key, etc. according to
ate music that captures the change in the distribu- the emotion word density in the text. We intro-
tion of emotion words. We list below some of the duce a novel method to determine the sequence of
benefits in pursuing this general line of research: notes (sequences of pitch and duration pairs) to be
played as per the change in emotion word density
• Creating audio-visual e-books that generate
in the text. We also list some guidelines to make
music when certain pages are opened—music
the sequence of notes sound like music as opposed
that accentuates the mood conveyed by the
to a cacophonous cascade of sounds.
text in those pages.
Certainly, there is no one right way of capturing
1
The term music comes from muses—the nine Greek god- the emotions in text through music, and there is no
desses of inspiration for literature, science, and arts.
2
Music is especially close to poetry as songs often tend to one right way to produce good music. Generating
be poems set to music. compelling music is an art, and TransProse can be
improved in a number of ways (we list several ad- melody (for example, in the song Mary Had A
vancements in the Future Work section). Our goal Little Lamb) or multiple melodies; they can last
with this project is to present initial ideas in an throughout the piece or appear in specific sections.
area that has not been explored before. A challenge for TransProse is to generate appro-
This paper does not assume any prior knowl- priate sequences of notes, given the infinite possi-
edge of music theory. Section 2 presents all the bilities of pitch, duration, and order of the notes.
terminology and ideas from music theory needed Tempo is the speed at which the piece should
to understand this paper. We present related work be played. It is usually indicated by the num-
in Section 3. Sections 4, 5, 6, and 7 describe our ber of beats per minute. A beat is a basic unit
system. In Sections 8 and 9, we present an analysis of time. A quarter-note is often used as one
of the music generated by our system for various beat. In which case, the tempo can be under-
popular novels. Finally in Section 10 we present stood simply as the number of quarter-notes per
limitations and future work. minute. Consider an example. Let’s assume it
is decided that the example melody specified in
2 Music
the earlier paragraph is to be played at a tempo of
In physical terms, music is a series of possibly 120 quarter-notes per minute. The total number of
overlapping sounds, often intended to be pleasing quarter-notes in the initial sequence (620Hz–half-
to the listener. Sound is what our ears perceive note, 1200Hz–whole-note, 840Hz–half-note, and
when there is a mechanical oscillation of pressure 660Hz–quarter-note) is 2 + 4 + 2 + 1 = 9. Thus the
in some medium such as air or water. Thus, differ- initial sequence must be played in 9/120 minutes,
ent sounds are associated with different frequen- or 4.5 seconds.
cies. In music, a particular frequency is referred The time signature of a piece indicates two
to as pitch. A note has two aspects: pitch and things: a) how many beats are in a measure, and
relative duration.3 Examples of relative duration b) which note duration represents one beat. It is
are whole-note, half-note, quarter-note, etc. Each written as one number stacked on another num-
successive element in this list is of half the dura- ber. The upper number is the number of beats per
tion as the preceding element. Consider the exam- measure, and the lower number is the note dura-
ple notes: 400Hz–quarter-note and 760Hz–whole- tion that represents one beat. For example, a time
note. The first note is the sound corresponding signature of 68 would mean there are six beats per
to 400Hz, whereas the second note is the sound measure, and an eighth note represents one beat.
corresponding to 760Hz. Also, the first note is to One of the most common time signatures is 44 , and
be played for one fourth the duration the second it is referred to as common time.
note is played. It is worth repeating at this point
Sounds associated with frequencies that are
that note and whole-note do not refer to the same
multiples or factors of one another (for exam-
concept—the former is a combination of pitch and
ple, 440Hz, 880Hz, 1760Hz, etc) are perceived
relative duration, whereas whole-note (and oth-
by the human ear as being consonant and pleas-
ers such as quarter-note and half-note) are used to
ing. This is because the pressure waves associ-
specify the relative duration of a note. Notes are
ated with these sounds have overlapping peaks and
defined in terms of relative duration to allow for
troughs. Sets of such frequencies or pitches form
the same melody to be played quickly or slowly.
pitch classes. The intervals between successive
A series of notes can be grouped into a mea-
pitches in a pitch class are called octaves. On a
sure (also called bar). Melody (also called tune) is
modern 88-key piano, the keys are laid out in in-
a sequence of measures (and therefore a sequence
creasing order of pitch. Every successive 12 keys
of notes) that creates the musical piece itself. For
pertain to an octave. (Thus there are keys pertain-
example, a melody could be defined as 620Hz–
ing to 7 octaves and four additional keys pertain-
half-note, 1200Hz-whole-note, 840Hz-half-note,
ing to the eighth octave.) Further, each of the 12
660Hz–quarter-note, and so on. There can be one
keys split the octave such that the difference in fre-
3
Confusingly, note is also commonly used to refer to pitch quency between successive keys in an octave is the
alone. To avoid misunderstanding, we will not use note in that same. Thus the corresponding keys in each octave
sense in this paper. However, some statements, such as play
that pitch may seem odd to those familiar with music, who form a pitch class. For example, the keys at posi-
may be more used to play that note. tion 1, 13, 25, 37, and so on, form a pitch class.
Similarly keys at position 2, 14, 26, 38, and so on, (attributed to Pythagoras) is that the simpler the
form another pitch class. The pitch classes on a ratio between the two frequencies, the more con-
piano are given names C, C#, D, D#, E, F, F#, G, sonant they are (Roederer, 2008; Tenney, 1988).
G#, A, A#, B. (The # is pronounced sharp). The Given a particular scale, some have argued that
same names can also be used to refer to a partic- the order of the pitches in decreasing consonance
ular key in an octave. (In an octave, there exists is as follows: 1st, 5th, 3rd, 6th, 2nd, 4th, and 7th
only one C, only one D#, and so on.) The octaves (Perricone, 2000). Thus for the C major—C (the
are often referred to by a number. On a standard base pitch, or 1st), D (2nd) , E (3rd), F (4th), G
piano, the octaves in increasing order are 0, 1, 2, (5th) , A (6th) , B (7th)—the order of the pitches
and so on. C2 refers to the key in octave 2 that is in decreasing consonance is—C, G, E, A, D, F, B.
in the pitch class C.4 Similarly, for C minor—C (the base pitch, or 1st),
The difference in frequency between successive D (2nd), D# (3rd), F (4th), G (5th), G# (6th), A#
piano keys is called a semitone or Half-Tone (Half (7th)—the order of pitches in decreasing conso-
for short). The interval between two keys sep- nance is—C, G, D#, G#, D, F, A#. We will use
arated by exactly one key is called Whole-Tone these orders in TransProse to generate more dis-
(Whole for short). Thus, the interval between cordant and unstable pitches to reflect higher emo-
C and C# is half, whereas the interval between tion word densities in the novels.
C and D is whole. A scale is any sequence of
pitches ordered by frequency. A major scale is 3 Related Work
a sequence of pitches obtained by applying the
ascending pattern: Whole–Whole–Half–Whole– This work is related to automatic sentiment and
Whole–Whole–Half. For example, if one starts emotion analysis of text (computational linguis-
with C, then the corresponding C major scale con- tics), the generation of music (music theory), as
sists of C, D (frequency of C + Whole interval), E well as the perception of music (psychology).
(frequency of D + Whole interval), F (frequency Sentiment analysis techniques aim to deter-
of E + Half interval), G, A, B, C. Major scales can mine the evaluative nature of text—positive,
begin with any pitch (not just C), and that pitch negative, or neutral. They have been applied
is called the base pitch. A major key is the set of to many different kinds of texts including
pitches corresponding to the major scale. Playing customer reviews (Pang and Lee, 2008), news-
in the key of C major means that one is primarily paper headlines (Bellegarda, 2010), emails
playing the keys (pitches) from the corresponding (Liu et al., 2003; Mohammad and Yang, 2011),
scale, C major scale (although not necessarily in a and tweets (Pak and Paroubek, 2010;
particular order). Agarwal et al., 2011; Thelwall et al., 2011;
Minor scales are series of pitches obtained Brody and Diakopoulos, 2011;
by applying the ascending pattern: Whole-Half– Aisopos et al., 2012; Bakliwal et al., 2012).
Whole–Whole–Half–Whole–Whole. Thus, C mi- Surveys by Pang and Lee (2008) and Liu and
nor is C, D, D#, F, G, G#, A#, C. A minor key Zhang (2012) give a summary of many of these
is the set of pitches corresponding to the minor approaches. Emotion analysis and affective
scale. Playing in major keys generally creates computing involve the detection of emotions
lighter sounding pieces, whereas playing in minor such as joy, anger, sadness, and anticipation
keys creates darker sounding pieces. in text. A number of approaches for emotion
analysis have been proposed in recent years
Consonance is how pleasant or stable one per-
(Boucouvalas, 2002; Zhe and Boucouvalas, 2002;
ceives two pitches played simultaneously (or one
Aman and Szpakowicz, 2007;
after the other). There are many theories on what
Neviarouskaya et al., 2009; Kim et al., 2009;
makes two pitches consonant, some of which are
Bollen et al., 2009; Tumasjan et al., 2010). Text-
culturally dependent. The most common notion
to-speech synthesis employs emotion detection
4
The frequencies of piano keys at a given position across to produce speech consistent with the emotions
octaves is in log scale. For example, frequencies of C1, in the text (Iida et al., 2000; Pierre-Yves, 2003;
C2,. . . , and so on are in log scale. The perception of sound Schröder, 2009). See surveys by Picard (2000)
(frequency) in the human ear is also roughly logarithmic.
Also, the frequency 440Hz (mentioned above) is A4 and it and Tao and Tan (2005) for a broader review of
is the customary tuning standard for musical pitch. the research in this area.
 Some prior empirical sentiment analysis work associated with joy and calm, whereas a se-
focuses specifically on literary texts. Alm and quence of disconsonant notes is associated
Sproat (2005) analyzed twenty two Brothers with excitement, anger, or unpleasantness
Grimm fairy tales to show that fairy tales often (Gabrielsson and Lindström, 2001).
began with a neutral sentence and ended with a • Major and Minor Keys: Major keys are
happy sentence. Mohammad (2012) visualized the associated with happiness, whereas mi-
emotion word densities in novels and fairy tales. nor keys are associated with sadness
Volkova et al. (2010) study human annotation of (Hunter et al., 2010; Hunter et al., 2008;
emotions in fairy tales. However, there exists no Ali and Peynirciolu, 2010;
work connecting automatic detection of sentiment Gabrielsson and Lindström, 2001;
with the automatic generation of music. Webster and Weir, 2005).
Methods for both sentiment and emotion analy- • Tempo: Fast tempo is associ-
sis often rely on lexicons of words associated with ated with happiness or excitement
various affect categories such as positive and neg- (Hunter et al., 2010; Hunter et al., 2008;
ative sentiment, and emotions such as joy, sad- Ali and Peynirciolu, 2010;
ness, fear, and anger. The WordNet Affect Lexi- Gabrielsson and Lindström, 2001;
con (WAL) (Strapparava and Valitutti, 2004) has a Webster and Weir, 2005).
few hundred words annotated with associations to
Studies have shown that even though many of
a number of affect categories including the six Ek-
the associations mentioned above are largely uni-
man emotions (joy, sadness, anger, fear, disgust,
versal, one’s own culture also influences the per-
and surprise).5 The NRC Emotion Lexicon, com-
ception of music (Morrison and Demorest, 2009;
piled by Mohammad and Turney (2010; 2013),
Balkwill and Thompson, 1999).
has annotations for about 14000 words with eight
emotions (six of Ekman, trust, and anticipation).6 4 Our System: TransProse
We use this lexicon in our project.
Automatic or semi-automatic generation of mu- Our system, which we call TransProse, generates
sic through computer algorithms was first popular- music according to the use of emotion words in
ized by Brian Eno (who coined the term generative a given novel. It does so in three steps: First, it
music) and David Cope (Cope, 1996). Lerdahl and analyzes the input text and generates an emotion
Jackendoff (1983) authored a seminal book on the profile. The emotion profile is simply a collection
generative theory of music. Their work greatly in- of various statistics about the presence of emotion
fluenced future work in automatic generation of words in the text. Second, based on the emotion
music such as that of Collins (2008) and Biles profile of the text, the system generates values for
(1994). However, these pieces did not attempt to tempo, scale, octave, notes, and the sequence of
explicitly capture emotions. notes for multiple melodies. Finally, these values
Dowling and Harwood (1986) showed that vast are provided to JFugue, an open-source Java API
amounts of information are processed when listen- for programming music, that generates the appro-
ing to music, and that the most expressive quality priate audio file. In the sections ahead, we de-
that one perceives is emotion. The communica- scribe the three steps in more detail.
tion of emotions in non-verbal utterances and in 5 Calculating Emotion Word Densities
music show how emotions in music have an evo-
lutionary basis (Rousseau, 2009; Spencer, 1857; Given a novel in electronic form, we use the NRC
Juslin and Laukka, 2003). There are many known Emotion Lexicon (Mohammad and Turney, 2010;
associations between music and emotions: Mohammad and Turney, 2013) to identify the
• Loudness: Loud music is associated with number of words in each chapter that are associ-
intensity, power, and anger, whereas soft ated with an affect category. We generate counts
music is associated with sadness or fear for eight emotions (anticipation, anger, joy, fear,
(Gabrielsson and Lindström, 2001). disgust, sadness, surprise, and trust) as well as
for positive and negative sentiment. We partition
• Melody: A sequence of consonant notes is the novel into four sections representing the begin-
5
http://wndomains.fbk.eu/wnaffect.html ning, early middle, late middle, and end. Each sec-
6
http://www.purl.org/net/NRCemotionlexicon tion is further partitioned into four sub-sections.
The number of sections, the number of subsec- more, the eventual intent is to include mid-piece
tions per section, and the number of notes gener- key changes for added effect. For example, chang-
ated for each of the subsections together determine ing the key from C major to A minor when the plot
the total number of notes generated for the novel. suddenly turns sad. The process of changing key
Even though we set the number of sections and is called modulation. Certain transitions such as
number of sub-sections to four each, these settings moving from C major to A minor are commonly
can be varied, especially for significantly longer or used and musically interesting.
shorter pieces of text.
For each section and for each sub-section the ra- 6.2 Melodies
tio of emotion words to the total number of words We use three melodies to capture the change in
is calculated. We will refer to this ratio as the over- emotion word usage in the text. The notes in
all emotions density. We also calculate densities one melody are based on the overall emotion word
of particular emotions, for example, the joy den- density (the emotion words associated with any of
sity, anger density, etc. As described in the section the eight emotions in the NRC Emotion Lexicon).
ahead, the emotion densities are used to generate We will refer to this melody, which is intended to
sequences of notes for each of the subsections. capture the overarching emotional movement, as
melody o or Mo (the ‘o’ stands for overall emo-
6 Generating Music Specifications tion). The notes in the two other melodies, melody
e1 (Me1 ) and melody e2 (Me2 ), are determined
Each of the pieces presented in this paper are
by the most prevalent and second most prevalent
for the piano with three simultaneous, but differ-
emotions in the text, respectively. Precisely how
ent, melodies coming together to form the musical
the notes are determined is described in the next
piece. Two melodies sounded too thin (simple),
sub-section, but first we describe how the octaves
and four or more melodies sounded less cohesive.
of the notes is determined.
6.1 Major and Minor Keys The octave of melody o is proportional to the
difference between the joy and sadness densities
Major keys generally create a more positive atmo-
of the novel. We will refer to this difference by JS.
sphere in musical pieces, whereas minor keys tend
We calculated the lowest density difference (JSmin )
to produce pieces with more negative undertones
and highest JS score (JSmax ) in a collection of nov-
(Hunter et al., 2010; Ali and Peynirciolu, 2010;
els. For a novel with density difference, JS, the
Gabrielsson and Lindström, 2001;
score is linearly mapped to octave 4, 5, or 6 of a
Webster and Weir, 2005). No consensus has
standard piano, as per the formula shown below:
been reached on whether particular keys them-
selves (for example, A minor vs E minor) evoke (JS − JSmin ) ∗ (6 − 4)
different emotions, and if so, what emotions are Oct(Mo ) = 4 + r( ) (1)
JSmax − JSmin
evoked by which keys. For this reason, the pro-
totype of Transprose does not consider different The function r rounds the expression to the closest
keys; the chosen key for the produced musical integer. Thus scores closer to JSmin are mapped
pieces is limited to either C major or C minor. (C to octave 4, scores closer to JSmax are mapped to
major was chosen because it is a popular choice octave 6, and those in the middle are mapped to
when teaching people music. It is simple because octave 5.
it does not have any sharps. C minor was chosen The octave of Me1 is calculated as follows:
as it is the minor counterpart of C major.) 
Whether the piece is major or minor is de- Oct(Mo ) + 1, if e1 is joy or trust


termined by the ratio of the number of positive 
Oct(M ) − 1, if e1 is anger, fear,
o
words to the number of negative words in the en- Oct(Me1 ) =
 sadness, or disgust
tire novel. If the ratio is higher than 1, C major


Oct(M ),

otherwise
is used, that is, only pitches pertaining to C major o

are played. If the ratio is 1 or lower, C minor is (2)

used. That is, Me1 is set to:
Experimenting with keys other than C major • an octave higher than the octave of Mo if e1
and C minor is of interest for future work. Further- is a positive emotion,
 • an octave lower than the octave of Mo if e1 is (with shorter notes making the piece sound more
a negative emotion, active), and longer notes during periods of lower
• the same octave as that of Mo if e1 is surprise emotional activity.
or anticipation. Pitch: If the number of notes for a measure
is n, then the corresponding sub-section is parti-
Recall that higher octaves evoke a sense of positiv- tioned into n equal parts and the pitch for each
ity, whereas lower octaves evoke a sense of nega- note is based on the emotion density of the cor-
tivity. The octave of Me2 is calculated exactly as responding sub-section. Lower emotion densities
that of Me1 , except that it is based on the second are mapped to more consonant pitches in the key
most prevalent emotion (and not the most preva- (C major or C minor), whereas higher emotion
lent emotion) in the text. densities are mapped to less consonant pitches in
the same scale. For example, if the melody is in
6.3 Structure and Notes
the key of C major, then the lowest to highest emo-
As mentioned earlier, TransProse generates three tion densities are mapped linearly to the pitches C,
melodies that together make the musical piece for G, E, A, D, F, B. Thus, a low emotion value would
a novel. The method for generating each melody create a pitch that is more consonant and a high
is the same, with the exception that the three emotion value would create a pitch that is more
melodies (Mo , Me1 , and Me2 ) are based on the dissonant (more interesting and unusual).
overall emotion density, predominant emotion’s Repetition: Once the four measures of a section
density, and second most dominant emotion’s den- are played, the same four measures are repeated
sity, respectively. We describe below the method in order to create a more structured and melodic
common for each melody, and use emotion word feeling. Without the repetition, the piece sounds
density as a stand in for the appropriate density. less cohesive.
Each melody is made up of four sections, repre-
senting four sections of the novel (the beginning, 6.4 Tempo
early middle, late middle, and end).In turn, each We use a 44 time signature (common time) be-
section is represented by four measures. Thus each cause it is one of the most popular time signatures.
measure corresponds to a quarter of a section (a Thus each measure (sub-section) has 4 beats. We
sub-section). A measure, as defined earlier, is a determined tempo (beats per minute) by first de-
series of notes. The number of notes, the pitch of termining how active the target novel is. Each
each note, and the relative duration of each note of the eight basic emotions is assigned to be ei-
are determined such that they reflect the emotion ther active, passive, or neutral. In TransProse, the
word densities in the corresponding part of the tempo is proportional to the activity score, which
novel. we define to be the difference between the aver-
Number of Notes: In our implementation, we age density of the active emotions (anger and joy)
decided to contain the possible note durations and the average density of the passive emotions
to whole notes, half notes, quarter notes, eighth (sadness). The other five emotions (anticipation,
notes, and sixteenth notes. A relatively high emo- disgust, fear, surprise, and trust) were considered
tion density is represented by many notes, whereas ambiguous or neutral, and did not influence the
a relatively low emotion density is represented by tempo.
fewer notes. We first split the interval between the We subjectively identified upper and lower
maximum and minimum emotion density for the bounds for the possible tempo values to be 180 and
novel into five equal parts (five being the num- 40 beats/minute, respectively. We determined ac-
ber of note duration choices – whole, half, quar- tivity scores for a collection of novels, and identi-
ter, eighth, or sixteenth). Emotion densities that fied the highest activity score (Actmax ) and the low-
fall in the lowest interval are mapped to a single est activity score (Actmin ). For a novel whose ac-
whole note. Emotion densities in the next interval tivity score was Act, we determined tempo as per
are mapped to two half-notes. The next interval the formula shown below:
is mapped to four quarter-notes. And so on, until
(Act − Actmin ) ∗ (180 − 40)
the densities in the last interval are mapped to six- tempo = 40 + (3)
Actmax − Actmin
teen sixteenth-notes (1/16th ). The result is shorter
notes during periods of higher emotional activity Thus, high activity scores were represented by
tempo values closer to 180 and lower activity been read by many people, the readers can com-
scores were represented by tempo values closer to pare their understanding of the story with the mu-
40. The lowest activity score in our collection of sic generated by TransProse. Table 1 presents de-
texts, Actmin , was -0.002 whereas the highest ac- tails of some of these novels.
tivity score, Actmax , was 0.017.
8.1 Overall Tone
7 Converting Specifications to Music TransProse captures the overall positive or nega-
tive tone of the novel by assigning an either major
JFugue is an open-source Java API that helps cre- or minor key to the piece. Peter Pan and Anne
ate generative music.7 It allows the user to easily of Green Gables, novels with overall happy and
experiment with different notes, instruments, oc- uplifting moods, created pieces in the major key.
taves, note durations, etc within a Java program. On the other hand, novels such as Heart of Dark-
JFugue requires a line of specifically-formatted ness, A Clockwork Orange, and The Road, with
text that describes the melodies in order to play dark themes, created pieces in the minor key. The
them. The initial portion of the string of JFugue to- effect of this is pieces that from the start have a
kens for the novel Peter Pan is shown below. The mood that aligns with the basic mood of the novel
string conveys the overall information of the piece they are based on.
as well as the first eight measures (or one section)
for each of the three melodies (or voices). 8.2 Overall Happiness and Sadness
The densities of happiness and sadness in a novel
KCmaj X[VOLUME]=16383 V0 T180
are represented in the baseline octave of a piece.
A6/0.25 D6/0.125 F6/0.25 B6/0.25
This representation instantly conveys whether the
B6/0.125 B6/0.25 B6/0.25...
novel has a markedly happy or sad mood. The
K stands for key and Cmaj stands for C major. overall high happiness densities in Peter Pan and
This indicates that the rest of the piece will be in Anne of Green Gables create pieces in an octave
the key of C major. The second token controls the above the average, resulting in higher tones and a
volume, which in this example is at the loudest lighter overall mood. Similarly, the overall high
value (16383). V0 stands for the first melody (or sadness densities in The Road and Heart of Dark-
voice). The tokens with the letter T indicate the ness result in pieces an octave lower than the aver-
tempo, which in the case of this example is 180 age, and a darker overall tone to the music. Nov-
beats per minute. els, such as A Clockwork Orange, and The Little
Prince, where happiness and sadness are not dra-
The tokens that follow indicate the notes of the
matically higher or lower than the average novel
melody. The letter is the pitch class of the note,
remain at the average octave, allowing for the cre-
and the number immediately following it is the
ation of a more nuanced piece.
octave. The number following the slash char-
acter indicates the duration of the note. (0.125 8.3 Activeness of the Novel
is an eighth-note (1/8th), 0.25 is a quarter note,
Novels with lots of active emotion words, such
0.5 is a half note, and 1.0 is a whole note.) We
as Peter Pan, Anne of Green Gables, Lord of the
used JFugue to convert the specifications of the
Flies, and A Clockwork Orange, generate fast-
melodies into music. JFugue saves the pieces as
paced pieces with tempos over 170 beats per
a midi files, which we converted to MP3 format.8
minute. On the other hand, The Road, which has
relatively few active emotion words is rather slow
8 Case Studies
(a tempo of 42 beats per minute).
We created musical pieces for several popular
8.4 Primary Emotions
novels through TransProse. These pieces are
available at: http://transprose.weebly.com/final- The top two emotions of a novel inform two of the
pieces.html. Since these novels are likely to have three melodies in a piece (Me1 and Me2 ). Recall
that if the melody is based on a positive emotion, it
7
http://www.jfugue.org will be an octave higher than the octave of Mo , and
8
The MP3 format uses a lossy data compression, but the
resulting files are significantly smaller in size. Further, a if it is based on a negative emotion, it will be an oc-
wider array of music players support the MP3 format. tave lower. For novels where the top two emotions
Table 1: Emotion and audio features of a few popular novels that were processed by TransProse. The
musical pieces are available at: http://transprose.weebly.com/final-pieces.html.
Book Title Emotion 1 Emotion 2 Octave Tempo Pos/Neg Key Activity Joy-Sad
A Clockwork Orange Fear Sadness 5 171 Negative C Minor 0.009 -0.0007
Alice in Wonderland Trust Fear 5 150 Positive C Major 0.007 -0.0002
Anne of Green Gables Joy Trust 6 180 Positive C Major 0.010 0.0080
Heart of Darkness Fear Sadness 4 122 Negative C Minor 0.005 -0.0060
Little Prince, The Trust Joy 5 133 Positive C Major 0.006 0.0028
Lord of The Flies Fear Sadness 4 151 Negative C Minor 0.008 -0.0053
Peter Pan Trust Joy 6 180 Positive C Major 0.010 0.0040
Road, The Sadness Fear 4 42 Negative C Minor -0.002 -0.0080
To Kill a Mockingbird Trust Fear 5 132 Positive C Major 0.006 -0.0013

are both positive, such as Anne of Green Gables times chaotic. The effect of increasing and de-
(trust and joy), the pieces sound especially light creasing the total number of sections and sub-
and joyful. For novels where the top two emotions sections is also presented. Additionally, the web-
are both negative, such as The Road (sadness and page displays pieces with tempos and octaves be-
fear), the pieces sound especially dark. yond the limits chosen in TransProse. We also
show other variations such as pieces for relatively
8.5 Emotional Activity positive novels generated in C minor (instead of
Unlike the overall pace of the novel, individual C major). These alternatives are not necessarily
segments of activity were also identified in the incorrect, but they tend to often be less effective.
pieces through the number and duration of notes
(with more and shorter notes indicating higher 10 Limitations and Future work
emotion densities). This can be especially heard
in the third section of A Clockwork Orange, the fi- We presented a system, TransProse, that gener-
nal section of The Adventures of Sherlock Holmes, ates music according to the use of emotion words
the second section of To Kill a Mockingbird, and in a given piece of text. A number of avenues
the final section of Lord of the Flies. In A Clock- for future work exist such as exploring the use of
work Orange, the main portion of the piece is mid-piece key changes and intentional harmony
chaotic and eventful, likely as the main charac- and discord between the melodies. We will fur-
ters cause havoc; at the end of the novel (as the ther explore ways to capture activity in music. For
main character undergoes therapy) the piece dra- example, an automatically generated activity lex-
matically changes and becomes structured. Sim- icon (built using the method proposed by Turney
ilarly, in Heart of Darkness, the piece starts out and Littman (2003)) can be used to identify por-
only playing a few notes; as the tension in the tions of text where the characters are relatively
novel builds, the number of notes increases and active (fighting, dancing, conspiring, etc) and ar-
their durations decrease. eas where they are relatively passive (calm, inca-
pacitated, sad, etc). One can even capture non-
9 Comparing Alternative Choices
emotional features of the text in music. For ex-
We examine choices made in TransProse by com- ample, recurring characters or locations in a novel
paring musical pieces generated with different al- could be indicated by recurring motifs. We will
ternatives. These audio clips are available here: conduct human evaluations asking people to judge
http://transprose.weebly.com/clips.html. various aspects of the generated music such as the
Pieces with two melodies (based on overall quality of music and the amount and type of emo-
emotion density and the predominant emotion’s tion evoked by the music. We will also evaluate
density) and pieces based on four melodies (based the impact of textual features such as the length of
on the top three emotions and the overall emotion the novel and the style of writing on the generated
density) were generated and uploaded in the clips music. Work on capturing note models (analogous
webpage. Observe that with only two melodies, to language models) from existing pieces of music
the pieces tend to sound thin, whereas with four and using them to improve the music generated by
melodies the pieces sound less cohesive and some- TransProse seems especially promising.
 References microblogs. In Proceedings of the Conference on
[Agarwal et al.2011] Apoorv Agarwal, Boyi Xie, Ilia Empirical Methods in Natural Language Process-
Vovsha, Owen Rambow, and Rebecca Passonneau. ing, EMNLP ’11, pages 562–570, Stroudsburg, PA,
2011. Sentiment analysis of twitter data. In Pro- USA. Association for Computational Linguistics.
ceedings of the Workshop on Languages in Social [Brown1970] Calvin S Brown. 1970. The relations be-
Media, LSM ’11, pages 30–38, Portland, Oregon. tween music and literature as a field of study. Com-
[Aisopos et al.2012] Fotis Aisopos, George Papadakis, parative Literature, 22(2):97–107.
Konstantinos Tserpes, and Theodora Varvarigou. [Collins2008] Nick Collins. 2008. The analysis
2012. Textual and contextual patterns for sentiment of generative music programs. Organised Sound,
analysis over microblogs. In Proceedings of the 13(3):237–248.
21st International Conference on World Wide Web
Companion, WWW ’12 Companion, pages 453– [Cope1996] David Cope. 1996. Experiments in musi-
454, New York, NY, USA. cal intelligence, volume 12. AR Editions Madison,
WI.
[Ali and Peynirciolu2010] S Omar Ali and Zehra F
Peynirciolu. 2010. Intensity of emotions con- [Dowling and Harwood1986] W Jay Dowling and
veyed and elicited by familiar and unfamiliar mu- Dane L Harwood. 1986. Music cognition, volume
sic. Music Perception: An Interdisciplinary Journal, 19986. Academic Press New York.
27(3):177–182. [Gabrielsson and Lindström2001] Alf Gabrielsson and
[Alm and Sproat2005] Cecilia O. Alm and Richard Erik Lindström. 2001. The influence of musical
Sproat, 2005. Emotional sequencing and develop- structure on emotional expression.
ment in fairy tales, pages 668–674. Springer. [Genereux and Evans2006] Michel Genereux and
[Aman and Szpakowicz2007] Saima Aman and Stan Roger P. Evans. 2006. Distinguishing affective
Szpakowicz. 2007. Identifying expressions of emo- states in weblogs. In Proceedings of the AAAI
tion in text. In Vclav Matoušek and Pavel Mautner, Spring Symposium on Computational Approaches
editors, Text, Speech and Dialogue, volume 4629 of to Analysing Weblogs, pages 27–29, Stanford,
Lecture Notes in Computer Science, pages 196–205. California.
Springer Berlin / Heidelberg. [Hatten1991] Robert Hatten. 1991. On narrativity in
[Bakliwal et al.2012] Akshat Bakliwal, Piyush Arora, music: expressive genres and levels of discourse in
Senthil Madhappan, Nikhil Kapre, Mukesh Singh, beethoven.
and Vasudeva Varma. 2012. Mining sentiments [Hunter et al.2008] Patrick G Hunter, E Glenn Schel-
from tweets. In Proceedings of the 3rd Workshop on lenberg, and Ulrich Schimmack. 2008. Mixed
Computational Approaches to Subjectivity and Sen- affective responses to music with conflicting cues.
timent Analysis, WASSA ’12, pages 11–18, Jeju, Re- Cognition & Emotion, 22(2):327–352.
public of Korea.
[Hunter et al.2010] Patrick G Hunter, E Glenn Schel-
[Balkwill and Thompson1999] Laura-Lee Balkwill and lenberg, and Ulrich Schimmack. 2010. Feelings
William Forde Thompson. 1999. A cross-cultural and perceptions of happiness and sadness induced
investigation of the perception of emotion in music: by music: Similarities, differences, and mixed emo-
Psychophysical and cultural cues. Music perception, tions. Psychology of Aesthetics, Creativity, and the
pages 43–64. Arts, 4(1):47.
[Bellegarda2010] Jerome Bellegarda. 2010. Emotion [Iida et al.2000] Akemi Iida, Nick Campbell, Soichiro
analysis using latent affective folding and embed- Iga, Fumito Higuchi, and Michiaki Yasumura. 2000.
ding. In Proceedings of the NAACL-HLT 2010 A speech synthesis system with emotion for assist-
Workshop on Computational Approaches to Analy- ing communication. In ISCA Tutorial and Research
sis and Generation of Emotion in Text, Los Angeles, Workshop (ITRW) on Speech and Emotion.
California.
[Juslin and Laukka2003] Patrik N Juslin and Petri
[Biles1994] John Biles. 1994. Genjam: A genetic al- Laukka. 2003. Communication of emotions in
gorithm for generating jazz solos. pages 131–137. vocal expression and music performance: Differ-
[Bollen et al.2009] Johan Bollen, Alberto Pepe, and ent channels, same code? Psychological bulletin,
Huina Mao. 2009. Modeling public mood and emo- 129(5):770.
tion: Twitter sentiment and socio-economic phe- [Kim et al.2009] Elsa Kim, Sam Gilbert, Michael J. Ed-
nomena. CoRR. wards, and Erhardt Graeff. 2009. Detecting sadness
[Boucouvalas2002] Anthony C. Boucouvalas. 2002. in 140 characters: Sentiment analysis of mourning
Real time text-to-emotion engine for expressive in- michael jackson on twitter.
ternet communication. Emerging Communication: [Lerdahl and Jackendoff1983] Fred Lerdahl and Ray S
Studies on New Technologies and Practices in Com- Jackendoff. 1983. A generative theory of tonal mu-
munication, 5:305–318. sic. MIT press.
[Brody and Diakopoulos2011] Samuel Brody [Liu and Zhang2012] Bing Liu and Lei Zhang. 2012.
and Nicholas Diakopoulos. 2011. A survey of opinion mining and sentiment analysis.
Cooooooooooooooollllllllllllll!!!!!!!!!!!!!!: us- In Charu C. Aggarwal and ChengXiang Zhai, edi-
ing word lengthening to detect sentiment in tors, Mining Text Data, pages 415–463. Springer.
[Liu et al.2003] Hugo Liu, Henry Lieberman, and Ted [Pierre-Yves2003] Oudeyer Pierre-Yves. 2003. The
Selker. 2003. A model of textual affect sensing us- production and recognition of emotions in speech:
ing real-world knowledge. In Proceedings of the 8th features and algorithms. International Journal of
International Conference on Intelligent User Inter- Human-Computer Studies, 59(1):157–183.
faces, pages 125–132, New York, NY. ACM. [Roederer2008] Juan G Roederer. 2008. The physics
[Micznik2001] Vera Micznik. 2001. Music and nar- and psychophysics of music: an introduction.
rative revisited: degrees of narrativity in beethoven Springer Publishing Company, Incorporated.
and mahler. Journal of the Royal Musical Associa- [Rousseau2009] Jean-Jacques Rousseau. 2009. Essay
tion, 126(2):193–249. on the origin of languages and writings related to
[Mihalcea and Liu2006] Rada Mihalcea and Hugo Liu. music, volume 7. UPNE.
2006. A corpus-based approach to finding happi- [Schröder2009] Marc Schröder. 2009. Expressive
ness. In Proceedings of the AAAI Spring Sympo- speech synthesis: Past, present, and possible futures.
sium on Computational Approaches to Analysing In Affective information processing, pages 111–126.
Weblogs, pages 139–144. AAAI Press. Springer.
[Mohammad and Turney2010] Saif M. Mohammad and [Spencer1857] Herbert Spencer. 1857. The origin and
Peter D. Turney. 2010. Emotions evoked by com- function of music. Frasers Magazine, 56:396–408.
mon words and phrases: Using Mechanical Turk
[Strapparava and Valitutti2004] Carlo Strapparava and
to create an emotion lexicon. In Proceedings of
Alessandro Valitutti. 2004. WordNet-Affect: An
the NAACL-HLT Workshop on Computational Ap-
Affective Extension of WordNet. In Proceedings of
proaches to Analysis and Generation of Emotion in
the 4th International Conference on Language Re-
Text, LA, California.
sources and Evaluation (LREC-2004), pages 1083–
[Mohammad and Turney2013] Saif M. Mohammad and 1086, Lisbon, Portugal.
Peter D. Turney. 2013. Crowdsourcing a word- [Tao and Tan2005] Jianhua Tao and Tieniu Tan. 2005.
emotion association lexicon. 29(3):436–465. Affective computing: A review. In Affective com-
[Mohammad and Yang2011] Saif M. Mohammad and puting and intelligent interaction, pages 981–995.
Tony (Wenda) Yang. 2011. Tracking sentiment in Springer.
mail: How genders differ on emotional axes. In Pro- [Tenney1988] James Tenney. 1988. A history of conso-
ceedings of the ACL Workshop on Computational nance and dissonance. Excelsior Music Publishing
Approaches to Subjectivity and Sentiment Analysis, Company New York.
WASSA ’11, Portland, OR, USA.
[Thelwall et al.2011] Mike Thelwall, Kevan Buckley,
[Mohammad2012] Saif M. Mohammad. 2012. From and Georgios Paltoglou. 2011. Sentiment in Twitter
once upon a time to happily ever after: Tracking events. Journal of the American Society for Infor-
emotions in mail and books. Decision Support Sys- mation Science and Technology, 62(2):406–418.
tems, 53(4):730–741.
[Tumasjan et al.2010] Andranik Tumasjan, Timm O
[Morrison and Demorest2009] Steven J Morrison and Sprenger, Philipp G Sandner, and Isabell M Welpe.
Steven M Demorest. 2009. Cultural constraints on 2010. Predicting elections with twitter : What 140
music perception and cognition. Progress in brain characters reveal about political sentiment. Word
research, 178:67–77. Journal Of The International Linguistic Association,
[Neviarouskaya et al.2009] Alena Neviarouskaya, Hel- pages 178–185.
mut Prendinger, and Mitsuru Ishizuka. 2009. Com- [Turney and Littman2003] Peter Turney and Michael L
positionality principle in recognition of fine-grained Littman. 2003. Measuring praise and criticism:
emotions from text. In Proceedings of the Proceed- Inference of semantic orientation from association.
ings of the Third International Conference on We- ACM Transactions on Information Systems, 21(4).
blogs and Social Media (ICWSM-09), pages 278–
281, San Jose, California. [Volkova et al.2010] Ekaterina P Volkova, Betty J
Mohler, Detmar Meurers, Dale Gerdemann, and
[Pak and Paroubek2010] Alexander Pak and Patrick Heinrich H Bülthoff. 2010. Emotional perception of
Paroubek. 2010. Twitter as a corpus for senti- fairy tales: Achieving agreement in emotion annota-
ment analysis and opinion mining. In Proceed- tion of text. In Proceedings of the NAACL HLT 2010
ings of the 7th Conference on International Lan- Workshop on Computational Approaches to Analysis
guage Resources and Evaluation, LREC ’10, Val- and Generation of Emotion in Text, pages 98–106.
letta, Malta, May. European Language Resources Association for Computational Linguistics.
Association (ELRA).
[Webster and Weir2005] Gregory D Webster and
[Pang and Lee2008] Bo Pang and Lillian Lee. 2008. Catherine G Weir. 2005. Emotional responses to
Opinion mining and sentiment analysis. Founda- music: Interactive effects of mode, texture, and
tions and Trends in IR, 2(1–2):1–135. tempo. Motivation and Emotion, 29(1):19–39.
[Perricone2000] J. Perricone. 2000. Melody in Song- [Zhe and Boucouvalas2002] Xu Zhe and A Boucou-
writing: Tools and Techniques for Writing Hit valas, 2002. Text-to-Emotion Engine for Real Time
Songs. Berklee guide. Berklee Press. Internet CommunicationText-to-Emotion Engine for
[Picard2000] Rosalind W Picard. 2000. Affective com- Real Time Internet Communication, pages 164–168.
puting. MIT press.