See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/2525888

Algorithms for Discovering Repeated Patterns in Multidimensional

Representations of Polyphonic Music

Article · August 2002

Source: CiteSeer

CITATIONS READS

82 238

6 authors, including:

Kjell Lemström Geraint Anthony Wiggins

University of Helsinki Vrije Universiteit Brussel
78 PUBLICATIONS   983 CITATIONS    221 PUBLICATIONS   4,056 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Look! It's moving! Is it alive? How movement affects humans' affinity to living and non-living entities View project

SerenA Project View project

All content following this page was uploaded by Kjell Lemström on 04 July 2014.

The user has requested enhancement of the downloaded file.

 Algorithms for discovering repeated patterns in multidimensional
representations of polyphonic music∗
David Meredith Kjell Lemström Geraint A. Wiggins
Department of Computing, Department of Computer Science, Department of Computing,
City University, London. University of Helsinki. City University, London.
[email protected] [email protected] [email protected]

March 10, 2003

Abstract wants to find a wide range of different types of musical

repeated pattern using a string-based approach, one
In this paper we give an overview of four algorithms generally has to run a variety of different algorithms
that we have developed for pattern matching, pattern on a number of different representations of the music.
discovery and data compression in multidimensional In our work we have avoided such difficulties by
datasets. We show that these algorithms can fruit- adopting a geometric approach in which the music
fully be used for processing musical data. In particular, to be analysed is represented as a multidimensional
we show that our algorithms can discover instances of dataset—that is, a set of points in a Euclidean space.
perceptually significant musical repetition that cannot We have found that by doing this we are able to, first,
be found using previous approaches. We also describe process polyphonic music as easily and efficiently as
results that suggest the possibility of using our data- monophonic music; second, compute some of the rep-
compression algorithm for modelling expert motivic- etitions that are difficult to find using a string-based
thematic music analysis. approach; and third, essentially dispense with multi-
ple representations because we can run the same small
set of algorithms on various orthogonal projections of
1 Introduction a single, rich multidimensional representation of the
In this paper we give an overview of some algorithms music.
that we have developed for discovering perceptually sig- In this paper we briefly describe two repetition dis-
nificant repeated patterns in polyphonic music. These covery algorithms, SIA and SIATEC, that are based on
algorithms can straightforwardly be adapted for mu- this new approach. SIA computes all the maximal re-
sic information retrieval (pattern matching) and data- peated patterns in a dataset and SIATEC computes all
compression. We’ll begin in the next section by pre- the occurrences of all the maximal repeated patterns in
senting some examples of perceptually significant re- a dataset. We then briefly discuss what happens when
peated patterns in music which illustrate just how di- you run these algorithms on music data. For detailed
verse this class of phenomena is. descriptions of the algorithms and precise definitions
Many music psychologists and music analysts (Bent of the functions that they compute, see Meredith et al.
and Drabkin, 1987; Lerdahl and Jackendoff, 1983; Nat- (2002a) and Meredith et al. (2002b).
tiez, 1975; Ruwet, 1972; Schenker, 1954; Temperley, Our experiments suggest that the repeated patterns
2001) have noted that being able to identify the per- that we are interested in finding are often either equal
ceptually significant repetitions in a passage of music to the maximal repeated patterns computed by SIA or
is often extremely important for achieving a rich un- derivable from them. However, typically, SIA also gen-
derstanding of it. However, the vast majority of exact erates many patterns that are not musically interest-
repetitions in a piece of tonal music do, in fact, go un- ing. Some post-processing is therefore usually required
noticed by listeners (see below). to isolate the interesting repetitions in the output of
It seems that most previous approaches to repetition SIA and SIATEC and we suggest some heuristics that
discovery in music have been based on the assumption seem to be useful for doing this.
that the music to be processed is represented in the When these heuristics are incorporated into a data-
form of a string or a set of strings. We will briefly re- compression algorithm based on SIATEC (which we call
view some of these string-based approaches and demon- COSIATEC), we find that we can generate some quite
strate that there seem to be certain important types of interesting motivic and thematic music analyses. We
musical repetition that are very difficult to find using then describe a pattern-matching algorithm based on
such approaches. In particular, it seems that if one SIA, which we call SIAMESE. This algorithm finds com-
∗ The algorithms described in this paper are the subject of an plete and partial matches of multidimensional query
international patent application (Meredith et al., 2002b). patterns in multidimensional datasets. Although in

1
 Meredith, Lemström and Wiggins, Discovering Repeated Pattern in Polyphonic Music

DD
D D DD E ¶  S
¡ ¡ ¡ O¡ ¡ ¡ D¡ ¡ ¡ ¡ O¡ ¡ ¡
A1 A2 A3
" DD D D E ¶  S ¡  S ¡ ¡
DD ¡ ¡  O ¡ ¡  ¡ ¡ O¡ ¡  ¡
¡  ¡ ¡¡ 
¡ ¡ O¡ ¡ ¡ O¡ ¡ ¡
3
DD Figure 2: Bars 14–16 from Fugue in C major from Book
D D DD
3 3
D¡ ¡ ¶
O¡ ¶ ¡ ¡¡  Ž ¡ ¡ Ž ¡ ¡¡  Ž ¡ ¡ ¡¡ O ¡ ŽD ¡¡ ¡ O ¡ D ¡ 1 of J. S. Bach’s Das Wohltemperirte Klavier (BWV
4 4 ¶ O¡
A3 (cont.) A4 A5 3 
846).
" DD D D ¡ 
D¡ ¡ O¡ DÌ ¡ ¡ O¡ DÌ D¡ O¡ D¡
DD
¡ D¡ ¡ O¡ DÌ ¡  ¡ ¡  O¡ DÌ
¡

Figure 1: Opening bars of Barber’s Sonata for Piano,

Op.26

this paper we focus on the musical applications of these

algorithms, it is important to remember that the algo-
rithms are, in fact, quite general and can be used to
process any data that can appropriately be represented
in the form of a multidimensional dataset. We end the
paper with some suggestions for further research

2 The diversity of musical repe-

tition
Many music psychologists and music analysts (Bent
and Drabkin, 1987; Lerdahl and Jackendoff, 1983; Nat-
tiez, 1975; Ruwet, 1972; Schenker, 1954; Temperley,
2001) have stressed that identifying the significant rep-
etitions in a piece of music is an essential part of achiev- Figure 3: Bars 16–19 of the first movement from
ing a rich and satisfying interpretation of it. Our work Mozart’s Symphony in G minor, K.550.
was originally motivated by the desire to develop a
computational model of expert music cognition and it
seems clear that one component of such a model would the repeated pattern involves the whole orchestra and
have to be able to discover perceptually significant rep- contains notes from 13 voices. These two examples also
etitions, or instances of parallelism, as it is generally show that the occurrences of a pattern may overlap as
called by music psychologists and analysts. they do in Figure 2 or they may occur consecutively, as
However, the class of perceptually significant repe- they do in Figure 3. The occurrences of a perceptually
titions is a very diverse set. There are at least two significant musical repeated pattern may also be widely
reasons for this: first, the patterns involved in such rep- separated in the music as they are, for example, in the
etitions vary widely in their structural characteristics; case of the exposition and recapitulation of a sonata-
and second, there are many different ways of trans- form movement.
forming a musical pattern to give another pattern that Repeated musical patterns also exhibit different
is perceived to be a version of it. For example, musi- types of compactness. For example, the repeated pat-
cal themes and motives can be truncated, augmented, tern in Figure 3 is temporally compact —that is, it con-
diminished, inverted, reversed, embellished and so on. tains all the notes that occur within the time period
The patterns involved in perceptually significant rep- spanned by the pattern. On the other hand, the rising
etitions vary widely in size from small motives (such bass pattern in Figure 1 is bounding box compact—it
as the rising bass line in Figure 1) to whole sections contains all the notes that occur within the pattern’s
of works containing hundreds of notes such as the ex- bounding box when the music is represented as a graph
position of a sonata-form movement. In polyphonic of chromatic pitch against time. In Figure 2, the re-
music in which the voices are unambiguously identifi- peated overlapping occurrences of the subject exhibit
able, the notes in a repeated pattern may all come from yet another type of compactness: each occurrence con-
one voice or they may come from two or more voices. tains all the notes that occur in a single voice within
For example, in the stretto passage shown in Figure 2, the time period spanned by the pattern. In Figure 5
each statement of the subject only contains notes from we see themes being transformed by diminution and in-
a single voice. version and in Figure 4 we see an example of a simple
On the other hand, in Figure 3, each occurrence of melodic idea being embellished by diminution (Forte
 Meredith, Lemström and Wiggins, Discovering Repeated Pattern in Polyphonic Music

A
A B B 3

 ¡ ¡ ¡ Ì ¡ Ì
¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡
1 2 3 4 1 2 3 4 5 6 7 8 9 10 11 12 13
Figure 7: Two versions of a melodic idea separated by
Figure 4: Example of diminution. a high edit distance.

¡ ¡ ¡ ¡ ¡
¡ ¡ ¡ ¡ ¡ ¡¡¡¡ are not musically interesting. One of the motivations
D E ‹
¡ ¡ ¡ behind our work has been to develop algorithms that
À
" EÌ ¡ ¡ ¡
extract only the interesting repeated patterns of a par-
¡ ¡ ¡
D , , Ì Ì , ticular type from the music. This involves formally
characterising what it is about the interesting repe-
4
¡ ¡ ¡ ¡ ¡ ¡ titions that distinguishes them from the many exact
D ¡ ¡ ¡ O ¡ ¡¡ ¡ ¡ ¡  ¡ ¡ ¡
¡ ¡ ¡ ¡ ¡ ¡¡  ¡ ¡ ¡ ¡ repetitions that the expert listener and analyst do not
recognize as being important.
" Ì ¡ ¡ ¡ ¡ ¡
D ¡ ¡ ¡ ¡ ¡
¡ ¡
4 Previous approaches to repe-
Figure 5: Bars 1–5 of Contrapunctus VI from J. S. tition discovery in music
Bach’s Die Kunst der Fuge (BWV 1080).
It seems that most previous attempts to develop a rep-
etition discovery algorithm for music have been based
and Gilbert, 1982), a process in which each long note
on the assumption that the music to be analysed is
is replaced by a sequence of shorter notes.
represented as a string of symbols or as a set of such
strings. An example of such an approach is Pierre-Yves
Rolland’s FlExPat program (Rolland, 1999). This pro-
3 Most repetitions in music are gram can find approximate repetitions within a mono-
not interesting phonic source. It can also be used to find repeated
monophonic patterns in a polyphonic work in which
Although identifying the important repetitions in a each voice is represented as a string. Also, it is ca-
piece significantly enriches a listener’s understanding, pable of finding repeated monophonic patterns which
not all repeated musical patterns are interesting and contain ‘gaps’. However it suffers from a few weak-
significant. For example, in Figure 6 the pattern con- nesses. First, it cannot deal with unvoiced polyphonic
sisting of the notes in elliptical boxes is repeated 7 music such as piano music. Second, it can only find
crotchets later, transposed up a minor ninth to give patterns whose sizes lie within a user-specified range
the pattern consisting of the notes in square boxes. and if the range is set so that it allows patterns of
Clearly, this repetition is just an artefact that results any size, the overall worst-case running time goes up
from the other musically significant repetitions that are to at least O(n4 ). Third, like most string-based ap-
occurring in this passage such as, for example, the ex- proaches to approximate pattern matching, it uses the
act repetition of bar 3 in bar 4. edit-distance approach to compute the similarity be-
In fact, it turns out that, typically, the vast majority tween patterns. Unfortunately, such an approach is
of exact repetitions that occur within a piece of music not typically capable of finding a match between a pat-
tern and a highly-embellished variation on the pattern
such as the one shown in Figure 7. In this example
the edit distance between the two occurrences is rather
large owing to the high number of insertions required
to transform the plain version (A) into the embellished
one (B). A program like Rolland’s regards two patterns
as being similar if the edit distance between them is less
than some threshold k. However, for these two patterns
to be considered ‘similar’ by Rolland’s algorithm, this
value of k would have to be set to at least 14 to allow
for all those extra notes to be inserted. Unfortunately,
this value would in general be too high because the
program would then start regarding highly dissimilar
Figure 6: Bars 1–6 of Rachmaninoff’s Prelude in C
patterns as being similar.
minor, Op.3, No.2. The pattern consisting of the notes Hsu et al. (1998) describe a repetition discovery al-
in square boxes is an exact transposed repetition of the gorithm for music that uses the dynamic programming
pattern consisting of the notes in elliptical boxes. technique. This algorithm suffers from a number of se-
 Meredith, Lemström and Wiggins, Discovering Repeated Pattern in Polyphonic Music

{
0, 27, 16, 2, 2,
1, 46, 27, 1, 1,
2, 39, 23, 2, 2,

2, 44, 26, 1, 1,
3, 46, 27, 1, 1,
4, 32, 19, 2, 2,

4, 47, 28, 1, 1,
5, 44, 26, 1, 1,
6, 39, 23, 2, 2,

6, 42, 25, 1, 1,
7, 44, 26, 1, 1,
8, 30, 18, 2, 2,

8, 46, 27, 1, 1,
9, 42, 25, 1, 1,
10, 39, 23, 2, 2,
... ... ...

26, 29, 17, 1, 2,
26, 51, 30, 2, 1,
27, 30, 18, 1, 2,

28, 32, 19, 1, 2,
28, 41, 24, 2, 1,
29, 29, 17, 1, 2,
Figure 8: Bars 1–2 of the Prelude in C minor from Book
30, 27, 16, 1, 2,
30, 50, 29, 2, 1,
}
31, 29, 17, 1, 2
2 of J. S. Bach’s Das Wohltemperirte Klavier (BWV
Figure 9: Five-dimensional dataset representing the
871).
music in Figure 8.

5 Representing music using mul-

tidimensional datasets
rious weaknesses. First, it cannot be used for analysing
unvoiced polyphonic music. Second, it is not capable, All the algorithms discussed in the previous section
as described, of finding transposed repetitions. Third, are based on the assumption that the music is rep-
it has a worst-case running time of O(n4 ) which means resented either as a 1-dimensional string of symbols
it is too slow to be used for analysing large pieces. Fi- or, in the case of polyphonic music, as a set of such
nally, it only finds repeated factors and therefore can- symbol strings and this assumption seems to be the
not find patterns ‘with gaps’. That is, it can only find cause of many of their short-comings. For example,
the repetitions of a pattern if the pattern contains all the fact that they process symbol strings means that
the notes in the piece that occur during the time inter- these algorithms cannot deal with unvoiced polyphonic
val spanned by the pattern. music such as keyboard music. Their string-matching
basis also causes problems when it comes to finding
patterns that are distributed between several voices or
Cambouropoulos’s (1998) General Computational finding transposed occurrences of polyphonic patterns
Theory of Musical Structure also contains a pattern with gaps.
discovery component, that, in the most recent incar- We have avoided these problems by adopting a geo-
nation of the theory, is based on Crochemore’s (1981) metric approach in which the music is represented as a
‘set partitioning’ algorithm. In Cambouropoulos’s the- multidimensional dataset. A multidimensional dataset
ory, this pattern-discovery algorithm is used to help is just a finite set of position vectors or datapoints in
with determining the boundaries of the segments that a Euclidean space with a finite number of dimensions.
are then categorised. Crochemore’s algorithm is very Our algorithms work with datasets of any dimension-
fast—it runs in O(n log2 n) time. However, the algo- ality and any size. Also the co-ordinates may take real
rithm does suffer from a couple of short-comings: first, values.
it cannot find patterns with gaps; and, second, it can-
There are many possible appropriate ways of repre-
not be used for finding patterns in unvoiced polyphonic
senting a piece of music as a multidimensional dataset.
music.
For example, Figure 9 shows a 5-dimensional dataset
that represents the score in Figure 8. Each datapoint
In a recent approach described by Conklin and Anag- in Figure 9 represents a single note event in the score.
nostopoulou (2001), a number of different string rep- The co-ordinate values in each datapoint represent on-
resentations each representing a different ‘viewpoint’ set time (in semiquavers), chromatic pitch (Meredith,
on the music, are derived from a rich representation of 1999, 2001) (essentially MIDI number—a number indi-
the music to be analysed. They then discover repeated cating the key on a normal piano keyboard that would
factors in these various string representations and iso- have to be pressed to produce a tone with that pitch),
late those factors that occur most frequently. Their morphetic pitch (Meredith, 1999, 2001) (essentially the
approach is interesting because it offers a crude way of same as Brinkman’s (1986) ‘continuous name code’),
identifying repeated patterns at higher structural levels duration and voice respectively.
than the musical surface—one of their viewpoints, for We can then consider various orthogonal projections
example, represents just the first note in each crotchet of such a dataset. For example, we could just consider
beat. However, such an approach would not be capa- the first two dimensions in the dataset in Figure 9 and
ble of finding the example shown in Figure 7 because get the projection in Figure 10 which tells us the chro-
the notes that are common to both occurrences of the matic pitch and onset time of each note.
pattern do not all fall on strong beats. There is a mul- If we consider just the first and third dimensions
titude of string-processing algorithms available for dis- in Figure 9 we get the two-dimensional graph of mor-
covering repeated factors in strings. However, there are phetic pitch (diatonic pitch) against onset time shown
far fewer algorithms available for finding repeated sub- in Figure 11. Note that some of the patterns that were
sequences (i.e. patterns with gaps) and most of these only similar in Figure 10 are now identical because we
seem to be NP-complete. are using a representation of diatonic pitch. It is often
 Meredith, Lemström and Wiggins, Discovering Repeated Pattern in Polyphonic Music

× ×
b e
3

× ×
d f
2
y

× ×
a c
1

0
0 1 2 3
x

Figure 12: A simple dataset.

Figure 10: Two-dimensional projection of dataset in
Figure 9 showing chromatic pitch against onset-time.
From
a b c d e f

1, 1
1, 3
2, 1
2, 2
2, 3
3, 2
6 6 6 6 6
a =
1, 1
b =
1, 3
0, 2
c =
2, 1
1, 0
1, −2
To
d =
2, 2
1, 1
1, −1
0, 1
e =
2, 3
1, 2
1, 0
0, 2
0, 1
f =
3, 2
2, 1
2, −1
1, 1
1, 0
1, −1

Figure 11: Two-dimensional projection of dataset in

Figure 9 showing morphetic pitch against onset-time. Figure 13: Vector table for dataset in Figure 12.

more profitable when analysing tonal music to look for say that a pattern is translatable by a vector if it can
exact repetitions in this type of projection than in the be translated by the vector to give another pattern in
chromatic-pitch-against-onset-time representation. the dataset. And we say that the maximal translatable
Adopting this geometric approach allows us to find pattern or MTP for a vector is the largest pattern that
classes of perceptually significant musical repetition can be translated by the vector to give another pattern
that are very difficult to compute using string-based in the dataset. SIA discovers all the non-empty MTPs
approaches. It also allows us to process polyphonic in a dataset.
music as simply and efficiently as monophonic music. The first step in SIA is to sort the dataset to be
It dispenses with the need for multiple representations analysed. Then the algorithm constructs a vector table
because we can run the same repetition discovery al- for the dataset like the one in Figure 13 which shows
gorithms on various orthogonal projections of a single, the vector table constructed by SIA for the dataset in
rich multidimensional dataset representation. It also Figure 12. A cell in the vector table contains the vector
allows us to discover repetitions in the dynamic, tim- from the datapoint at the head of the column of that
bre and rhythmic structure of a piece as well as its pitch cell to the datapoint at the head of the row for that
structure. cell.
SIA computes all the values in the vector table below
the leading diagonal as shown in Figure 13. In other
6 SIA: Discovering maximal re- words, it computes for each datapoint all the vectors
from that datapoint to every other datapoint in the
peated patterns in multidi- dataset greater than it. Note that each of the vectors
mensional datasets in the vector table (Figure 13) is stored with a pointer
that points back to the “origin” datapoint for which it
SIA takes a multidimensional dataset as input and was computed (that is, the datapoint at the top of its
finds for every possible vector the largest pattern in column).
the dataset that can be translated by that vector to SIA then simply sorts the vectors in the vector ta-
give another pattern in the dataset. For example, if ble using a slightly modified version of merge sort that
we consider the dataset in Figure 12, then the largest takes advantage of the fact that the columns in this
pattern that can be translated by the vector 1, 0 is table are already sorted. This results in a list like the
the pattern {a, b, d} and the largest pattern that can be one shown in Figure 14 which shows the result of sort-
translated by the vector 1, 1 is the pattern {a, c}. We ing the vectors in the vector table in Figure 13. Note
 Meredith, Lemström and Wiggins, Discovering Repeated Pattern in Polyphonic Music

Vector Datapoint currences of any given pattern within the dataset. So,

0, 1 -
2, 1 for the dataset in Figure 12, SIATEC would actually
compute the complete vector table as shown in Fig-

0, 1 -
2, 2
ure 15 and would then use the region below the lead-

0, 2 -
1, 1 ing diagonal in this table to compute the MTPs as in

0, 2 -
2, 1 SIA. Before computing this vector table, the dataset is
sorted so that the vectors increase as one descends a

1, −2 -
1, 3
column and decrease as one moves from left to right

1, −1 -
1, 3 along a row.

1, −1 -
2, 3 We know that a given column in the vector table (see
Figure 15) contains all the vectors that the datapoint

1, 0 -
1, 1
at the top of the column can be translated by to give

1, 0 -
1, 3 another point in the dataset. Say we want to find all

1, 0 -
2, 2 the occurrences of the pattern {a, c} in Figure 12 which

1, 1 -
1, 1 is the maximal translatable pattern in this dataset for
the vector 1, 1. When we say that we want to “find

1, 1 -
2, 1
all the occurrences” of a pattern, all we actually need

1, 2 -
1, 1 to find is all the vectors that the pattern is translatable

2, −1 -
1, 3 by. So, for example, the pattern {a, c} is only translat-
able by the vectors 1, 1 and 0, 2. We know that the

2, 1 -
1, 1
column of vectors under a contains all the vectors that
the point a can be translated by; and we know that the
Figure 14: The vectors in the vector table in Figure 13 column under c contains all the vectors that the point
sorted into increasing order. c can be translated by. So we know that the pattern
{a, c} can only be translated by the vectors that occur
in both of these columns. In other words, to find the
that each vector in this list is still linked to the dat-
set of occurrences for a given pattern we simply have
apoint at the head of its column in the vector table.
to find the intersection set of the columns headed by
Simply reading off all the datapoints attached to the
the datapoints in the pattern. By exploiting the or-
adjacent occurrences of a given vector in this list gives
deredness of this table, we can find all the occurrences
us the maximal translatable pattern for that vector.
of a k-dimensional pattern of size m in a dataset of size
The complete set of non-empty maximal translatable
n in a worst-case running time of O(kmn). We know
patterns can be obtained simply by scanning the list
that the complete set of maximal translatable patterns
once, reading off the attached datapoints and start-
is found by SIA simply by sorting the vectors below the
ing a new pattern each time the vector changes. Each
leading diagonal in the vector table. If there are l such
box in the right-hand column of the list corresponds to
patterns and mi is the size of the ith pattern then this
a maximal translatable pattern. The most expensive
implies
step in this process is sorting the vectors which can be

l
n(n − 1)
done in a worst-case running time of O(kn2 log2 n) for a mi ≤ .
k-dimensional dataset of size n. The space complexity i=1
2
of this version of the algorithm is O(kn2 ). However,
So the overall worst-case running time of SIATEC is
Ukkonen et al. (2003) describe an implementation of
the algorithm that uses only O(n) working space.  l   2 

kn (n − 1)
O kmi n ≤ O .
i=1
2
7 SIATEC: Discovering all the oc-
The worst-case time complexity of the algorithm is
currences for each maximal therefore O(kn3 ) for a k-dimensional dataset of size n.
translatable pattern The space complexity of the algorithm is O(kn2 ) if one
stores the complete vector table. However, if one uses
SIATEC first generates all the maximal translatable pat- the techniqe described by Ukkonen et al. (2003), this
terns using a slightly modified version of SIA and then worst-case space complexity can be reduced to O(kn).
it finds all the occurrences of each MTP.
As explained in the previous section, SIA only com-
putes the vectors below the leading diagonal in the vec- 8 Running SIA and SIATEC on
tor table (see Figure 13). This is because the maximal music data
translatable pattern for a vector −v is the same as the
pattern that you get by translating the maximal trans- SIA and SIATEC were implemented in C and run on a
latable pattern for v by the vector v itself. However, 500MHz Sparc machine on 52 datasets ranging in size
it turns out that by computing all the vectors in the from 6 to 3500 datapoints and in dimensionality from
vector table we can more efficiently discover all the oc- 2 to 5 dimensions. The graph in Figure 16 shows the
 Meredith, Lemström and Wiggins, Discovering Repeated Pattern in Polyphonic Music

From
a =
1, 1 b =
1, 3 c =
2, 1 d =
2, 2 e =
2, 3 f =
3, 2

a =
1, 1
0, 0
0, −2
−1, 0
−1, −1
−1, −2
−2, −1
b =
1, 3
0, 2
0, 0
−1, 2
−1, 1
−1, 0
−2, 1
c =
2, 1
1, 0
1, −2
0, 0
0, −1
0, −2
−1, −1
To d =
2, 2
1, 1
1, −1
0, 1
0, 0
0, −1
−1, 0
e =
2, 3
1, 2
1, 0
0, 2
0, 1
0, 0
−1, 1
f =
3, 2
2, 1
2, −1
1, 1
1, 0
1, −1
0, 0

Figure 15: Vector table computed by SIATEC for dataset in Figure 12.

Pitch 6

s

A

A

As s

A


A
AA

s
sA sA
-
Time

Figure 16: Running time of SIA on 2-dimensional Figure 18: A dataset with three occurrences of a pat-
datasets. tern indicated.

Pitch 6

As

A

s sA

As
As

A
A

s sA
s sA
-
Time

Figure 17: Running time of SIATEC on 2-dimensional

datasets. Figure 19: A dataset with three occurrences of a pat-
tern indicated.

running time of SIA for the 2-dimensional datasets in

the sample. The smooth curve in Figure 16 represents ten either equal to the maximal translatable patterns
a running time of kn2 log2 n. The graph in Figure 17 generated by SIA or straightforwardly derivable from
shows the running time of SIATEC on the same machine them. Nevertheless, only a very small proportion of the
for the 2-dimensional datasets in the sample. In this patterns generated by SIA would be considered musi-
graph, the smooth curve represents a running time of cally interesting by an analyst or expert listener. For
kn3 . As can be seen from these two graphs, it took less example, SIA discovers over 70000 MTPs in Rachmani-
than 2 minutes for SIA to process a piece containing noff’s Prelude, Op.3 No.2 (see Figure 6) and probably
3500 notes and about 13 minutes for SIATEC to process fewer than 100 of these would be considered interesting
a piece containing 2000 notes. by a music analyst. This means that we need to devise
systems that evaluate the output of SIA and SIATEC
and isolate various classes of musically interesting rep-
9 Isolating significant repeti- etitions.
tions How can we isolate the theme-like and motive-like
patterns in a passage? Some feasible heuristics for do-
A dataset of size n contains 2n distinct subsets and the
2
ing this will now be described.
number of patterns generated by SIA is less than n2 . First of all, it seems useful to define the coverage
Therefore, for all but the smallest datasets, SIA gen- of a pattern to be the number of datapoints in the
erates only a tiny fraction of all patterns in a dataset. dataset that are members of occurrences of the pat-
Our experiments suggest that the repeated patterns tern. For example, in Figure 18 the coverage of the tri-
that we’re interested in (including many that are very angular pattern is 6, but in Figure 19, the coverage of
hard to find using string-matching techniques) are of- the triangular pattern is 9. Note that coverage is gen-
 Meredith, Lemström and Wiggins, Discovering Repeated Pattern in Polyphonic Music

pitch-against-onset-time graph of the piece. If we use

Pitch6 × u × the bounding box, as in Figure 21 then the compactness
× of the pattern consisting of the points marked by dots
4
u u would be 10 = 25 . However, if we use the convex hull,
as shown in Figure 22 then the compactness value of
× the same pattern in the same dataset would be 46 = 23 .
× u ×
× ×- Typically, at least one occurrence of a theme-like pat-
tern will have a high compactness value, even if the
Time
other occurrences are highly embellished. Another in-
teresting heuristic that seems to be useful for isolating
theme-like patterns is the compression ratio that can
Figure 20: Defining ‘the region spanned by a pattern’
be achieved by representing the set of points covered
to be the time segment spanned by the pattern.
by all occurrences of the pattern by specifying simply
one occurrence of the pattern and all the vectors by
u which the pattern can be translated. For example, in
Pitch6 × × Figure 18 the set of points covered by the occurrences
× of the triangular pattern can be represented by speci-
u u fying the points in one occurrence of the pattern and
× the two translation vectors that map that occurrence
× u × onto the other two occurrences of the pattern in this
× ×- dataset. That is, we can represent the 6 datapoints
Time in Figure 18 using 3 position vectors and 2 translation
vectors, thus achieving a compression ratio of 65 . If we
represent the dataset in Figure 19 in the same way we
Figure 21: Defining ‘the region spanned by a pattern’ can achieve a compression ratio of 95 . In general, it
to be the bounding-box spanned by the pattern. seems that a high compression ratio can be achieved if
we represent the set of points covered by all the occur-
rences of an important thematic pattern by specifying
erally greater for patterns whose occurrences overlap just one occurrence of the pattern and all the vectors
less. It is also generally greater for larger patterns and by which the pattern can be translated. In the next
for those that occur more often. In general, it seems section we use this idea of compression ratio in a com-
that the most theme-like and motive-like patterns in pression algorithm based on SIATEC that can be used
music have relatively high coverage. for generating analyses of pieces that in some cases re-
Next it seems useful to define the compactness of a semble the thematic/motivic analyses carried out by
pattern to be the ratio of the number of points in the human music analysts.
pattern to the total number of points in the dataset
that occur within the region spanned by the pattern
within a particular representation. One can define ‘the 10 COSIATEC: a compression algo-
region spanned by a pattern’ in a number of different
ways. For example, we could define it to be the smallest
rithm based on SIATEC
time segment that contains the pattern, as illustrated
COSIATEC is a compression algorithm based on
in Figure 20. If we define it in this way, then the pat-
SIATEC. The flow-chart in Figure 23 describes how
tern consisting of the four round points in Figure 20it works. First we run SIATEC on the dataset
4
has a compactness value of 12 = 13 . Alternatively, one
to be compressed. This generates a list of
could define the region spanned by a pattern to be thePATTERN, TRANSLATOR SET pairs. The trans-
bounding-box or the convex hull of the pattern in thelator set for each pattern contains all the vectors by
which the pattern is translatable within the dataset
apart from the zero vector. In general, this gives a
more efficient representation of the set of points covered
Pitch6 × u ×

QQ by the occurrences of the pattern. Then the heuris-

×Q tics described in the previous section—compression ra-
u

Q
Qu
A  tio, coverage and compactness—are used to choose the
× 
A  ‘best’ pattern P and this pattern is printed out together
× Au
 × with its translator set. Then all the points covered by
× ×- P —that is all the points that are members of occur-
Time rences of P —are removed from the dataset. Then if
the dataset is empty the algorithm terminates. How-
ever, if there are still datapoints in the dataset, SIATEC
Figure 22: Defining ‘the region spanned by a pattern’ is again run on the remaining points and the cycle is
to be the convex hull spanned by the pattern. repeated. The result is a print out of the ‘best’ pat-
 Meredith, Lemström and Wiggins, Discovering Repeated Pattern in Polyphonic Music

START

? 
DATASET 


? Figure 26: Pattern discovered on first iteration of
SIATEC COSIATEC when run on Bach’s Two-Part Invention in
 ?  D major, BWV 774.
LIST of
PATTERN, TRANSLATOR SET PAIRS


? 6
PRINT OUT BEST PATTERN P AND ITS TRANSLATORS

?
REMOVE OCCURRENCES OF P FROM DATASET
Figure 27: Pattern discovered on second iteration of
? 
IS DATASET EMPTY? - NO COSIATEC when run on Bach’s Two-Part Invention in

D major, BWV 774.
? 
YES


?
FINISH two patterns are precisely the ones identified by Boyd
(1983, p. 96) to be the two most important motivic
patterns in this piece.
Figure 23: COSIATEC When COSIATEC is run on the Two-Part Invention in
D major, BWV 774, the pattern that emerges on the
first iteration consists of the anacrusis and first bar of
the subject (Figure 26) and the pattern that is found
on the second iteration consists of the second part of
the subject (Figure 27). Similarly, when COSIATEC is
Figure 24: Pattern discovered on first iteration of run on the Invention in D minor, BWV 775, the sub-
COSIATEC when run on Bach’s Two-Part Invention in ject of the invention is generated on the first iteration
C major, BWV 772. (Figure 28).

tern and its translators for each iteration of the cycle,

and this printout is, in general, a compressed represen-
tation of the input dataset. Obviously, the degree of
compression achieved depends directly on the amount
of repetition in the dataset.
COSIATEC has been run on morphetic-pitch- Figure 28: Pattern discovered on first iteration of
against-onset-time representations of all 15 of Bach’s COSIATEC when run on Bach’s Two-Part Invention in
Two-part Inventions and the results are quite encour- D minor, BWV 775.
aging. In particular, it seems that the patterns that
achieve the highest compression ratios on the early iter-
1 6 × ×d
y
ations of COSIATEC quite often correspond to the most b
important themes and motives in the music.
0× ×c -
a
For example, Figures 24 and 25 show the patterns 0 1 2 3
discovered by COSIATEC on the first and second iter- x
ations, respectively, when the algorithm is run on the
Two-Part Invention in C major, BWV 772. The pat- Figure 29: Example input query pattern for SIAMESE.
tern in Figure 24 is the inversion of the subject and
the pattern in Figure 25 is the subject itself. These
y3
6 ×h
2 ×f ×i
×e ×
g
1

0 -
0 1 2 3 4
Figure 25: Pattern discovered on second iteration of x
COSIATEC when run on Bach’s Two-Part Invention in
C major, BWV 772. Figure 30: Example input dataset for SIAMESE.
 Meredith, Lemström and Wiggins, Discovering Repeated Pattern in Polyphonic Music

11 SIAMESE: Pattern matching in VECTOR DATAPOINT

multidimensional datasets
−2, 0 -
3, 1

−1, 1 -
2, 0
We have also developed a pattern-matching algorithm
based on SIA which we call SIAMESE. This algorithm
−1, 1 -
3, 1

takes a multidimensional query pattern and a multi-
0, 0 -
1, 1

dimensional dataset as input and finds all exact com-
0, 0 -
3, 1
plete and partial matches of the query pattern in the
0, 2 -
2, 0
dataset. For example, if we run SIAMESE on the query -

0, 2
3, 1
pattern in Figure 29 and the dataset in Figure 30 then

1, 1 -
0, 0
the output will tell us (amongst other things) that the
complete query pattern {a, b, c, d} can be matched to
1, 1 -
1, 1
the pattern {e, f, g, i} in the dataset. It will also tell us
1, 1 -
2, 0
that the three point pattern {a, b, c} in the query can
1, 1 -
3, 1
be matched to the pattern {f, h, i} in the dataset, that
1, 3 -
2, 0
the points {c, d} can be matched to {e, f } and {f, h}

2, 0 -
1, 1
and so on.
SIAMESE works in essentially the same way as SIA.
2, 2 -
0, 0

We begin by sorting the points in the query and the
2, 2 -
1, 1

2, 2 -
2, 0
From
a c
3, 1 -
0, 0
b d

0, 0
1, 1
2, 0
3, 1
3, 1 -
1, 1
6 6 6 6
3, 3 -
0, 0
e =
1, 1
1, 1
0, 0
−1, 1
−2, 0
4, 2 -
0, 0
f =
2, 2
2, 2
1, 1
0, 2
−1, 1

To g =
3, 1
3, 1
2, 0
1, 1
0, 0 Figure 32: List of sorted vectors generated by SIAMESE
h =
3, 3
3, 3
2, 2
1, 3
0, 2 for the query pattern in Figure 29 and the dataset in
i =
4, 2
4, 2
3, 1
2, 2
1, 1 Figure 30.

Figure 31: The vector table computer by SIAMESE for matched which tells us that a complete occurrence of
the query patter in Figure 29 and the dataset in Fig- the query pattern occurs at a displacement of 1, 1.
ure 30. Similarly, if we look at the datapoints attached to the
consecutive occurrences of the vector 2, 2 in this list,
points in the dataset and then we construct a vector we find that the points {a, b, c} are matched when the
table like the one in Figure 31 which shows the vector query is translated by this vector.
table constructed for the query pattern in Figure 29
The most expensive step in this process is sorting
and the dataset in Figure 30. Each entry in this table
the vectors in the vector table. Using a comparison
gives the vector from the query datapoint at the head
sort such as merge sort, this step can be achieved
of the colum in which the entry occurs to the dataset
in a worst-case running time of O(knm log2 (mn)) for
point at the head of the row in which the entry occurs.
a k-dimensional query pattern of size m and a k-
Note that as in SIA, each vector in the vector table
dimensional dataset of size n.
has a pointer that points back to the query pattern
datapoint at the head of the column in which it occurs.
Having constructed this table, we then simply sort
all the vectors in it to give a list like the one in Figure 32 12 Possible directions for fur-
which shows the list that results when the vectors in ther work
the table in Figure 31 are sorted. This list gives us
all the vectors that we can translate the query pattern We finish by suggesting some directions in which fur-
by to give a non-empty match in the dataset. The ther work could be carried out.
fact that each of these vectors still has a pointer to the
query pattern datapoint at the head of its column in • Versions of SIA and SIATEC, COSIATEC and
the vector table means that, for each vector, we can SIAMESE need to be developed that discover ap-
simply read off the points in the query pattern that proximate repetitions (see Ukkonen et al., 2003).
have matches in the dataset when the query pattern
is translated by that vector. For example, if we look • Algorithms (possibly based on SIA) could be devel-
at the query pattern datapoints that are pointed to oped for discovering repetitions where the patterns
by vectors with the value 1, 1 in Figure 32, we find are related by rotation, reflection or dilatation (en-
that all four of the points in the query pattern are largement).
 Meredith, Lemström and Wiggins, Discovering Repeated Pattern in Polyphonic Music

• The running times of SIA and SIATEC could be im- Meredith, D. (2001). MIPS: A formal language
proved by using word parallelism or by designing for the mathematical investigation of pitch sys-
PRAM versions of the algorithms. tems (version 2001-09-10). Available online at
http://www.titanmusic.com/papers.html .
• Further heuristics and algorithms could be devel-
oped for isolating various classes of perceptually Meredith, D., Lemström, K., and Wiggins, G. A.
significant repetition. (2002a). Algorithms for discovering repeated
patterns in multidimensional representations of
• Applications of the algorithms could be developed
polyphonic music. Journal of New Music Re-
for use in specific domains (e.g., music, images,
search. To appear. (Draft available online at
video, bioinformatics):
http://www.titanmusic.com/papers.html).
– SIA, SIATEC and COSIATEC can be used for
Meredith, D., Wiggins, G. A., and Lemström, K.
∗ data compression, (2002b). Method of pattern discovery. PCT patent
∗ database indexing and application number PCT/GB02/02430, UK patent
∗ data mining. application number 0211914.7. Applied for by City
– SIAMESE can be used for University, London and filed on 23 May 2002. (Pri-
ority date: 23 May 2001, draft available online at
∗ information retrieval and http://www.titanmusic.com/papers.html).
∗ computer-based learning systems.
Nattiez, J.-J. (1975). Fondements d’une sémiologie de
la musique. Union Générale d’Éditions, Paris.
References
Rolland, P.-Y. (1999). Discovering patterns in musical
Bent, I. and Drabkin, W. (1987). Analysis. New Grove sequences. Journal of New Music Research, 28(4),
Handbooks in Music. Macmillan. 334–350.
Boyd, M. (1983). Bach. The Master Musicians. J. M. Ruwet, N. (1972). Langage, Musique, Poésie. Éditions
Dent, London. du seuil, 27, rue Jacob, Paris VI.
Brinkman, A. R. (1986). A binomial representation of Schenker, H. (1954). Harmony. University of Chicago
pitch for computer processing of musical data. Music Press, London. Edited by Oswald Jonas and trans-
Theory Spectrum, 8, 44–57. lated by Elisabeth Mann Borgese from the 1906 Ger-
man edition.
Cambouropoulos, E. (1998). Towards a General Com-
putational Theory of Musical Structure. Ph.D. the- Temperley, D. (2001). The Cognition of Basic Musical
sis, University of Edinburgh. Structures. MIT Press, Cambridge, MA.
Conklin, D. and Anagnostopoulou, C. (2001). Repre- Ukkonen, E., Lemström, K., and Mäkinen, V. (2003).
sentation and discovery of multiple viewpoint pat- Sweepline the music! In R. Klein, H.-W. Six, and
terns. In Proceedings of the International Computer L. Wegner, editors, Computer Science in Perspective
Music Conference, 2001, Havana Cuba. (LNCS 2598), pages pp. 330–342.
Crochemore, M. (1981). An optimal algorithm for com-
puting the repetitions in a word. Information Pro-
cessing Letters, 12(5), 244–250.
Forte, A. and Gilbert, S. E. (1982). Introduction to
Schenkerian Analysis. Norton, New York.
Hsu, J.-L., Liu, C.-C., and Chen, A. L. (1998). Effi-
cient repeating pattern finding in music databases.
In Proceedings of the 1998 ACM 7th International
Conference on Information and Knowledge Manage-
ment, pages 281–288. Association of Computing Ma-
chinery.
Lerdahl, F. and Jackendoff, R. (1983). A Generative
Theory of Tonal Music. MIT Press, Cambridge, MA.
Meredith, D. (1999). The computational repre-
sentation of octave equivalence in the Western
staff notation system. In Cambridge Music Process-
ing Colloquium, September 1999. Available online at
http://www-sigproc.eng.cam.ac.uk/music proc/submissions/ .

View publication stats