Brain-Computer Music Interface For Composition and Performance
Brain-Computer Music Interface For Composition and Performance
net/publication/228750904
CITATIONS READS
37 246
1 author:
SEE PROFILE
Some of the authors of this publication are also working on these related projects:
All content following this page was uploaded by Eduardo Reck Miranda on 30 March 2014.
University of Plymouth, Interdisciplinary Centre for Computer Music Research (ICCMR), Drake Circus, Plymouth, United
Kingdom
Abstract: This paper introduces a new brain-computer interface (BCI) system that uses electroencephalogram (EEG) information to steer
generative rules to compose and perform music. It starts by noting the various attempts at the design of BCI systems, including systems for
music. Then it presents a short technical introduction to EEG sensing and analysis. Next, it introduces the generative music component of the
system, which employs our own adapted version of a machine-learning technique based on ATNs (Augmented Transition Networks) for the
computer-replication of musical styles. The system constantly monitors the EEG of the subject and activates generative rules associated with
the activity of different frequency bands of the spectrum of the EEG signal. The system also measures the complexity of the EEG signal to
modulate the tempo (beat) and dynamics (loudness) of the performance. Subjects can be trained to select between different classes of
generative rules to produce original pieces of music.
Keywords: assistive music technology, brain-computer interface, generative music systems, bio-signal music controller, United Kingdom
Correspondence: Eduardo Reck Miranda, MSc, PhD, Professor of Computer Music, University of Plymouth, SoCCE, B326 Portland Square,
Drake Circus, Plymouth PL4 8AA, United Kingdom. E-mail: [email protected]
Submitted: February 02, 2006. Revised: March 01, 2006. Accepted: March 05, 2006.
INTRODUCTION research in this area did not take off until the early 1990s,
Research into brain-computer interface (BCI) for music is probably due to technological limitations. In 1990, Jonathan
an interesting arena for the development of new possibilities Wolpaw and colleagues developed a system to allow some
in recreational and therapeutic devices for people with control of a computer cursor by individuals with severe
physical and neurological disabilities. We shall refer to such motor deficits. Users were trained to use aspects of their
systems as brain-computer music interfaces (BCMI). EEG to move a cursor on a computer screen (3). In 1998,
Various music-making devices are available for those with Christoph Guger and Gert Pfurtscheller reported impressive
disabilities, and even though such devices have proved to advances in BCI research: an EEG-based system to control
work very effectively, they often do not allow as much a prosthetic hand (4). Many attempts followed
control for those with severe physical disabilities. At
present, access music tutors and therapists use gesture
devices and adapted accessible technology to make this
possible, which achieve excellent results. Yet, for people
with severe physical disabilities, having complete control of
the environment created for them by the facilitator or
therapist can sometimes be difficult. This paper introduces a
BCMI that uses the EEG to steer generative musical rules to
compose and perform music on a MIDI-controlled
mechanical acoustic piano (Figure 1).
Human brainwaves were first measured in 1924 by Hans
Berger, who termed these measured brain electrical signals
the electroencephalogram, literally "brain electricity
writing" (1). Today, the EEG has become one of the most
useful tools in the diagnosis of epilepsy and other
neurological disorders. Further, the fact that a machine can
read signals from the brain has sparked the imaginations of
scientists, artists and other enthusiasts, and EEG has made
its way into applications other than strictly clinical ones. In
the early 1970s, Jacques Vidal did the first tentative work
towards a brain-computer interface (BCI) system. The Fig. 1: The BCMI system in action. Note the keys of the piano
results of this work were published in 1973 (2). Despite a being played by the system; no hands or any kind of
number of isolated initiatives at building BCI systems, physical movement are needed.
00
62 EDUARDO RECK MIRANDA
(ERP), principal component analysis (PCI) and correlation, interesting because it represents each time step (or window)
to cite but a few. Brief non-mathematical introductions to using only these three attributes, and this is done without
EEG power spectrum and Hjorth analyses are given below conventional frequency domain description. The signal is
due to their relevance to the system presented in this paper. measured for successive epochs (or windows) of one to
Although powerful mathematical tools for analysing the several seconds. Two attributes are obtained from the first
EEG already exist, we still lack a good understanding of and second time derivatives of the amplitude fluctuations in
their analytical semantics in relation to musical cognition. the signal. The first derivative is the rate of change of the
However, continual progress in the field of Neuroscience of signal’s amplitude. At peaks and troughs, the first derivative
Music (12) is substantially improving this scenario. Once is zero; at other points, it will be positive or negative
these issues are better understood we will probably be able depending on whether the amplitude increases or decreases
to recognize sophisticated patterns of cognitive activity in with time. The steeper the slope of the wave, the greater
the brainwaves and activate appropriate musical algorithms will be the amplitude of the first derivative. The second
associated with such patterns. Our preliminary work in this derivative is determined by taking the first derivative of the
regard has been reported in (11,13). first derivative of the signal. Peaks and troughs in the first
Power spectrum analysis is derived from techniques of derivative, which correspond to points of greatest slope in
Fourier analysis, such as the Discrete Fourier Transform the original signal, result in zero amplitude in the second
(DFT). In short, DFT analysis breaks the EEG signal into derivative, and so forth.
different frequency bands and reveals the distribution of Activity is the variance of the amplitude fluctuations in
power between them. This technique is useful because the the epoch. Mobility is calculated by taking the square root of
distribution of power in the spectrum of the EEG reflects the variance of the first derivative divided by the variance of
distinct “mental states”. In general, a signal characterized by the primary signal. Complexity is the ratio of the mobility of
low-frequency components (e.g., lower than 8 Hz) may be the first derivative of the signal to the mobility of the signal
associated with a state of drowsiness, whereas a signal itself. There is no clear agreement as to what these
characterized by high-frequency components (higher than measurements mean in terms of cognition. It is common
14 Hz, up to 30 Hz or so) may be associated with a state of sense to assume that the longer a subject remains focused on
alertness. When the cortex is most actively engaged in a specific mental state, the more stable the signal is, and
processing information, whether generated by sensory input therefore the variance of the amplitude fluctuation is lower.
or by some internal process, the activity level of cortical
neurons is relatively high, but also relatively unsynchro- THE GENERATIVE MUSIC ENGINE
nized. By contrast, when the cortex is less actively engaged Our system features our own adapted version of a machine-
in processing information, the activity level of the cortical learning algorithm based on Augmented Transition
neurons is relatively low, but also relatively synchronized. Networks (ATNs) that learns generative music rules from
The higher the synchronization of the neurons, the higher given examples (i.e., musical scores in machine-readable
will be the overall amplitude of the EEG (14). Four format), originally developed by David Cope (17). Due to
recognized frequency bands of EEG activity, also referred limited availability of space, this paper focuses only on the
to as EEG rhythms, are associated with different mental functioning of the generative component of the system.
states in healthy adult brains, as shown in Table 1 (15). The EEG signal can influence in a well-defined way the
mixture of different style-elements found in the different
Table 1: Electroencephalographic rhythms. musical examples given to train the system. The system can
generate music that contains, for example, more Schumann-
like elements when the spectrum of the subject’s EEG is
Rhythm Bandwidth Meaning
characterized by alpha rhythms or more Beethoven-like
Delta Lower than 4 Hz Sleep (non-dreaming) elements when the spectrum of the EEG is characterized by
Theta Between 4 Hz - 7 Drowsiness beta rhythms; these associations are arbitrary.
Hz Example-based musical-generation systems are often
based on formalisms such as transition networks or Markov
Alpha Between 8 Hz - 13 Relaxed; aware but with eyes Chains to re-create the transition-logic of what-follows-
Hz closed what, at the level of notes and at the level of similar
Beta Higher than 13 Hz Awake, alertness, intense “vertical slices” of music. The act of recombining the
mental activity building blocks of music material together with some
typical patterns and structural methods has proved to have
great musical potential (17). Self-learning predictors of
Hjorth introduced an interesting method for clinical musical elements based on previous musical elements can
EEG analysis, which measures three attributes of the signal: be used at any level or for any type of musical element such
activity, mobility, and complexity (16). Essentially, Hjorth as musical note, chord, bar, phrase, section, and so on. The
analysis is a time-based amplitude analysis. This method is current version of the system uses a simple statistical
64 EDUARDO RECK MIRANDA
Table 2: Excerpt from a database of musical elements where: CO = composer (SCHU = Schumann.), P-CLASS = pitch class, P
= pitch, PCL = pitch-class leading, PL = pitch leading and TYPE = type.
ID SCHU-1-1-CAD ID SCHU-1-1-MEA-6
CO SCHU CO SCHU
P-CLASS ( (0 2 7) (0 2 4 5 7 11) ) P-CLASS ( (5 9) (0 5 7 9) )
P 74 P 81
PCL ( (0 4 9) (0 2 4 5 7 9 11) ) PCL ( (0 2 7) (0 2 4 5 7 11) )
PL 76 PL 74
TYPE CAD TYPE BAR
ID SCHU-1-1-MEA-1 ID SCHU-1-1-MEA-4
CO SCHU CO SCHU
P-CLASS ( (0 4) (0 3 4 6 7 9) ) P-CLASS ( (0 4) (0 3 4 6 7 9) )
P 76 P 83
PCL ( (2 7 11) (2 5 7 9 11) ) PCL ( (0 4) (0 4 7) )
PL 83 PL 76
TYPE INC TYPE BAR
ID SCHU-1-1-MEA-3 ID SCHU-1-1-MEA-2
CO SCHU CO SCHU
P-CLASS ( (0 4) (0 3 4 6 7 9) ) P-CLASS ( (2 7 11) (2 5 7 9 11) )
P 76 P 83
PCL ( (2 7 11) (2 5 7 9 11) ) PCL ( (0 4) (0 3 4 6 7 9) )
PL 83 PL 76
TYPE BAR TYPE BAR
predictor at the level of short vertical slices of music such as by applying the list of constraints in left-to-right order to the
a bar or half-bar, where the predictive characteristics are set of all musical elements until there are no constraints left,
determined by the chord (harmonic set of pitches, or pitch- or there is only one musical element left. This means that
class) and by the first melodic note following the melodic some of the given constraints might not be applied.
notes in those vertical slices of music. The system uses a The database of all musical elements contains informa-
method for generating short musical phrases with a tion from different composers (or musical styles), with
beginning and an end that allows for real-time steering with elements tagged by their musical function such as “measure
information extracted from the EEG. The system generates 1” for the start of a phrase, “cadence” for the end,
music by defining top-level structures of sequences and “composer” for the composer or musical style, and the
methods of generating similarity- or contrast-relationships special tags “pitch” and “pitch-class” that are both used for
between elements. Consider the following example (LISP- correct melodic and harmonic progression or direction. As
like notation): an example, Table 2 lists excerpts from a database showing
the main attributes that are used to recombine musical
S -> (INC BAR BAR BAR BAR BAR HALF-CADENCE 8BAR-COPY) elements.
P-CLASS (for pitch-class) is a list of two elements. The
From this top-level the system generates rules for selecting first is the list of start-notes, transposed to the range of 0-11.
a valid musical building block for each symbol, including The second is the list of all notes in this element (also
rules for incorporating the EEG information in all decisions. transposed to 0-11). P is the pitch of the first (and highest)
For example: melodic note in this element. By matching this with the
melodic note that the previous element was leading to, the
INC -> ( (EQUAL 'MEASURE-1) (EQUAL 'COMPOSER EEG- system can generate a melodic flow that adheres in some
SET-COMPOSER) ) way to the logic of “where the music wants to go”. The PCL
(for pitch-class leading) elements contain the same infor-
BAR -> ( (CLOSE 'PITCH 'PREV-PITCH-LEADING) mation about the original next bar; this is used to find a
(CLOSE 'PITCH-CLASS 'PREV-PITCH-CLASS-LEADING) possible next bar in the recombination process. Then there
(EQUAL 'COMPOSER EEG-SET-COMPOSER) ) are the INC, BAR, and CAD elements. These are used for
establishing whether those elements can be used for phrase-
This defines a framework to generate a valid sequence with starts (incipient), or cadence. Simply by combining the
a beginning and an end, including real-time EEG control musical elements with the constraint-based selection
through the variable EEG-SET-COMPOSER. The generative process that follows from the terminals of the phrase-
engine will find a musical element for each of the structure rewrite-rules, the system ends up with a generative
constraint-sets that are generated above from INC and BAR, method that can take into account the EEG information.
BRAIN-COMPUTER MUSIC INTERFACE 65
Fig. 4: Spectral information is used to activate generative music ptocesses to compose music in real-time and the signal complexity is used
to control the tempo and dynamics of the performance
The EEG is sensed with seven pairs of gold EEG electronically, which in turn feeds back information about
electrodes on the scalp, forming a bipolar montage: F8-T4, our body or physiological state. This information is often
F4-C4, Fz-Cz, F3-C3, F7-T3, P3-O1, and P4-O2. A displayed through audio-visual stimuli. As a result, the
discussion for the rationale of this configuration falls subject can learn to modify these signals and subsequently
beyond the scope of this paper. It suffices to say that we are learn to gain greater control of the biological signals.
not looking for signals emanating from specific cortical Biofeedback technology has been used to treat and control a
sites; rather, the idea is to sense the EEG over a wide area of number of conditions; examples include migraine headaches
the cortex. The electrodes are plugged into a biosignal and epilepsy. However, the problem of intentional control
amplifier and a real-time acquisition system. The analysis “by thinking” is a difficult one because while EEG analysis
module performs power spectrum and Hjorth analyses in can help us to deduce whether a person is thinking, it is
real-time. It generates two streams of control parameters. unlikely to give us any clear indication of what a person is
One stream contains information about the most prominent actually thinking.
frequency band in the signal and is used to trigger We acknowledge that the music produced by the
generative music processes. The other stream contains system is of limited appeal for those interested in modern
information about the complexity of the signal and is used new music. Furthermore, the pieces produced by our
to control the tempo (beat) and dynamics (loudness) of the computer-replication of musical style engine may not
performance (Figure 4). always sound convincing to discerning listeners. However,
The system is programmed to look for information in we decided to adopt the ATNs-based approach inspired by
the EEG signal and match the findings with assigned the work of David Cope (17) as a starting point for this
generative musical processes corresponding to different research because such technique is well understood and its
musical styles. As mentioned in the previous section, these use in music is well documented. Nevertheless, we are
assignments are arbitrary. The present implementation studying the possibility of using other interesting machine
activates generative processes for two different styles of learning and generative techniques such as those proposed
music (e.g., Schumann-like and Beethoven-like), depending by Assayag and Dubnov (21).
on whether the EEG is characterised by the prominence of An aspect that needs to be addressed in future versions
alpha rhythms or beta rhythms. Every time it has to produce of our system is the non-ergonomic nature of the electrode
a musical bar (or half-bar), it checks the power spectrum of technology for sensing the EEG. The electrodes cap can be
the EEG at that moment and activates the generative uncomfortable and awkward to wear. There are various
processes accordingly. The system is initialized with possibilities for innovations in the hardware design of EEG
reference tempo and amplitude values, which are modulated capture devices. Inexpensive auto-scanning and auto-
by the result of signal complexity analysis (Hjorth analysis). negotiating wireless chips are now available and could be
placed on the head along with the small preamplifiers. Thus,
CONCLUDING REMARKS building wearable EEG amplifiers with built-in signal pro-
To allow for greater intentional control over the system, we cessing and wireless data transmission is technically possible.
are studying ways to develop methods to train subjects to
achieve specific EEG patterns to control musical algorithms. ACKNOWLEDGMENTS
We have initial evidence that this can be made possible The development of this project was possible thanks to
using a technique known as biofeedback. Biofeedback is capital investment funds from the Faculty of Technology of
when biological information or signals are monitored the University of Plymouth, UK. The author would like to
BRAIN-COMPUTER MUSIC INTERFACE 67
thank two former members of his post-graduate research of early experiments. Vancouver, BC, Canada:
team, Andrew Brouse and Bram Boskamp, for their valuable Aesthetic Res Center Can Publ, 1975.
contributions to this project. 10. Rosenboom D. The performing brain. Comp Mus J
1990;14(1):48-65.
11. Miranda ER, Sharman K, Kilborn K, Duncan A. On
REFERENCES harnessing the electroencephalogram for the musical
1. Berger H. Über Das Elektrenkephalogramm Des braincap. Comp Mus J 2003;27(2):80-102.
Menschen. Arch Psychiatrie Nervenkrankheiten 1929; 12. Peretz I, Zatorre RJ. The cognitive neuroscience of
87:527-70. [German] music. Oxford,UK: Oxford Univ Press, 2003.
2. Vidal JJ. Toward direct brain-computer communi- 13. Miranda ER, Roberts S, Stokes M. On generating
cation. Ann Rev Biophysics Bioengineering 1973:2: EEG for controlling musical systems. Biomed Technik
157-80. 2004;49(1):75-6. [German]
3. Wolpaw J, McFarland D, Neat G, Forneris C. An 14. Giannitrapani D. The electrophysiology of intellectual
EEG-based brain-computer interface for cursor control. functions. Basel, Switzerland: Karger, 1985.
Electroencephalogr Clin Neurophysiol 1991:78(3): 15. Misulis KE. Essentials of clinical neurophysiology.
252-9. Boston, MA, USA: Butterworth-Heinemann, 1997.
4. Guger C, Pfurtscheller G. EEG based brain-computer 16. Hjorth B. EEG analysis based on time series properties.
interface (BCI) running in real-time under Windows. Electroencephalogr Clin Neurophysiol 1970;29:306-10.
Electroencephalogr Clin Neurophysiol 1998;106(1): 17. Cope D. Virtual music. Cambridge, MA, USA: MIT
91-100. Press, 2001.
5. IEEE Trans Biomedical Eng 2004;51. 18. Birbaumer N, Ghanayin N, Hinterberger, Iversen I,
6. Adrian ED, Matthews BHC. The Berger rhythm: Kotchoubey B, Kubler A, Perelmeouter J, Taub E,
Potential changes from the occipital lopes in man. Flor H. A spelling device for the paralysed. Nature
Brain 1934;57:355-85. 1999;398:297-8.
7. Teitelbaum R. In tune: Some early experiments in 19. Anderson C, Sijercic Z. Classification of EEG signals
biofeedback music (1966-1974). In: Rosenboom D, from four subjects during five mental tasks. Solving
ed. Biofeedback and the arts. Results of early Engineering Problems with Neural Networks. Conf
experiments. Vancouver, BC, Canada: Aesthetic Res Engineering Applicat Neural Networks (EANN’96),
Center Can Publ, 1975. London, UK, 1996.
8. Lucier A. Statement on: Music for solo performer. In: 20. Petsche H, Etlinger SC. EEG and thinking. Vienna,
Rosenboom D, ed. Biofeedback and the arts. Results Austria: Austrian Acad Sci, 1998.
of early experiments. Vancouver, BC, Canada: 21. Assayag G, Dubnov S. Using factor oracles for
Aesthetic Res Center Can Publ, 1975. machine improvisation. Soft Computing 2004;8(9):
9. Rosenboom R, ed. Biofeedback and the arts. Results 604-10.