ARTICLE IN PRESS

Int. J. Human-Computer Studies 65 (2007) 460–477

www.elsevier.com/locate/ijhcs

Berlin Brain–Computer Interface—The HCI communication

channel for discovery
Roman Krepkia,, Gabriel Curiob, Benjamin Blankertza, Klaus-Robert Müllera,c
a
Research Group for Intelligent Data Analysis (IDA), Fraunhofer Institute for Computer Architecture and Software Technology (FhG-FIRST),
Kekuléstr, 7, D-12489 Berlin, Germany
b
Department of Neurology, Neurophysics Group, Free University of Berlin, Campus Benjamin Franklin, Charité—University Medicine,
Hindenburgdamm 30, D-12200 Berlin, Germany
c
Faculty for Computer Science, Department for Neuroinformatics, University of Potsdam, August-Bebel-Street, 89, D-14482 Potsdam, Germany
Available online 18 January 2007

Abstract

The investigation of innovative Human–Computer Interfaces (HCI) provides a challenge for future interaction research and
development. Brain–Computer Interfaces (BCIs) exploit the ability of human communication and control bypassing the classical
neuromuscular communication channels. In general, BCIs offer a possibility of communication for people with severe neuromuscular
disorders, such as amyotrophic lateral sclerosis (ALS) or complete paralysis of all extremities due to high spinal cord injury. Beyond
medical applications, a BCI conjunction with exciting multimedia applications, e.g., a dexterity discovery, could define a new level of
control possibilities also for healthy customers decoding information directly from the user’s brain, as reflected in EEG signals which are
recorded non-invasively from the scalp.
This contribution introduces the Berlin Brain–Computer Interface (BBCI) and presents set-ups where the user is provided with
intuitive control strategies in plausible interactive bio-feedback applications. Yet at its beginning, BBCI thus adds a new dimension in
HCI research by offering the user an additional and independent communication channel based on brain activity only. Successful
experiments already yielded inspiring proofs-of-concept. A diversity of interactive application models, say computer games, and their
specific intuitive control strategies are now open for BCI research aiming at a further speed up of user adaptation and increase of learning
success and transfer bit rates.
BBCI is a complex distributed software system that can be run on several communicating computers responsible for (i) the signal
acquisition, (ii) the data processing and (iii) the feedback application. Developing a BCI system, special attention must be paid to the
design of the feedback application that serves as the HCI unit. This should provide the user with the information about her/his brain
activity in a way that is intuitively intelligible. Exciting discovery applications qualify perfectly for this role. However, most of these
applications incorporate control strategies that are developed especially for the control with haptic devices, e.g., joystick, keyboard or
mouse. Therefore, novel control strategies should be developed for this purpose that (i) allow the user to incorporate additional
information for the control of animated objects and (ii) do not frustrate the user in the case of a misclassification of the decoded brain
signal.
BCIs are able to decode different information types from the user’s brain activity, such as sensory perception or motor intentions and
imaginations, movement preparations, levels of stress, workload or task-related idling. All of these diverse brain signals can be
incorporated in an exciting discovery scenario. Modern HCI research and development technologies can provide BCI researchers with
the know-how about interactive feedback applications and corresponding control strategies.
r 2006 Elsevier Ltd. All rights reserved.

Keywords: Brain–Computer Interface; Scientific discovery; Electroencephalography; Digital Signal Processing; Machine learning; Bio-feedback; Control
strategy; Gaming; Virtual limbs

Corresponding author. Present address: Bergweg 6, D-61462 Königstein im Taunus, Germany. Tel.: +49 175 5766504; fax: +49 6173 94 46504.
E-mail address: [email protected] (R. Krepki).

1071-5819/$ - see front matter r 2006 Elsevier Ltd. All rights reserved.
doi:10.1016/j.ijhcs.2006.11.010
 ARTICLE IN PRESS
R. Krepki et al. / Int. J. Human-Computer Studies 65 (2007) 460–477 461

1. Introduction 1995; Quek et al., 2002). Since all these information

streams pass its own interface (hand/skin, eye, ear, nose,
In the seven decades since the original publication of muscles) yet indirectly converge or emerge in the brain, the
Berger (1929) the electroencephalogram (EEG) has been investigation of a direct communication channel between
used mainly to evaluate neurological disorders and to the application and the human brain should be of high
investigate brain function. Besides, people have also interest to HCI researchers (Ebrahimi et al., 2003).
speculated that it could be used to decipher thoughts or Furthermore, Steriadis and Constantinou (2003) state,
intents, such that a person will be able to control devices that development of Human–Computer Interfaces (HCIs)
directly by her/his brain activity, bypassing the normal for people with severe disabilities, e.g., amyotrophic lateral
channels of peripheral nerves and muscles. However, due sclerosis (ALS), or quadriplegia due to high spinal cord
to the large amount of data to be analysed, it could attract injury or brainstem stroke patients, is an important issue
serious scientific attention only in the last decade, for integrating them into an emerging Information Society.
promoted by the rapid development in computer hardware Due to the damage of normal communication paths, e.g.,
and software, as nowadays it is possible to distribute tasks peripheral nerves and muscles that are required for
of a complex system over different computers commu- interaction with computers or other devices, information
nicating with each other and to process acquired data in a on intention of movement execution can be extracted from
parallel manner and in real-time. the last faultless communication stage. Green et al. (1999)
Moreover, recent research in digital signal processing have shown that motor and sensory cortices of patients
and data analysis provide the possibility to develop with amputated extremities, e.g., arms or legs remain intact
intelligent and automatically adapting systems, which do and produce normal spatio-temporal activation patterns
not rely on prior knowledge about the user, but set-up its on intentions to move the absent part of the body, as they
model at first contact and online. Obviously, a test person, can be observed in healthy people. Accordingly, a
interacting with a computer application, generates distinct technique for recognizing and deciphering those patterns
spatio-temporal neuroelectric patterns in cortices of her/his and translating them into device control commands might
brain. In particular, the cerebral processing of visual or serve as the core for a wide variety of applications in the
auditory information provided by the monitor screen or field of HCI, which will provide to handicapped people the
loudspeakers produces specific EEG patterns in the ability to communicate with their environment or to
primary visual or, respectively, auditory sensory cortices. control various devices.
Those can easily be observed and classified by EEG A special role must be assigned to the intelligibility of
recorded over the respective cortex. Notably, further feedback. Ramachandran (1999) reports that patients
cognitive processing of this information is widely distrib- following lateral hemisphere stroke display an indifference
uted across the cortex posing a hard challenge for non- to objects and events in the contralateral side of the world
invasive recording techniques to disentangle the contribu- (neglect). Looking into a mirror and imagining moving the
tions from different cortical processing modules. Thus, absent arm, which is a reflection of the other intact arm
while the intentions to control the system by performing helps to allay the phantom pains and accelerates the
some motor, i.e., muscle activity, emerges inside the brain’s recovery from neglect. Producing natural feedback on a
high-level decision centers, finally they take their way to the computer screen with actions, correlated to the intentions
primary motor cortices, such that rising neural activity can of the patient might have similar helpful consequences for
be read out from surface EEG electrodes placed over the the convalescence.
motor brain regions, which fortunately have a regular Finally, talking about bio-feedback as the core of matter,
somatotopic arrangement, i.e., the body is represented in a one cannot avoid mentioning the recently developed
orderly topography (Penfield and Rasmussen, 1950). commercial system ‘‘The Journey to the Wild Divine’’
Due to physical limits in spatial resolution of surface (www.wilddivine.com). This incorporates different bio-
EEG, the discrimination of nearby located cortical areas signals which can be acquired with inexpensive devices
represents a challenging problem for data analysis, since and in home environment, like heart beat pulse, skin
each single electrode acquires superposed data from within surface conduction measured at the fingertips, breath
a certain neighbourhood radius, where many originally frequency and depth, etc. to train relaxation and medita-
different signals are superimposed. tion. This incorporates sumptuous graphics and anima-
Currently, modern HCI and multimedia technologies tions to visualize the user’s bio-signals in a relaxing gaming
address only a subset of I/O channels humans use for scenario. Its greatest advantage is that it is made
interaction with a computer application or a device. Those amendable to a wide range of users. However, it forces
demand mainly motor (joystick, pedal), visual (graphics, the user to generate a certain bio-signal pattern, rather than
animation) and acoustic (sound, music, speech) senses. employing some machine adaptation technique for fitting
Recent research tries to include also olfaction (Harel et al., the user’s current condition and learning to recognize the
2003), tactile sensation (MacIntyre and Feiner, 1996; user’s intension. Moreover, it is indistinct up to what
Hardwick et al., 1996), interpretation of facial emotions extend it is able to employ brain activity as an independent
(Pantic and Rothkrantz, 2000) and gestures (Pentland, control channel. Undisputable, this product marks a great
 ARTICLE IN PRESS
462 R. Krepki et al. / Int. J. Human-Computer Studies 65 (2007) 460–477

step in bio-feedback research and development; however, 2.2. P300-based BCI

its scientific contribution to the area of signal processing
and brain–computer interfacing is limited. BCI systems are defined to be independent, if they do not
In Section 2 we give a short introduction in state-of-the- rely on any muscular activity, if the message is not carried
art in BCI, and in Section 3, we introduce a novel by peripheral nerves and muscles, and, furthermore, if
communication channel that can be used in HCI as based activity in these pathways is not needed to generate the
on a new technique for information retrieval directly from brain activity (e.g., EEG) that does carry the message. For
the brain. This is followed by a demonstration of a set of example, a subject waiting for the occurrence of a rare
interactive applications used for bio-feedback, in Section 4. stimulus on the background of a series of standard stimuli
Section 5 concludes with a discussion on future perspec- evokes a Positive peak over parietal cortex about 300 ms
tives of BCIs in the fields of human–computer interaction, (P300) after appearance. Donchin and Smith (1970)
control and discovery. presented a P300-based BCI used for typing of ca.
5 letters/min. However, those techniques remain limited
2. State of the art in BCI to letter selection paradigms, similar to that one described
in the previous subsection.
A recent review on BCI defines a Brain–Computer Approaches for independent BCIs are of greater
Interface as a system for controlling a device, e.g., computer, theoretical interest than for dependent BCIs, because they
wheelchair or a neuroprosthesis by human intentions, which offer the brain a completely new output pathway and are
does not depend on the brain’s normal output pathways of likely to be more useful for people with most severe
peripheral nerves and muscles (Wolpaw et al., 2002). neuromuscular disabilities.
There are several non-invasive methods of monitoring
brain activity encompassing functional Near-infrared 2.3. BCI based on motor imagery
Imaging (fNIR), Positron Emission Tomography (PET),
functional Magnetic Resonance Imaging (fMRI), Magne- In Albany, New York, Jonathan Wolpaw directs the
toencephalography (MEG) or Electroencephalography development of a BCI system that lets the user steer a
(EEG) techniques, which all have advantages and short- cursor by oscillatory brain activity into one of two or four
comings.1 Notably, EEG alone yields data that is easily possible targets (Wolpaw et al., 1991). In the first training
recorded with comparatively inexpensive equipment, is sessions most of the subjects use some kind of motor
rather well studied and provides high temporal resolution. imagery, which is then, during further feedback sessions,
Thus, it outperforms remaining techniques as an excellent replaced by adapted strategies. Well-trained users achieve
candidate for BCI. EEG-based BCI systems can be hit rates of over 90% in the two-target set-up; however,
subdivided into several groups according to the electro- each selection typically takes 4–5 s.
physiological signals they use. The lab in Graz of Gert Pfurtscheller develops a BCI
system that is based on event-related modulations of the
2.1. Visual-evoked potentials m- and/or the central b-rhythm of sensorimotor cortices.
For control paradigm the focus is on motor preparation
Visual-evoked potentials (VEPs) define a dependent BCI, and imagination. Feature vectors calculated from sponta-
i.e., they depend on oculomotor control of gaze direction, neous EEG signals by adaptive auto-regressive modelling
such that activity in the normal information pathways, e.g., are used to train a classifier. In a ternary classification task
peripheral nerves and muscles is needed to generate the accuracies of over 96% were obtained in an offline study
brain activity. Sutter (1992) described a brain response with trial duration of 8 s (Peters et al., 2001).
interface (BRI) applying it as a keyboard interface: by
selecting a symbol from a set of 64 proposed in an 8  8
2.4. Event related (de-)synchronization
matrix by focusing on it volunteers were able to type
10–12 words/min. Symbols were changing their colour or
Physiologically meaningful signal features can be ex-
flashing with a certain frequency, which induces a distinct
tracted from various frequency bands of recorded EEG,
spatiotemporal pattern in the visual cortex of the user’s
e.g., Pfurtscheller (1999) reports that m and/or b rhythm
brain. However, this method requires stable control over
amplitudes serve as effective input for a BCI. Movement
oculomotor muscles, needed for focusing a letter. A
preparation, followed by execution or even only motor
dependent BCI is essentially an alternative method for
imagination is usually accompanied by a power decrease in
detecting messages carried out in the brain’s normal output
certain frequency bands, labelled as event-related desyn-
pathways, but does not give the user a new communication
chronization (ERD); in contrast, their increase after a
channel that is independent of conventional channels.
movement indicates relaxation and is due to an event-
1
Please note that this list of currently available technologies for related synchronization (ERS) in firing rates of large
measuring and visualizing brain activity does not claim for any populations of cortical neurons. Table 1 summarizes
exhaustiveness or completeness. frequency bands (marginal frequency values are highly
 ARTICLE IN PRESS
R. Krepki et al. / Int. J. Human-Computer Studies 65 (2007) 460–477 463

Table 1 10
Frequency bands 0
Recorder M
bp
sE
Band Frequency (Hz) Occur while/indicate th
er
ne
t
d 0.5–3.5 Movement preparation
y 3.5–8 Memory 4 5
a (m) 8–13 Relaxation, sensory idling
b 13–22 Motor idling
g 22–40 Feature binding Database

subject-specific) and neurophysiological features they are 3

assumed to encode. Amplifier
Signal
Processing
2.5. Slow cortical potentials
6

Slow cortical potentials (SCP) are voltage shifts gener-

ated in cortex lasting over 0.5–10 s. Slow negativation is
usually associated with cortical activation, e.g., evoked by Brain-Cap
2
the implementation of a movement or by the accomplish-
ment of a mental task, whereas positive shifts indicate
cortical relaxation (Birbaumer, 1997). Further studies Feedback
showed that it is possible to learn SCP control. Conse-
quently, it was used in Birbaumer et al. (1999) to control
movements of an object on a computer screen in a BCI
7
referred to as Thought Translation Device (TTD). After
repeated training sessions over months, through which
patients achieve accuracies over 75%, they are switched to 1
a letter support programme, which allows selection of up to
3 letters/min.
A new letter selection protocol, involving a predictive Fig. 1. Distributed software design of the BBCI system.
algorithm that uses a set of first letters of a word to select
the whole word from a lexicon which adapts to the user’s
vocabulary simultaneously, increases the communication is based on lateralized readiness potentials (LRP), which is
rate and provides Internet access to a disabled user a form of SCP and appear during movement preparation.
(Birbaumer et al., 2000). Interestingly, the intrinsic movement execution is not
essential since LRP variants can be observed also for
2.6. Invasive methods for BCI imagined movements in healthy test persons. Note,
however, that the intention of a person to move her/his
Using information recorded invasively from an animal amputated arm is not identical with imagined arm move-
brain Nicolelis and Chapin (2002) report a BCI able to ments of a healthy person because in the latter case an
control a robot. Four arrays of fine microwires penetrate additional ‘‘no-go’’ or ‘‘veto’’ signal is required to prevent
the animal’s scull and connect to different regions inside the actual motor performance. Therefore, the BBCI
the motor cortex. A robotic arm remotely connected over focuses on the preparation of real, rather than imagined
the Internet implements roughly the same trajectory as the movements.
owl monkey gripping for food. This invasive technology The enormous amount of data to be processed in a
allows the extraction of signals with fine spatial and limited time forced the distribution of processing tasks over
temporal resolution, since each microelectrode integrates several computers communicating via client–server inter-
firing rates of few dozens of neurons. However, to make a faces. Moreover, this distributed concept allows advanta-
BCI attractive to an everyday-user it should be non- geous replacement of single modules according to
invasive, fast mounted and leave no marks. particular communication protocols. Fig. 1 illustrates the
distributed software design of the BBCI system.
3. The Berlin Brain–Computer Interface (BBCI) The volunteer user (1) is facing a computer screen.
A drapery brain-cap (2) furnished with 128 electrodes is put
This section presents an independent non-invasive EEG- on her/his head. Four flat cables of 32 wires each connect
based online-BCI, developed at Fraunhofer FIRST and the the cap with four amplifiers (3), which also perform
Neurophysics Group of the Charité—Universitaetsmedizin an A/D-conversion and transmit the acquired EEG at
Berlin that overcomes limitations mentioned above. BBCI sampling rate of 5 kHz and accuracy of 16 bits via a fiber
 ARTICLE IN PRESS
464 R. Krepki et al. / Int. J. Human-Computer Studies 65 (2007) 460–477

Feature
Classification
Selection
Detection

Synchronization
Detection
Data
Acquisition Combination
RDA
Feature
Classification
Selection
Discrimination
Discrimination

Storage

Fig. 2. Parallel manner of the BBCI’s data processing unit.

optic cable (to avoid electromagnetic interferences) to the (EMG) signals, which detect muscle activity at both
recorder PC (4). The recorder performs some predefined forearms, as well as horizontal and vertical electrooculo-
simple preprocessing operations, i.e., subsampling to gram (EOG) signals, which reflect eye movements, were
1 kHz, optional low/high/band-pass or notch filters, and recorded. All signals were band-pass filtered between 0.05
stores the data in raw format for later offline analysis into and 200 Hz and sampled at 1000 Hz. For online analysis,
the database (5). Additionally it acts as Remote Data the data signals were then subsampled to 100 Hz to
Access (RDA) server, which allows up to 10 client minimize the data processing effort.
connections and serves one data block of acquired EEG The labels of electrodes are composed of some letters
and auxiliary information, e.g., control signals or event and a number. The letters refer to anatomical structures
markers, each 40 ms. A second computer (6) runs a (Frontal, Parietal, Occipital, Temporal lobes and Central
corresponding client, which performs, after data acquisi- sulcus), while the numbers denote sagittal (anterior–pos-
tion, some preprocessing steps for feature selection (details terior) lines. Odd numbers correspond to the left hemi-
in Section 3.4) in a parallel manner. For the detection and sphere, while even numbers to the right; small ‘z’ marks
the discrimination of user actions two separate non- electrodes on the central sagittal line. Labels with 1 or 2
blocking threads were employed. Each is followed after an capital letters correspond to the 64 electrodes of the
optional synchronization by a classification step of the extended international 10–20-system as defined in Shar-
current extracted feature vector (details in Section 3.5). brough et al. (1991) while labels with three capital letters
Finally, a combiner joins the two-classifier results and were composed from the neighbouring electrode labels and
produces a control command. Fig. 2 illustrates the parallel denote additional channels in a 128-channel set-up. EEG
approach of data processing. The online classifier (6) acts activity is measured against the reference electrode (Ref)
as a server for various feedback clients (7) and serves each mounted on the nasion, while the ground electrode (Gnd)
40 ms the control command that has been produced by the is mounted on the forehead. Locations of the electrodes
combiner. and corresponding labels are illustrated in Fig. 3.
The feedback client is an interactive application that may The voltage measured by the electrodes is very low and
run on a separate computer and acquires the control fluctuates rapidly within the range of 7100 mV around a
commands produced by the combiner module of the data baseline. Electrical noise from the surrounding environ-
processing server. It is conceived to rely on simple control, ment (mainly 50 Hz, resp., 60 Hz power outlet frequency)
e.g., left/right movements, which may be expressed by a interferes with the data via connecting wires, which act as
small command set, and should give the user a feeling of small ‘‘antennas’’. To assure low impedances between the
being inside the simulation. Currently, we employed simple electrodes and the scalp (desired below 5 kO), electrolyte
computer games like Pacman or Tele-Tennis, however, gel is filled into each electrode before experiments start.
other more sophisticated and challenging applications, like
Tetris or manoeuvreing through a Virtual Reality (VR) 3.2. Task and its neurophysiology
maze are conceivable.
We let our subjects (all without neurological deficits)
3.1. Data acquisition take a binary (left/right hand) decision coupled to a motor
output, i.e., self-paced typewriting on a computer key-
We recorded brain activity with multi-channel EEG board. Using multi-channel scalp EEG recordings, we
amplifiers (Brain ProductsTM, Munich) using 128 channels analyse the single-trial differential potential distributions of
from a cap with Ag/AgCl Electrodes (+ of the contact the Bereitschaftspotential (BP/Readiness potential) preced-
region is 5 mm). Additionally, surface electromyogram ing voluntary (right or left hand) finger movements over
 ARTICLE IN PRESS
R. Krepki et al. / Int. J. Human-Computer Studies 65 (2007) 460–477 465

Ref

Gnd

Fp1 AF AF FP2
p1 p2
AF7 AF8

FAF AF3 AF4 FAF

5 6 F1
F9 AFz
FAF FAF 0
F7 1 2 F8
F5 F6
FFC F3 F4 FFC
9 FFC F1 Fz F2 FFC 10
7 FFC FFC 8
5 FFC FFC FFC FFC 6 FT10
FT9 3 4
FT7 1 2 FT8
FC5 FC3 FC4 FC6
CFC FC1 FCz FC2 CFC
9 CFC CFC CFC CFC CFC CFC 10
7 CFC CFC 8
5 3 1 2 4 6

T7 C5 C3 C1 Cz C2 C4 C6 T8
CCP CCP CCP CCP CCP CCP
CCP 1 2 CCP
7 5 3 4 6 8
CP1 CPz CP2
CP5 CP3 CP4 CP6
TP7 PCP PCP PCP PCP TP8
TP9 PCP 1 2 PCP TP10
3 4
PCP 5 6 PCP
PCP 7 P1 Pz P2 8 PCP
9 P3 P4 10
P5 PPO PPO P6
P7 1 2 P8
PPO POz PPO
P9 5 6 P10
PPO PO3 PO4 PPO
9 OPO OPO 10
PO7 1 2 PO8
PO9
O1 O2

Oz
O9 O10

Iz

Fig. 3. Locations of electrodes and labels of the corresponding channels.

the corresponding (left/right) primary motor cortex. As we about 1 s prior to the actual movement onset; it is assumed
study brain signals from healthy subjects executing real to reflect mainly the growing neuronal activation (apical
movements, our paradigm requires a capability to predict dendritic polarization) in a large ensemble of pyramidal
the laterality of imminent hand movements prior to any cells. Previous studies of Lang et al., 1989 and Cui et al.
EMG activity to exclude a possible confound with afferent (1999) showed that in most subjects the spatial scalp
feedback from muscle and joint receptors contingent upon distribution of the averaged BP correlates consistently with
an executed movement. the moving hand (focus of brain activity is contralateral to
The basic BBCI idea is focusing on control applications, the performing hand).
such as ‘‘virtual keyboard typing’’, that can be conceived as The upper part of Fig. 4 shows Laplace filtered EEG
potentially resulting from a natural sequence of motor around the left and right hand motor cortices (electrodes
intention, followed by preparation and completing by the C3 and C4) within a time range of [450:200] ms relative to
execution. Accordingly, our neurophysiological approach the key tap, averaged selectively for left-hand vs. right-
aims to capture EEG indices of preparation for an hand taps. The grey bars indicate a 100 ms baseline
immediately upcoming motor action. At present, we correction. The lateralization of BP is clearly specific for
exploit the BP, i.e., a slow negative EEG shift, which left, resp., right finger movements. Four potential maps
develops over the activated motor cortex during a period of show the scalp topographies of the BP averaged over time
 ARTICLE IN PRESS
466 R. Krepki et al. / Int. J. Human-Computer Studies 65 (2007) 460–477

C4 laplacian C3 laplacian

left hand right hand

[-400 : -200 ms]
[-200 : 0 ms]

Fig. 4. Averaged Lateralized Readiness Potentials (LRP).

windows (upper) before movement preparation and (lower) arms, legs or the whole body. These would induce
when BP reaches its maximum negativation, again electromyographic (EMG) noise activity that interferes
averaged over left-hand and right-hand taps separately. with EEG signals, such that the signal-to-noise-ratio
Bold crosses mark electrode positions C3 and C4. (SNR) decreases. Eye movements are to be minimized for
We would like to emphasize that the paradigm is shaped the same reason. To prevent possible (involuntarily)
presently for fast classifications in normally behaving cheating, e.g., producing eye movements correlated with
subjects and thus could open interesting perspectives for the performed tasks, vertical and horizontal EOG are
a BCI assistance of action control in time-critical recorded, which can also be used for artifact correction,
behavioural contexts. Notably, also a possible transfer to i.e., cleaning up EEG signals of interfering EOG by a
BCI controlled by paralyzed patients appears worthwhile weighted subtraction.
to be studied further because these patients were shown to The training is performed in 3–4 sessions, each of about
retain the capability to generate BPs with partially 7 min, as illustrated in Fig. 5. Tasks are performed for a
modified scalp topographies (Green et al., 1999). period of 6 min repeatedly with an interval of 0.5–2 s. All
training sessions may be performed in two experimental
3.3. Training procedure kinds: (i) imagined, i.e., queried, (ii) executed, i.e., self-paced:
In the executed task experiment we acquire response markers
The Leitmotiv of BBCI is: ‘‘Let the machines learn!’’, via keyboard, while the user determines her-/himself which
thus the user should invest only a minimum of time for movement to perform next. During the imagined task
training the BBCI algorithms: The training procedure experiment a visual cue indicates the movement, which has
described here serves for ‘‘teaching the machine’’ and to be executed on the next auditory beat produced by a
adjusting its model parameters to better match the user and digital metronome. Both stimuli place corresponding mar-
her/his brain signal’s properties. During the training kers into the data, stored with a time stamp.
procedure we acquire example EEG from the user while To train the learning machine and adjust its parameters,
performing a certain task, e.g., execution or imagination of we select time series of EEG activity acquired within a
left vs. right hand movement of the index or pinky fingers. certain time region before the marker, which gives the
During the training session the user is instructed to sit training sample its label. We search for event markers in
comfortably and, as far as possible, to omit any muscular the acquired data; these markers are (i) keyboard taps as
artifacts, like biting, gulping, yawning, moving the head, responses for executed movements, or (ii) a combination of
 ARTICLE IN PRESS
R. Krepki et al. / Int. J. Human-Computer Studies 65 (2007) 460–477 467

the visual cue with the metronome beat stimuli for be paid in fast-pace experiments to the issue that samples of
imagined experiments. We then examine each marker for the rest class do not intersect with action class samples of
affiliation to one of the classes of interest. Each class of the preceding event marker, as they should not include any
interest covers its own sample-selection parameter set information about action.
SSP ¼ ({mrk}, n, td, ti), where a set of marker labels mrk
identifies the affiliation of markers to classes, n gives the 3.4. Preprocessing and feature selection
number of training samples to be selected from the data, td
and ti are time constants indicating the delay of the initial To extract relevant spatiotemporal features of slow brain
sample and the inter-sample interval. Beside the classes potentials we subsample signals from all or a subset of all
indicating action, e.g., execution or imagination of a available channels and take them as high-dimensional
movement, which in Fig. 6 provide samples 1a, 2a and feature vectors. We apply a special treatment because in
3a, an additional class indicating rest is introduced. This pre-movement trials most information is expected to
provides in an analogue manner (incorporating t0 d and t0 i appear at the end of the given interval. Starting point of
time constants that indicate the delay of the initial rest- this treatment are epochs of 128 data points (width of a
sample and the inter-sample interval) training samples 1r sample window) of raw EEG data, corresponding to
and 2r that are used together with action samples for 1280 ms as it is depicted in Fig. 7(a) for a single EEG
determining the detection of the task accomplishment, channel from 1400 to 120 ms (td) relative to the
though we use action samples only, for determining the timestamp of the desired event marker. To emphasize the
discrimination of which task has been completed. For late signal content, we first multiply the signal by a one-
sample selection in the training procedure, negative time sided cosine function (1), as shown in Fig. 7(b).
constants are preferred, positive are allowed, however, they   np 
make no sense for online analysis. Special attention must 8n ¼ 0; . . . ; 127 : wðnÞ :¼ 0:5  1  cos . (1)
128
A Fast Fourier Transformation (FFT) filtering techni-
relax perform task repeatedly relax
que is applied to the windowed signal. From the complex-
valued FFT coefficients all are discarded but the ones in the
40 sec 6 min 20 sec pass-band (including the negative frequencies, which are
not shown); cf. Fig. 7(c). Transforming the selected bins
Fig. 5. Set-up of a single training session. back into the time domain gives the smoothed signal of
which the last 200 ms are subsampled at 20 Hz by
t i' t d' calculating means of consecutive non-overlapping inter-
...
... ti td vals, each of 5 samples, resulting in 4 feature components
t per channel, see Fig. 7(d).
#1r ... #1a
Marker
#2r #2a (Stim./Resp.) 3.5. Linear methods for classification
#3a
In BCI research it is very common to use linear
Fig. 6. Selection procedure for training samples. classifiers, but although linear classification already uses a

Fig. 7. Preprocessing procedure.

 ARTICLE IN PRESS
468 R. Krepki et al. / Int. J. Human-Computer Studies 65 (2007) 460–477

classifier is that procedures have been established, e.g.,

w
Duda et al. (2001); or Vapnik (1995), on how to select the
optimal classifier w for unseen data.
Linear Support Vector Machines (SVM) realize the large
margin by determining the normal vector w. The one
particular strength of SVMs is that they can be turned into
nonlinear classifiers in an elegant and effective way that is
extensively described in Vapnik (1995), Schölkopf (1997),
and Mika et al. (2001).

3.5.2. Fisher’s discriminant

The event related potential (ERP) features are super-
positions of task-related and many task-unrelated
signal components, e.g., background auditory, visual or
receptional noise or task independent thoughts. The
mean of the distribution across trials is the non-oscillatory
task-related component, ideally the same for all trials.
Fig. 8. Linear classifier and margins. From Müller et al. (2001).
The covariance matrix depends only on task-unrelated
components. Our analysis showed that the distribution
of ERP features is indeed normal. The important observa-
tion here is that the covariance matrices of both classes
very simple model, the analysis can still go wrong if the (left/right movements) look very much alike (Blankertz
underlying assumptions do not hold, e.g., in the presence of et al., 2003). This property of the ERP data set proposes
outliers or strong noise, which are situations very typically the application of the Fisher’s Discriminant as the
encountered in BCI data analysis. We will discuss these classificator.
pitfalls and point out ways around them. The Fisher’s Discriminant is searching for a separating
Let us first fix the notation and introduce the linear hyperplane which subdivides the feature space into two
hyperplane classification model upon which we will rely classes, optimizing its model parameters in two ways: (i) it
mostly in the following (cf. Fig. 8). In a BCI set-up we maximizes the distance between the centers of mass of the
measure k ¼ 1 y K samples xk, where x are some two classes (inter-class variance) and (ii) minimizes the
appropriate feature vectors in n-dimensional space. In the variances of the data inside each class (intra-class
training data we have a class label, e.g., ykA{1, +1} for variances). The principal separation procedure used by
each sample point xk. To obtain a linear hyperplane RFD is illustrated in Fig. 9.
classifier Several other regularized linear classification procedures,

like Linear Perceptron with weight decay or Linear
y ¼ sign wT x þ b , (2)
Programming Machines (LPM) have been employed for
we need to estimate the normal vector of the hyperplane w this task in Blankertz et al. (2002a, b); however, no
and a threshold b from the training data by some significant difference in classification accuracy could be
optimization technique. On unseen data x, i.e., in a BCI determined. The major advantage of the RFD-based
feedback session, we compute the projection of the new classifier is due to its lower computational costs,
data sample onto the direction of the normal w via Eq. (2), such that the gain in performance could be maximized.
thus determining what class label y should be given to x As well several nonlinear classifiers, e.g., Quadratic
according to our linear model. Discriminant Analysis (QDA) or SVMs with Gaussian
A linear classifier is defined by a hyperplane’s normal kernels, as proposed in (Mika et al., 2003), have been
vector w and an offset b, i.e., the decision boundary is applied, however, their classification accuracy was
{x|wTx+b ¼ 0} (thick line). Each of the two half-spaces seldom higher than that of the best linear classifiers,
defined by this hyperplane corresponds to one class, i.e., often they performed even worse. This fits with our
f ðxÞ ¼ signðwT x þ bÞ. The margin of a linear classifier is the experience that the ERP features of different classes (e.g.,
minimal distance of any training point to the hyperplane. left and right hand) of motor trials are Gaussian
In this case it is the distance between the dotted lines and distributed with equal covariance matrices. Thus, in this
the thick line (cf. Fig. 8). case the classes are linear separable and hence linear
classifiers are more appropriate. Of course, nonlinear
3.5.1. Large margin classification classifiers can also learn linear problems, but due to the
For linearly separable data there is a vast number of increase of complexity of their models they are more
possibilities to determine (w; b), that all classify correctly on susceptible to noise which is a principle concern in EEG
the training set, however, that vary in quality on the unseen data (see also the discussion of this issue in Müller et al.
data (test set). An advantage of the simple hyperplane (2003) and Krepki (2004)).
 ARTICLE IN PRESS
R. Krepki et al. / Int. J. Human-Computer Studies 65 (2007) 460–477 469

40

20

µV at C4
max

-20

min
-40 min

-60 y0
-80 -60 -40 -20 0 20 40
µV at C3

Fig. 9. Principal separation procedure by the Regularized Fisher’s Discriminant (RFD) for a 2-class problem.

a b c

Fig. 10. The problem of finding a maximum margin ‘‘hyper-plane’’ on reliable data (a), data with an outlier (b) and with a mislabelled pattern (c). From
Rätsch et al. (2001).

3.5.3. Regularization and non-robust classifiers complexity of the classifier (e.g., to avoid Fig. 10c) and
Linear classifiers are generally more robust than their (iii) the raggedness of the decision surface (e.g., to avoid
nonlinear counterparts, since they have only limited Fig. 10c).
flexibility (less free parameters to tune) and are thus less The cross-validation procedure that is employed for
prone to over-fitting. Note, however, that in the presence of regularization purposes in the BBCI system is discussed in
strong noise and outliers even linear systems can fail. In more detail at the beginning of Section 4.
Fig. 10, one can clearly observe that one outlier or strong
noise event can change the decision surface drastically, if 3.6. Bio-feedback
the influence of single data points on learning is not
limited. The solid line in Fig. 10 shows the resulting Finally, an interactive application, running on a separate
decision line, whereas the dashed lines mark the margin computer, receives combined results of classification via an
area. In the middle and on the right the original decision asynchronous client–server interface based on the User
line is plotted grey. Illustrated is the noise sensitivity: only Datagram Protocol (UDP) and acquires them in a
one strong noise/outlier pattern can spoil the whole temporal queue. It examines the queue repeatedly for
estimation of the decision line. stationary past signals persisting for a certain time length,
Although this effect can yield strongly decreased i.e., a Command Activation Term (CAT), and emits the
classification results for linear learning machines, it can command corresponding to the class label of the classifica-
be even more devastating for nonlinear methods. A more tion result (left/right/rest). After a command has been
formal way to control one’s mistrust in the available emitted, it then falls into ‘‘relaxation’’ for a certain
training data is to use regularization, as it is proposed in time period, i.e., Command Relaxation Term (CRT),
Poggio and Girosi (1990) and Orr and Müller (1998) and which should be at least as long as the CAT. During this
elsewhere. Regularization helps to limit (i) the influence of period combiner outputs remain being collected in the
outliers or strong noise (e.g., to avoid Fig. 10b), (ii) the queue, but further command emissions are suppressed.
 ARTICLE IN PRESS
470 R. Krepki et al. / Int. J. Human-Computer Studies 65 (2007) 460–477

This procedure, for three classes: left (black), right (grey) 3.7. On erroneously emitted commands
and rest (dashed) is illustrated in Fig. 11. Here the
combiner yields the class label (denoted as colour of bars) It is a well-known finding in human psychophysics, that
and the fuzzy values P~ max ¼ maxi P~ i of the most likely a subject’s recognition of having committed a response
recognized class (depicted as amplitude) distributed over error is accompanied by specific EEG variations that can
time at a frequency of 25 Hz. CAT is set to 10 periods easily be observed in averaged ERPs. Blankertz et al.
(400 ms), and CRT is set to 14 periods (560 ms). (2002a, b) present a pattern recognition approach that
This flexible set-up allows individual adjustments for allows for a robust single trial detection of this error
the user and the control strategy of the bio-feedback potential from multi-channel EEG signals. It is an elegant
application: (i) long CAT helps to avoid false-positively approach to overcome the problem of low classifi-
emitted commands; (ii) short CAT, in contrast, allows fast cation accuracy by a response checking mechanism that
emission of commands, i.e., before the real movement is is based on the subject’s brain signals themselves, such
executed; (iii) intraindividually adjusted CRT prevents that those persons benefit most who otherwise can only
erroneous emissions and, respectively, allows volitional reach a modest BCI control because of a substantial
successive emissions of the last command. These para- fraction of classification errors. The ERP after an error
meters depend strongly on the user and should be set trial is characterized by two components: a negative wave
initially to values calculated from the results of the (NE) with a fronto-central maximum, and a following
application of trained classifier to the training data. broader positive peak (PE) with a centro-parietal max-
At starting point CAT0 may be set to the median length imum. NE seems to initiate some kind of comparison
of the stable signal containing a marker of the same process since it is present also in most correct trials, while
action class, and CRT0 to a value larger than CAT0 by PE seems to indicate brain’s reaction of recognizing that
twice the amount of the standard deviation of the the subject’s action was erroneous. Fig. 12 shows average
distribution of lengths of stable signals. The values of miss-minus-hit EEG-traces at electrodes along the vertex
CAT and CRT should then be adjusted according to the and the scalp EEG potential topographies around that
user’s demand. region.
The underlying interactive feedback application should To assess the potential value of error detection for
be intuitive, simple to understand, and the control strategy improving BCI transmission rates we calculate the amount
should give the user a feeling of natural acting; however, it of information that can be read out from a 2-class decision
should require a small (at present: binary) control set of experiment (N ¼ 2) with a BCI providing an accuracy of
commands, i.e., left-turn/right-turn, avoid fast animation 85% (p ¼ 0.85) using Shannon’s information criterion (3):
and high-contrast changes to prevent or at least to
minimize spoiling of data affected by artifacts, e.g., brisk 1p
IðpÞ :¼ log2 N þ p log2 p þ ð1  pÞlog2 , (3)
movements of eye, head or body. An issue of particular N 1
importance for a fast pacing of control commands is a
‘‘natural mapping’’ of the action required in the virtual I(0.85) ¼ 0.39 bits per selection for the system alone,
reality scenario to the ‘‘action space’’ of the human however, it can be improved by more than 75% to 0.69 bits
operator, which is coded in egocentric coordinates. To this per selection in a system involving an error correction
end the on-screen environmental perspective must con- method working with 20% false-negative (FN ¼ 0.2) and
tinuously represent the viewing direction of the human 3% false positive (FP ¼ 0.03), where the accuracy of the
operator, so that, e.g., a selection of the option of right- improved system can be calculated by
turn can be addressed by the intention to move the right
hand and vice versa. p0 ðp; FP; FNÞ ¼ p  ð1  FPÞ þ ð1  pÞ  ð1  FN Þ
¼ 0:94; and Iðp0 Þ ¼ 0:69 bit ð4Þ

Note, that this is only a theoretical value used as

BCI performance measure. Achievement of this informa-
tion transmission rate would require a specific coding
of the information by the BCI user. Fig. 13 shows a plot
of the theoretical information rate (I) in a two-class
t experiment as a function of the accuracy (p) of the pure
BCI system with and without the error correction
CAT CRT CAT CRT
procedure working with an assumed rate of 20% of
400 ms Class "left" real left/right action false negatives and 3% of false positives. Obviously the
Class "right"
gain gets less, the higher the original BCI accuracy is; note
emited commands that with the assumed parameters an error correction
Class "rest"
approach is useful, as long as the pure BCI accuracy is
Fig. 11. Time structure of the command emission queue. below 96%.
 ARTICLE IN PRESS
R. Krepki et al. / Int. J. Human-Computer Studies 65 (2007) 460–477 471

Fig. 12. Averaged miss-minus-hit error-related potentials and the corresponding scalp topographies.

Cl 1 Tst

Cl 2 Tst

Cl 3 Tst

Cl n Tst

Fig. 14. The n-fold cross-validation procedure.

50 0
classification error [%]

40

[bits per minute]

2
30
5
20
Fig. 13. Improvement of the BCI system by the error correction 10
procedure. 10 EEG 15
EMG 20
0 30
-500 -400 -300 -200 -100 0
4. Results point in time of causal classification [ms]

To enable the classifier training, we initially let the user Fig. 15. Classification errors based on EEG (brain activity) vs. EMG
(muscles activity) estimated by the 10  10-fold cross-validation proce-
execute or imagine the required movement repeatedly. For dure.
real movements, which can be monitored, the user may
perform tasks ‘‘self-paced’’. For imagined movements, or
in paralyzed patients, the lateralization of each action (left/ error would indicate that the model is too complex for the
right) is queried by an auditory and/or visual cue. We given data, such that the risk of over-training is high due to
extract training samples, preprocess each as described in bad generalization ability. Notably, test errors of the cross-
Sections 3.3 and 3.4, calculate a set of optimal classifiers on validation procedure depend on the choice of the delay
a selection of 90% of the markers and test each on time td in the pre-processing procedure. Obviously
remaining 10% as it is described in Section 3.5. classification is ambiguous for large values of td and
This procedure is repeated 10 times with all non-over- mostly correct for td ¼ 0. Fig. 15 shows the cross-
lapping test-sets, which is called 10-fold cross-validation, validation test-error of classification of EEG single trials
cf. Fig. 14. In a 10  10-fold cross-validation the whole as a function of td for a single subject performing in a self-
procedure is repeated 10 times with random foldings of the paced experiment with 30 taps/min.
data set. The right ordinate enumerates the theoretical informa-
By calculating means of training and test errors, we tion transfer rate in bits per minute that can be extracted
obtain a measure for effectiveness of a particular classifier from the classification results. Compared to the errors of
model. A test error essentially higher than the training classification based on EMG (upper curve), which reflects
 ARTICLE IN PRESS
472 R. Krepki et al. / Int. J. Human-Computer Studies 65 (2007) 460–477

the muscle activity in the forearms, the EEG approach 1.5

yields superior classification results which become feasible
already 120 ms prior to the actual movement execution.
The EEG-based classification procedure retains its higher 1.0
performance, as classifications obtained after the hit
marker is presented are not interesting any more. This
0.5
phenomenon is neurophysiologically evident, because the
decision about lateralization of the movement has to be
met in the brain firstly, followed by the preparation of 0
cortical neurons and emission of the command down to the
spinal cord, peripheral nerves and to the effector muscles
spending at least 60–80 ms. -0.5

4.1. Feedback scenario ‘‘Jumping-Cross’’ -1.0

-1.5 -1.0 -0.5 0 0.5 1.0 1.5
Initially we implemented a very simple visual bio- Fig. 17. Accumulated feedback trials (64 left and 64 right trials).
feedback application to provide the user with a first feeling
of her/his intentions: a thick black cross is moving over a
full-screened window containing a thin static fixation cross 24 10
in the center and two target fields (dark-red and dark-
green—indicating left-hand and right-hand movements,
respectively). The ordinate position of the ‘‘jumping-cross’’
reflects the normalized decision of the movement detection
classifier (‘‘up’’ indicating action vs. ‘‘down’’ indicating
rest), i.e., the cross jumps into the upper half of the
screen on upcoming action. The abscissa position provides
the natural mapping of the discrimination classifier
result (left vs. right). The ‘‘jumping-cross’’ trails a history
tail of 4 points (data drawn at 40 ms intervals). The single
action trial is indicated as completed, when (i) the screen
freezes on occurrence of an event marker, i.e., after an Fig. 18. Feedback scenario ‘‘Brain-Pong’’.
actual movement is performed, and when (ii) the corre-
sponding lateralization field, the cross is actually located in,
4.2. Feedback scenario ‘‘Brain-Pong’’
is highlighted. Fig. 16 illustrates a typical left and right
event.
In following experimental sessions, we introduced a
A series of single trials acquired over the whole
simple discovery scenario based on a well-known game of
experiment (here: 64 left and 64 right trials) may be
Tele-Tennis. Set-up for a single-player, it is similar to the
represented in an instructive summary plot, cf. Fig. 17.
‘‘Ping Pong’’ video game, cf. Fig. 18. A yellow ball is
Here, crosses were replaced, for clarity; by bold dots and
moving continuously over the screen in a certain direction,
the history tails are painted bold for the three most recent
being bounced from the two lateral sides and the upper
periods and thin for another four preceding periods. The
border of the screen. There is a movable bar (racket) at the
axes represent the classification results of the discrimina-
bottom of the screen, which can be controlled by the
tion and detection classifiers, respectively. It can be
subject’s intentions using several strategies. We describe
recognized at a single glance, that the majority of trials
these in detail in Section 4.2.1.
have been classified correctly.
Several subjects who used the BBCI system to play
‘‘Brain-Pong’’ reported several long phases with many
successive successful trials. This provided them with the
feeling that the racket was becoming integrated as a part of
their own body and that no particular effort was required
to maintain control. Moreover, it was reported that in
successful phases, performance improved even more,
whereas in failure phases filled with mismatching trials,
performance dropped drastically. These observations are
also evident from the neurological and machine-learning
Fig. 16. Feedback scenario ‘‘Jumping-Cross’’ (typical left and right trials) point of view. Since the user is not put under pressure
with a 4-periods history tail. during training sessions, she/he generates ordinary brain
 ARTICLE IN PRESS
R. Krepki et al. / Int. J. Human-Computer Studies 65 (2007) 460–477 473

patterns that are fed into the learning machine. However, control strategy has only a virtual ‘‘rest’’-class for which
during application sessions with feedback, the user may fuzzy value may be calculated from the combination of the
become greedy and at some point generate different brain results of the ‘‘left’’- and ‘‘right’’-class fuzzy values. The
patterns induced by an additional effort to re-seize control outward racket deflections are calculated each time a new
over the feedback. To conclude, the user puts her/himself result data block is received from the classifiers, i.e., at a
in a conflicting situation by applying more effort to gain frequency of 25 Hz. Each deflection is proportional to the
control; the effort induces an immediate change in the difference between the momentary graded values of the two
brain pattern feature space, spanned by the training data action classes if and only if the graded value of the virtual
and thus recognizable by the classifier, which results even ‘‘rest’’-class is below a certain (usually low) threshold;
more in losing control. ‘‘Brain Gamers’’ further report that otherwise, it is set to zero, such that the racket is almost
when control performance worsens significantly (subjec- always on the move. The value of the ‘‘rest’’-class threshold
tively measured), simply adopting a relaxed state or controls the trigger sensitivity for the displacement activa-
indifferent attitude to the game helps to re-gain control tion and can be adjusted, together with the constant of
over the machine. proportionality for the racket outward deflection, depend-
ing on the individual user’s experience and demand.
4.2.1. Control strategies
Basically, we have implemented two different control 4.3. Feedback scenario ‘‘Pacman’’
strategies for the racket, while both appear appropriate in
this scenario; in other applications one might be preferred Finally, the well-known Pacman video game has been
over the other. adapted to serve as bio-feedback. The idea is to combine
In the ‘‘stepwise displacement’’ strategy the racket is the information, available from the ‘‘jumping-cross’’ feed-
moved by a fixed displacement step (e.g., defined as back with an aim-gain inventively in a discovery applica-
percentage of the window width) into the direction tion. A random labyrinth is generated in a full-screened
indicated by the class label (left or right) if one particular window, which has exactly one shortest way (without
command class is recognized for at least one CAT detours) from the entry (in the left wall) to the exit (in the
duration. The racket remains stationary if a ‘‘rest’’ right wall). This path is marked with grey track marks. The
command is detected (a stable signal for at least the CAT player may also decide to run the Pacman through the rest
duration) or if no stable signal is present (if combined of the labyrinth, e.g., to receive additional credits for
classifier result indicates more than one class label within a harvesting some apples.
single CAT duration). Each step is followed by a relaxation As control strategy we use the following approach: The
period (CRT), which, for this control strategy, is set just Pacman makes one step each 1.5–2 s and moves always
slightly longer than CAT. Thus, the racket could be moved straight ahead until it reaches a wall or a right- or left-turn
into the same direction by more than one step, with the command is received. The direction the Pacman is intended
minimal inter-step interval given by CRT+1. to make in the next step is pointed by its yellow nose. When
Notably, although the Lateralized Readiness Potential the system recognizes a turn-command for at least one
(LRP) can indicate the preparation for one particular CAT duration, Pacman turns its head indicating the
movement, a series of movements (and movement repeti- recognition of the command and takes the next crossing
tions in particular) need not necessarily be linked with possibility in the maze. After a turn command is acknowl-
LRPs of the same strength for every single movement; edged, the Pacman does not accept any further commands
rather, the LRP might index primarily the start of the for at least CRT. The simulation is finished when the
whole series, invoking subcortico-cortical routines for the Pacman reaches the exit of the labyrinth, cf. Fig. 19.
execution of the repeated actions. This points to a potential A healthy subject will be able to navigate the Pacman
limitation of the ‘‘stepwise displacement’’ strategy: if a fast through the presented labyrinth within 40 s (20 steps, each
racket transition from one side of the screen to the other is of 2 s) using a conventional keyboard or a mouse. Using
required, the user has to emit a series of identical brain control it takes considerably longer, however, the
commands. The identification of optimal EEG correlates ‘‘fun-factor’’ of navigating just by intentions of the own
for such a hyper-command will require further neurological brain turned out to be very appealing. Moreover, it is
studies. highly interesting that when immersed into the BCI-game
A second control strategy utilizes a ‘‘graded displace- scenario the user has sometimes the feeling that the
ment’’ code which exploits the strength of the recognized Pacman moves in the correct direction though the user
command signal, e.g., a measure of BP amplitude, instead was consciously not aware of his decision, sometimes
of just the command emission time point identified by the consciously not even ready for a decision.
combiner. From its initial location at the screen center, the
racket can be deflected laterally (outward), but, as it is 4.4. Feedback scenario ‘‘Virtual Arm’’
attached to an imaginary spring, it will be returned to its
initial point (inward) whenever the fuzzy values of neither The ‘‘Virtual Arm’’ feedback scenario is based on a
the ‘‘left’’ nor the ‘‘right’’ classes are significantly high. This Virtual Reality (VR) platform X-Rooms (www.x-rooms.de).
 ARTICLE IN PRESS
474 R. Krepki et al. / Int. J. Human-Computer Studies 65 (2007) 460–477

Fill line:
Indication of the
Intention to turn

Nose:
Proxim.
move
direction

Discrimination Line:
Indication of competing
laterality intentions

Fig. 19. Feedback scenario ‘‘Pacman’’ and the construction of Pacman’s head.

The user is instructed to control a part of the body that may

be absent and that is displayed on a computer screen. This
feedback application still remains an open research field that
requires prior solving of many neurological, technical and
fundamental research problems, as well as design and
conduction of further experiments with patients.
During the training session the user is instructed to
perform imagined movements of the arm, e.g., bending it at
the proximal (shoulder or elbow) and distal (hand or
fingers) joints, which are triggered by an external query.
During the application session the user is presented with a
black full-screen window containing an image of a human
arm. The BBCI is capable of recognizing the two spatio-
temporal brain patterns as belonging to two different
classes. The proximal joint flexion action is implemented by
the up and down movement of the entire arm, i.e., by
bending it in the shoulder or elbow joint. The distal
joint flexion actions then are implemented by the hand Fig. 20. Feedback scenario ‘‘Virtual Arm’’.
closure and opening movement, i.e., by finger movements,
cf. Fig. 20.
A future vision is that a patient not capable of
controlling her/his own limb (in this case an arm) could limb’’ that would be able to perform some simple, but
then gain control over a virtual limb in a natural way. As a important tasks in some difficult accessible environment.
result, if the patient’s Central Nervous System (CNS) is still Orthosis can be employed for users whose limbs are
intact and capable of generating the appropriate control present, yet who have no control of these limbs due to a
commands for bending and grasping, and if she/he is still neuromuscular disease. In this case, if the executive muscles
able to feel or imagine an amputated limb, the BBCI of the limb are still intact and usable to implement
system would serve as a bridge between the command movements, an electro-stimulator sleeve (e.g., an arm or
emission unit and the executing extremity, bypassing all the wrist band) is employed to process the user’s brain signals
intermediate interfacing elements such as spinal cord, encoded as control commands. The sleeve transforms
peripheral nerves and muscles. control signals from the Virtual Arm interface into an
For users with absent limbs, prostheses can be replaced electrical stimulation of limb muscles, such that the
by intelligent robotic mechanisms that can be controlled by imagination of a certain movement yields the execution
the user’s brain signals encoded as emitted commands. of the desired movement in the limb. The underlying intent
Furthermore, a Virtual Arm that is physically located at a of this feedback scenario was to investigate a virtual
remote place, but connected to the BBCI user via the version of such prosthesis and to define the control
Internet can be controlled easily by generating brain procedure for the orthosis. The technical realization of
activity patterns independent from those of a real arm such a prototype is a challenge to be met by experts in the
movement. This could provide the operator with a ‘‘third field of robotics and prosthesis development and research.
 ARTICLE IN PRESS
R. Krepki et al. / Int. J. Human-Computer Studies 65 (2007) 460–477 475

Experiments with patients are still to be conducted within respective neural correlate, but no means for an actual
the scope of future BBCI research. movement.
Another issue with pioneering appeal is the thrilling
5. Conclusion and discussion possibility that, because the BBCI bypasses the conduction
delays from brain to muscles, it could speed up the
BCIs have traditionally been conceived and used in initiation of actions in competitive, dual-player scenarios
assistance systems for the disabled, e.g., Wolpaw et al. or applications that require ultra-fast actions, like emer-
(1991, 2002) and Birbaumer et al. (1999) and many more. gency braking. However, the experimental design of this
We have shown in this contribution that our BBCI kind of application must differ to some extent from those
discovers also the interesting path towards interactive performed previously and presented within the scope of
applications; exemplified here as gaming and robotic arm this work. The conceptual scheme, design and implementa-
control. tion of such competitive scenarios as a feedback applica-
The field of HCI research is expanding to encompass tion will require considerable innovation.
brain signal-based communication and interaction. That Let us finally discuss how much information we can
trend had its onset when, among other factors, BCIs expect to transmit in such a new BCI channel. Invasive
introduced a new technique for reliably decoding technologies, like those proposed in Nicolelis and
brain signals and converting these into control commands, Chapin (2002) can achieve bit-rates that are high enough
which can be used for appealing interaction. Currently, the for, e.g., an online 3-D robot control (as discussed earlier),
two most prominent and promising paths of BBCI but require hundreds of microelectrodes implanted
application are rehabilitation, discovery and gaming, into the brain’s cortex, which is an unlikely condition for
although further application fields are conceivable, such healthy subjects. For non-invasive techniques our own
as the monitoring of the mental state of a patient or vehicle earlier studies have shown that in a pseudo-online
driver. The latter could be used to prevent overfatigue of idealized evaluation (i.e., data are recorded and analysed
truck drivers and guard against accidents. The bio- later as if online) record bit-rates of up to 50 bits per
feedback signals introduced in this contribution hold minute are possible (Blankertz et al., 2003). Our recent
potentials for further optimization, but already now allow experiments have shown that even higher transfer-bit
a user who has taken a ‘‘cold-start’’ to explore and discover rates can be achieved if several data processing models
her/his individual possibilities in using the new commu- are combined (Dornhege et al., 2004a, b). In spelling task
nication channel. Concerning rehabilitation the most experiments conducted by Wolpaw et al. (1991),
promising perspectives, e.g., for quadriplegic patients, are Pfurtscheller et al. (1993) and Birbaumer et al. (1999)
the intuitive use of mental motor commands to trigger that are truly online with bio-feedback, single subjects
goal-directed actions of mechanical devices, such as a can reach a level of 2–3 letters/min. A recent BBCI
motorized wheel-chair, or to use a ‘‘mental keyboard’’ for study even achieves 37 bits/min in a real-time feedback
typing messages. set-up (Blankertz et al., 2006). At first sight, this
In general, the question surrounding an ideal bio- might appear rather slow for a communication device, as
feedback signal for BBCI will find different answers other communication devices, e.g., a computer mouse can
appropriate for each new application. However, we achieve 300–350 bits/min (MacKenzie, 1991). Yet, one
propose that bio-feedback in an exciting gaming scenario should realize that a BCI communication channel is
(Pacman), or within a native VR environment (Virtual largely independent of other channels and offers a unique
Arm), can be realized more naturally and thus more feature of ultra-fast action emissions for each single
successfully. Eventually such bio-feedback can enable the reaction trial.
user to adapt to the classification engine and vice versa; the Finally, there is a strong agreement that BCI research
classification engine might find it easier to classify correctly will seek to develop new and more natural feedback modi
in the course of mutual adaptation (Krepki, 2004). and feedback applications rather than re-developing and
While most BCIs (except VEP or P300-based) require adapting well-known applications to be controlled by brain
extensive training (4200 h) from their users, it is one signals. The reason for that is because the latter were
distinctive feature of the BBCI that it employs advanced designed to rely on classical communication and control
signal processing and machine learning technology for strategies. Moreover, the field of human–computer inter-
training the computer rather than the human subject, such action will be increasingly in demand and will be called on,
that the user can start ‘‘communicating’’ without extensive in particular, to provide new techniques and communica-
prior training. There are several aspects for further tion protocols that can serve as a basis for BCI-based
improvement of BBCI: so far we have used a paradigm, communication. In conclusion, we discussed state-of-the-
where the user actually implements or imagines the art BCI research and presented recent results that could be
accomplishment of a movement, i.e., typing with the left achieved by providing interactive feedback to the user.
or right index or pinky fingers. In ongoing research we BCIs are able to open up new vistas for exploring own
transfer this paradigm to assistance systems where a brain skills and discovering new ways in human–computer
disabled person still has movement intentions and their communication.
 ARTICLE IN PRESS
476 R. Krepki et al. / Int. J. Human-Computer Studies 65 (2007) 460–477

Acknowledgements Harel, D., Carmel, L., Lancet, D., 2003. Towards an odor communication
system. Computational Biology and Chemistry 27, 121–133.
Many thanks to our colleagues at the Fraunhofer- Krepki, R., 2004. Brain–computer interfaces: design and implemen
tation of an online BCI-system for the control in gaming applications
FIRST: Guido Dornhege and Christin Schäfer; and at the and virtual limbs. In: Müller, K.-R., Jähnichen, S., Obermayer, K.
UKBF: Florian Losch and Volker Kunzmann who helped (Eds.), Doctoral Dissertation, Technische Universität, Berlin,
to develop the system described here. Partial funding by the Germany.
BMBF is gratefully acknowledged. Lang, W., Zilch, O., Koska, C., Lindinger, G., Deecke, L., 1989.
Negative cortical DC shifts preceding and accompanying simple and
complex sequential movements. Experiments in Brain Research 74,
References 99–104.
MacIntyre, B., Feiner, S., 1996. Future multimedia user interfaces.
Berger, H., 1929. Über das elektroenzephalogramm des menschen. Archiv Multimedia Systems Journal 4 (5), 250–268.
Psychiatrischer Nervenkrankheiten 87, 527–580. MacKenzie, I.S., 1991. Fitt’s law as a performance model in human–
Birbaumer, N., 1997. Slow Cortical Potentials: their origin meaning computer interaction. Doctoral Dissertation. University of Toronto,
and clinical use. Brain and Behavior: Past, Present and Future Canada.
25–39. Mika, S., Rätsch, G., Müller, K.-R., 2001. A mathematical programming
Birbaumer, N., Ghanayim, N., Hinterberger, T., Iversen, I., Kotchoubey, approach to the kernel fisher algorithm. Advances in Neural
B., Kübler, A., Perelmouter, J., Taub, E., Flor, H., 1999. A spelling Information Processing Systems 13, 591–597.
device for the paralyzed. Nature 398, 297–298. Mika, S., Rätsch, G., Weston, J., Schölkopf, B., Smola, A.J., Müller,
Birbaumer, N., Kübler, A., Ghanayim, N., Hinterberger, T., Perelmouter, K.-R., 2003. Learning discriminative and invariant nonlinear features.
J., Kaiser, J., Iversen, I., Kotchoubey, B., Neumann, N., Flor, H., IEEE Transactions on Pattern Analysis and Machine Intelligence 25
2000. The thought translation device (TTD) for completely paralyzed (5), 623–628.
patients. IEEE Transactions on Rehabilitation Engineering 8, Müller, K.-R., Mika, S., Rätsch, G., Tsuda, K., Schölkopf, B., 2001. An
190–192. introduction to kernel-based learning algorithms. IEEE Transactions
Blankertz, B., Curio, G., Müller, K.-R., 2002a. Classifying single trial on Neural Networks 12 (2), 181–201.
EEG: towards brain–computer interfacing. Advances in Neural Müller, K.-R., Anderson, C., Birch, B., 2003. Linear and nonlinear
Information Processing Systems 14, 157–164. methods for brain computer interfaces. IEEE Transactions
Blankertz, B., Schäfer, C., Dornhege, G., Curio, G., 2002b. Single trial on Neural Systems and Rehabilitation Engineering 11 (2),
detection of EEG error potentials: a tool for increasing BCI 165–169.
transmission rates. In: Proceedings of the International Conference Nicolelis, M.A.L., Chapin, J.K., 2002. Controlling robots with the mind.
on Artificial Neural Networks—ICANN’02, pp. 1137–1143. Scientific American 287 (4), 46–53.
Blankertz, B., Dornhege, G., Lemm, S., Krauledat, M., Curio, G., Müller, Orr, G., Müller, K.-R., 1998. Neural Networks: Tricks of the
K.-R., 2006. The Berlin Brain-Computer Interface: machine learning Trade. Lecture Notes in Computer Science, Springer, Berlin,
based detection of user specific brain states. Journal of Universal vol. 1524.
Computer Science 12 (6), 581–607. Pantic, M., Rothkrantz, L.J.M., 2000. Automatic analysis of facial
Blankertz, B., Dornhege, G., Schäfer, C., Krepki, R., Kohlmorgen, J., expressions. IEEE Transactions on Pattern Analysis and Machine
Müller, K.-R., Kunzmann, V., Losch, F., Curio, G., 2003. Boosting Intelligence 22 (12), 1424–1445.
bit rates and error detection for the classification of fast-pace Penfield, W., Rasmussen, T., 1950. The Cerebral Cortex of Man.
motor commands based on single-trial EEG analysis. IEEE Transac- MacMillan, New York.
tions on Neural Systems and Rehabilitation Engineering 11 (2), Pentland, A., 1995. Machine understanding of human action. In:
127–131. Proceedings of the Seventh International Forum Frontier of Tele-
Cui, R.Q., Huter, D., Lang, W., Deecke, L., 1999. Neuroimage of communication Technology, Tokyo, Japan, pp. 757–764.
voluntary movement: topography of the Bereitschaftspotential, Peters, B.O., Pfurtscheller, G., Flyvbjerg, H., 2001. Automatic differentia-
a 64-channel DC current source density study. Neuroimage 9, tion of multichannel eeg signals. IEEE Transactions on Biomedical
124–134. Engineering 48 (1), 111–116.
Dornhege, G., Blankertz, B., Curio, G., Müller, K.-R., 2004a. Boosting Pfurtscheller, G., Flotzinger, D., Kalcher, J., 1993. Brain–computer
bit-rates in non-invasive EEG single-trial classifications by feature interface—a new communication device for handicapped persons.
combination and multi-class paradigms. IEEE Transactions on Journal on Microcomputer Applications 16, 293–299.
Biomedical Engineering 51 (6), 993–1002. Pfurtscheller, G., 1999. EEG event-related desynchronization (ERD) and
Dornhege, G., Blankertz, B., Curio, G., Müller, K.-R., 2004b. Increase event-related synchronization (ERS). In: Electroencephalography:
information transfer rates in BCI by CSP extension to multi-class. Basic Principals and Clinical Applications, pp. 958–967.
In: Advances in Neural Information Processing Systems. NIPS’03, Poggio, T., Girosi, F., 1990. Regularization algorithms for learning
vol. 16. that are equivalent to multilayer nerworks. Science 247, 978–982.
Duda, R.O., Hart, P.E., Stork, D.G., 2001. Pattern Classification, second Quek, F., McNeill, D., Bryll, R., Duncan, S., Ma, X.-F., Kirbas, C.,
ed. Wiley, New York. McCullough, K.E., Ansari, R., 2002. Multimodal human discourse:
Donchin, E., Smith, D.B., 1970. The contingent negative variation and the gesture and speech. ACM Transactions on Computer–Human Inter-
late positive wave of the average evoked potential. Electroencephalo- action 9 (3), 171–193.
graphy and Clinical Neurophysiology 29, 201–203. Ramachandran, V.S., Altschuler, E.L., Stone, L., Al-Aboudi, M.,
Ebrahimi, T., Vesin, J.-M., Garcia, G., 2003. Brain–computer interface—a Schwartz, E., Siva, N., 1999. Can mirrors alleviate visual hemineglect?
new frontier in multimedia communication. Signal Processing Maga- Medical Hypotheses 52 (4), 303–305.
zine 20 (1), 14–24. Rätsch, G., Onoda, T., Müller, K.-R., 2001. Soft margins for AdaBoost.
Green, J.B., Sora, E., Bialy, Y., Ricamato, A., Thatcher, R.W., 1999. Machine Learning 42 (3), 287–320 (also NeuroCOLT Technical
Cortical motor reorganization after paraplegia: an EEG study. Report NC-TR-1998-021).
Neurology 53 (4), 736–743. Schölkopf, B., 1997. Support Vector Learning. Oldenburg Verlag,
Hardwick, A., Rush, J., Furner, S., Seton, J., 1996. Feeling it as well as Munich.
seeing it. Haptic displays within gestural HCI for Multimedia. In: Sharbrough, F., Chatrian, G.-E., Lesser, R.P., Lüders, H., Nuwer, M.,
Proceedings of the York Gesture Workshop, pp. 105–116. Picton, T.W., 1991. American electroencephalographic society guidelines
 ARTICLE IN PRESS
R. Krepki et al. / Int. J. Human-Computer Studies 65 (2007) 460–477 477

for standard electrode position nomenclature. Journal of Clinical Vapnik, V., 1995. The Nature of Statistical Learning Theory. Springer,
Neurophysiology 8, 200–202. New York.
Steriadis, C.E., Constantinou, P., 2003. Designing human–computer Wolpaw, J.R., McFarland, D.J., Neat, G.W., Forneris, G.W., 1991. An
interfaces for quadriplegic people. ACM Transactions on Computer– EEG-based brain–computer interface for cursor control. Electroence-
Human Interaction 10 (2), 87–118. phalography and Clinical Neurophysiology 78, 252–259.
Sutter, E.E., 1992. The brain response interface: communication through Wolpaw, J.R., Birbaumer, N., McFarland, D.J., Pfurtscheller, G.,
visually induced electrical brain responses. Journal on Microcomputer Vaughan, T.M., 2002. Brain–computer interfaces for communication
Applications 15, 31–45. and control. Clinical Neurophysiology 113, 767–791.