Handbook of Clinical Neurology, Vol.

168 (3rd series)

Brain-Computer Interfaces
N.F. Ramsey and J. del R. Millán, Editors
https://doi.org/10.1016/B978-0-444-63934-9.00008-1
Copyright © 2020 Elsevier B.V. All rights reserved

Chapter 8

Applications of brain-computer interfaces to the control

of robotic and prosthetic arms

MARCO VILELA1 AND LEIGH R. HOCHBERG1,2,3*

1
School of Engineering and Carney Institute for Brain Science, Brown University, Providence, RI, United States
2
Center for Neurorestoration and Neurotechnology, Veterans Affairs Medical Center, Providence, RI, United States
3
Center for Neurotechnology and Neurorecovery, Massachusetts General Hospital, Harvard Medical School, Boston, MA,
United States

Abstract
Brain-computer interfaces (BCIs) have the potential to improve the quality of life of individuals with
severe motor disabilities. BCIs capture the user’s brain activity and translate it into commands for the con-
trol of an effector, such as a computer cursor, robotic limb, or functional electrical stimulation device. Full
dexterous manipulation of robotic and prosthetic arms via a BCI system has been a challenge because of the
inherent need to decode high dimensional and preferably real-time control commands from the user’s neu-
ral activity. Nevertheless, such functionality is fundamental if BCI-controlled robotic or prosthetic limbs
are to be used for daily activities. In this chapter, we review how this challenge has been addressed by BCI
researchers and how new solutions may improve the BCI user experience with robotic effectors.

Nicolas-Alonso and Gomez-Gil, 2012; Chaudhary et al.,

INTRODUCTION
2015; Schwartz, 2016). Toward this goal, BCI systems
Several injuries and disorders, including severe amyo- are engineered to interface between the user’s voluntarily
trophic lateral sclerosis (ALS), brainstem stroke, and modulated neural activity and the assistive technology of
spinal cord injury, can leave individuals with limited or their choice. Fig. 8.1 shows the three major components
no voluntary limb muscle control (K€ ubler et al., 2001) of a typical BCI system: a sensor to collect the neural activ-
(Chapters 3, 4, and 6). People with these nervous system ity, an algorithm to decode the recorded neural activity, and
disorders are usually cognitively intact but require the an effector which performs the final intended actions. Raw
continuous presence of a caregiver to perform routine voltage signals are recorded either by placing electrodes
daily activities. In some cases, even the ability to commu- implanted directly into or on top of the brain, or via elec-
nicate is completely lost, highlighting the importance of trodes placed on the scalp. These signals reflect the user’s
the problem and reinforcing the search for a solution. The brain activity, usually from motor-related areas of the cor-
field of neuroengineering has proposed the use of brain- tex, and therefore, they contain information about the user’s
computer interfaces (BCIs) to provide people with severe motor intentions. A series of signal processing techniques
speech and motor impairments with the means to commu- are applied to these recorded voltages in order to increase
nicate and to interact with their own surrounding environ- signal-to-noise ratio (SNR) and facilitate the extraction
ment (Birbaumer et al., 1999; Lebedev and Nicolelis, 2006; of meaningful information (Andersen et al., 2004).
Donoghue, 2008; Chestek et al., 2009; Hatsopoulos Different informative features, such as neuronal firing
and Donoghue, 2009; Green and Kalaska, 2011; rate or the power of field potentials (FPs), can be

*Correspondence to: Leigh R. Hochberg, MD, PhD, 131 Waterman St, Providence, RI, 02912, United States. Tel: +1-401-863-6944,
Fax: + 1-617-643-3939, E-mail: [email protected]
88 M. VILELA AND L.R. HOCHBERG

Fig. 8.1. Basic components of a BCI system for robotic arm control. The user’s neural activity is captured by a sensor that can be
placed in or on the brain or distributed over the scalp. Recorded signals are sent to a decoder, which translates this information into
commands that are passed to a robotic arm, which in turn, performs the intended action. Visual feedback allows the user to con-
tinuously intervene and correct the effector motion; recently somatosensory feedback has been incorporated as well.

extracted from the recorded raw voltages depending upon continuously processed and fed to the decoder, which,
where the raw signal is collected. Because of the physical in turn, maps the extracted neural features into control
proximity to the signal source, intracortical electrodes commands for an external device. Many decoding strate-
(electrodes placed directly into the cortex) can provide gies have been employed for BCI use, and they are pred-
the most informative type of signals, ranging from single icated on the amount of information present in the input
neuron spiking to the full FP spectrum (Donoghue, 2008) signal. Kinematic decoders are usually linear or nonlinear
(Chapter 20). The electroencephalogram (EEG), recorded “black boxes” that take neural features and map them into
by electrodes placed on the scalp, has also shown to be a position or velocity vectors. Alternatively, discrete direc-
feasible source for BCI control although it carries less tion of motion can also be decoded by classification algo-
information than intracortical recordings (Lotte et al., rithms (Chen et al., 2019). In this case, the direction is
2015) (Chapter 18). Features extracted from raw voltages decoded from the neural activity, and the kinematic step
correlate with not only motor parameters such as hand is preset in the logic of the BCI system. Analogous to
velocity but also with high-level cognitive parameters kinematics, grasp decoding can be performed either by a
such as the expected value and magnitude of a reward classifier or continuously through a mapping function.
(Musallam et al., 2004; Pesaran et al., 2006). Although The former initiates a predetermined grasp motion
these correlations have been shown to be useful for (Hochberg et al., 2012), and the latter continuously sets
BCI control, continued research is elucidating the rela- the aperture of the robotic gripper (Wodlinger et al., 2015).
tionship between neural features and the intended motion A number of different assistive devices including a
(Shenoy and Carmena, 2014). In addition to electrical wheelchair, computer cursor, and robotic limb have been
measurements, brain monitoring technologies such as reported in the literature as end effectors controlled by
magnetoencephalography (MEG) (Mellinger et al., BCI users (Hochberg et al., 2006; Carlson and Millán,
2007) or functional magnetic resonance imaging (fMRI) 2013; Collinger et al., 2013; Jarosiewicz et al., 2015).
(Sitaram et al., 2007) (Chapter 21) can also be used as part In this chapter, we focus on past and current develop-
of a BCI system. However, these techniques are com- ments of the use of BCI systems with robotic and pros-
monly restricted to laboratory use because of their sub- thetic arms. The idea of using neural activity from the
stantial expense and lack of portability. motor cortex to control external devices was postulated
After the neural signals are recorded, extracted neural over 50 years ago (Frank, 1968). Soon after, a series of
features are translated into volitional actions by a com- experiments performed in different laboratories demon-
putational framework often referred to as a decoder. strated that nonhuman primates could modulate single
In real-time applications, recorded neural signals are cortical neuron firing rates in order to control a radio
 BRAIN-COMPUTER INTERFACES FOR ROBOTIC CONTROL 89
meter or simple light switch device, and that force could RECORDING NEURAL ACTIVITY
be decoded from the firing rates (Fetz, 1969; Humphrey
Determining the physical location where the brain activ-
et al., 1970; Schmidt, 1980). Almost two decades later,
ity will be recorded is an important step for the design of a
rodents implanted with microelectrode arrays in the pri-
BCI system. Most BCI systems rely on EEG to record
mary motor (MI) cortex and ventrolateral (VL) thalamus
brain signals to drive the output of the system
were shown to control a one-dimensional robotic lever
(Wolpaw et al., 2002; Lotte et al., 2015). This technique
by modulating the population activity of the recorded
measures FP activity of the cortex through electrodes
neurons (Chapin et al., 1999). Subsequent demonstra-
spatially distributed over the user’s scalp. Therefore, this
tions of two- and three-dimensional control of virtual
signal is the result of the aggregated activity of millions
and physical devices using the activity of cortical neu-
of neurons. Beyond the low spatial resolution provided
rons in nonhuman primates (Serruya et al., 2002;
by this technique, EEG recordings are also susceptible
Taylor et al., 2002; Carmena et al., 2003; Velliste
to muscle activity artifacts (Nicolas-Alonso and
et al., 2008) helped to set the stage, first for the launch
Gomez-Gil, 2012) and they contain a limited bandwidth
of pilot clinical trials of intracortical BCIs and later for
due to scalp and skull filtering (Pfurtscheller and Cooper,
dexterous robotic arm control.
1975). Nevertheless, EEG recordings have been success-
The early animal studies of BCIs integrated with
ful in generating slow three-dimensional control of vir-
robotic/prosthetic arms laid the foundation for further
tual or physical robotic arms (Mcfarland et al., 2010;
development of systems designed for human applica-
Meng et al., 2016).
tions. In order to make full use of these technologies, a
A more informative recording modality is electrocor-
BCI should be able to decode the movement of the
ticography (ECoG), which also records FPs but by using
arm in three-dimensional space as well as the desired
electrodes placed subdurally on the cortical surface
grasp state of the robotic system. Focusing on the clinical
(Crone et al., 1998a,b; Leuthardt et al., 2006). This tech-
needs of people with tetraplegia, intracortical control of a
nique is commonly deployed in a clinical setting to iden-
robot arm by a human was initially shown in 2006
tify epileptic foci in patients suffering from medically
(Hochberg et al., 2006), followed by multiple studies
refractory seizures. Typically, electrodes are arranged
demonstrating the ability of people with paralysis to
on a flexible grid or strip with center-to-center spacing
use BCIs for the control of a computer cursor and more
varying between 10 mm for macro-ECoG and 1 mm for
dexterous robotic arms, including limbs prosthetic or
micro-ECoG (Thongpang et al., 2011). A craniotomy
robotic arm systems capable of supporting self-feeding
is required to place the grip or strip directly over the cor-
(Hochberg et al., 2012; Collinger et al., 2013;
tex, and the signals are brought out by a wire, although
Pandarinath et al., 2017; Nuyujukian et al., 2018)
fully implantable wireless systems have been developed
(Chapters 13 and 22).
(Matsushita et al., 2013; Vansteensel et al., 2016).
In more recent years, upper limb robotic devices con-
Because of the proximity to the signal source, ECoG
trolled by BCI systems have been used for rehabilitation
recordings have a wider frequency content, a higher
of patients after stroke (Daly and Wolpaw, 2008; Ang
SNR and better spatial resolution compared with EEG
et al., 2010; Várkuti et al., 2013; Venkatakrishnan
(Schalk and Leuthardt, 2011).
et al., 2014; Soekadar et al., 2015; Van Dokkum et al.,
Both EEG and ECoG recordings reflect the electrical
2015) (Chapter 9). In this endeavor, patients learn to
activity of a population of cells in the brain. A different
modulate their neural activity to control a robotic system
and more fundamental electrical signal generated in the
which guides the paretic arm toward the desired motion,
brain is the action potential of individual neurons
promoting neural plasticity and the rehabilitation of nat-
(Buzsáki et al., 2012) (Chapter 20). These brief signals
ural limb movement.
(
1 ms) propagate information within a network of neu-
In the following section, we describe the applications
rons and can be measured by inserting microelectrodes
of BCI systems to control the positioning and grasp
directly into the cortex. More than 50 years of neurophys-
states of robotic/prosthetic arms. Starting with the
iology research has made use of intracortical electrode
recording modality, we explore the basic characteristics
recordings to elucidate basic neuroscience. For instance,
of the implanted and scalp surface recording methods,
early studies showed that cortical neurons in monkeys
discussing their limitations and advantages. Several
increase their firing rate when performing a specific
decoding algorithms employed with robotic systems
motor task (Evarts, 1968) and that monkeys could voli-
are also discussed. The idea of shared autonomy is iden-
tionally modulate the firing rates of individual neurons to
tified as a new development in the BCI robotic field,
gain food rewards (Fetz, 1969). This rich body of litera-
and we conclude the chapter with potential challenges
ture sparked the development of current intracortical BCI
and directions for further development of these
systems. Today, four different types of electrodes are
applications.
90 M. VILELA AND L.R. HOCHBERG
available for intracortical neurophysiology recordings: carried by the recorded signals (Chao et al., 2010). It is
microelectrode arrays, microwires, multisite probes, also important to point out that most of human ECoG
and cone electrodes (Donoghue, 2008). All of these BCI research to date has been performed with the assis-
technologies have been used to record neural activity tance of patients with epilepsy who are undergoing clin-
in animals, but only microarrays and cone electrodes ically indicated brain monitoring; thus, the positioning of
have been deployed chronically in human pilot clinical the grids/strips on the cortex and the duration of the
studies (Kennedy et al., 1992; Kennedy and Bakay, recoding periods are dictated by the clinical needs, lim-
1998; Kennedy et al., 2000; Hochberg et al., 2006; iting the exploration of the capability of these signals for
Simeral et al., 2011b; Churchland et al., 2012; Collinger BCI technology (Leuthardt et al., 2004b). Although it has
et al., 2013; Aflalo et al., 2015; Bouton et al., 2016; higher spatial resolution compared with EEG, ECoG sig-
Ajiboye et al., 2017). In addition to the spiking activity nals still lack the resolution to measure single cell
of individual neurons, intracortical electrodes can also activity. Intracortical electrodes also require surgery
measure cortical FPs from small volumes of tissue that and penetrate the pia, but they contain the most
are then referred as local potential fields (LFPs). This information-rich neural signal available for BCI use. It
information can be extracted from the same raw voltages has the highest spatial resolution, and it can capture
used to detect the spikes but are processed with different the firing of single neurons as well as the LFP activity.
filtering settings. Day to day variability of microelectrode array signals
The region of the cortex to be implanted is an impor- in humans have been characterized (Perge et al., 2013),
tant decision associated with the choice of intracortical and its long-term use has also been demonstrated
electrodes as sensors. The primary motor cortex (MI) (Simeral et al., 2011a; Hochberg et al., 2012). Neverthe-
has been identified as a major contributor to planning less, more conclusive studies about device function
and execution of volitional motor movements (Scott, (Barrese et al., 2013; Barrese et al., 2016), and the influ-
2003), making this region a prime candidate for BCI ence of tissue reaction on the recorded signals still is a
applications. Other sites like the parietal reach region topic of active discussion within the BCI scientific
(PRR) (Musallam et al., 2004) and the ventrolateral thal- community.
amus have also been explored within the BCI context
(Chapin et al., 1999). Firing rates from MI, dorsal premo-
DECODING NEURAL ACTIVITY
tor cortex (PMd), supplementary motor area (SMA), pos-
terior parietal cortex (PP), and primary somatosensory The major challenge of a BCI system is to map the user’s
cortex (S1) have been used to predict hand position neural activity into useful commands for control of an
and velocity, grasp force and EMG traces (Carmena external (or other implanted) device. The mapping in
et al., 2003). This work concluded that all implanted cor- such systems is performed by a decoder. In a real-time
tical areas presented some level of information about system, features extracted from the recorded neural sig-
motor parameters; however, the number of neurons nals are provided as a constantly changing input to the
required to achieve a given performance in movement decoder, which then interprets these inputs and acts on
prediction was different for each area, with M1 being a the effector to produce the intended action. Here, we
particularly rich location for predicting movement veloc- focus on decoding architectures applied to robotic/pros-
ity. Ongoing research will reveal the extent to which thetic arm systems. In this scenario, the decoders are
homologous areas in humans present similar properties. responsible for the positioning of the arm in space and
All three recording modalities described earlier have defining its grasping state.
advantages and disadvantages. EEG-based recordings A wide range of BCI applications have used EEG
are low cost and stable over time (Lebedev and recordings to control external devices (Millán et al.,
Nicolelis, 2006) and do not require surgery. However, 2004; Horki et al., 2011; Lotte et al., 2015). The two-
this method provides a limited bandwidth signal that is dimensional control of a computer cursor used the ampli-
also susceptible to corruption from electromyography tude of specific frequencies as inputs to a continuous
(EMG) and electrooculography (EOG) activity (Lotte linear decoder (Wolpaw and McFarland, 2004). Partici-
et al., 2015). Compared to EEG signals, ECoG record- pants in this study learned to modulate the amplitude
ings have better SNR and wider bandwidth. This of two frequencies, 12 and 24 Hz, in order to move
surgery-requiring method provides clear amplitude the cursor horizontally and vertically, respectively.
information for higher frequency bands (gamma, A similar approach was extended to three-dimensional
>30 Hz), which carry relevant motor information control (Mcfarland et al., 2010). In this work, partici-
(Schalk and Leuthardt, 2011). Experiments with nonhu- pants used left-hand, right-hand, and foot movement
man primates have demonstrated the long-term use of imagery to control a cursor in a 3D virtual environ-
this technology and the stability of the information ment. EEG recordings have also been used to control
 BRAIN-COMPUTER INTERFACES FOR ROBOTIC CONTROL 91
the grasping states of a hand functional electrical neuro- movements and grasp (Yanagisawa et al., 2011, 2012;
prosthesis (Lauer et al., 1999). This study used the b band Pistohl et al., 2012; Shimoda et al., 2012; Pistohl et al.,
(25–28 Hz) amplitudes extracted from the EEG signals as 2013; Wang et al., 2013; Bundy et al., 2016; Hotson
input to a simple threshold classifier to determine the open et al., 2016), few ECoG-based BCIs with prosthetic/
and closed states of the hand. Although EEG based BCIs robotic arm effectors have been implemented. A linear
have been developed for 3D control of various effectors discriminant analysis (LDA) and a threshold classifier
(Royer et al., 2010; Doud et al., 2011; LaFleur et al., that used the average power of the high gamma band
2013), applications to robotic/prosthetic arms are limited. (72.5–110 Hz) have been used to execute simple reach-
Support vector machine (SVM) methods were used to and-grasp movements (Fifer et al., 2014), with different
control an industrial robotic arm in two dimensions activation of motor regions identified when participants
through the classification of four different mental states attempted reach only, grasp only, and the two motions
(Hortal et al., 2015). In a more complex implementation, simultaneously.
an EEG-based BCI system allowed participants to per- Several applications of intracortical recordings and
form reach-and-grasp tasks in a three-dimensional work- different effectors, including robotic/prosthetic arms,
space (Meng et al., 2016). A two-stage strategy was have been proposed (Chapin et al., 1999; Wessberg
employed where participants first controlled the arm in et al., 2000; Serruya et al., 2002; Taylor et al., 2002;
a two-dimensional plane above the targets, and once the Hochberg et al., 2006) (Chapters 13 and 22). The idea
arm gripper was over the desired one, participants moved of decoding volitional control commands from spiking
the arm in the vertical direction to position the arm’s hand activity evolved from nonhuman primate experiments
and consequently grasp the target. The decoding in this that showed the existence of a relationship between fea-
work was based on amplitude changes effected by tures extracted from recorded neural signals and param-
event-related de/synchronization (ERD-ERS). More spe- eters related to the execution of voluntary movements
cifically, they first used an autoregressive model (AR) (Fetz, 1969; Humphrey et al., 1970). A fundamental
to estimate the amplitude of the upper mu band set of papers published in the early 1980s showed that
(10–14 Hz). This power was then linearly mapped into different cortical neurons have specific directional
velocities used to control the robotic arm. Participants responses, peaking their firing rates when the volitional
imagined moving their left hand, right hand, and both motion had a specific direction and decreasing it when
hands, and relaxation of both hands to move the arm left, the direction of the movement changed (Georgopoulos
right, forward (and up), and backward (and down). The et al., 1982; Georgopoulos et al., 1986). These experi-
forward/backward motion was replaced by the up/down ments gave rise to the idea that primary motor cortex neu-
once the arm’s hand was positioned over a target for 2 s. rons have a preferred direction (PD) and that their firing
It has also been shown that high accuracy can be rates are cosine tuned, which to date is the basis for many
achieved by discretizing the plane covering the robotic of the linear decoder strategies used in intracortical BCI
arm working area and mapping specific frequencies to applications.
specific positions of the discrete plane (Chen et al., More recently, nonlinear algorithms have also been
2019). The nonintuitive nature of these imageries is a deployed as decoders of neuronal ensemble firing rates.
hallmark of EEG BCIs, given the large spatial summa- For instance, neural networks have been used to demon-
tion that contributes to the signal detected by scalp- strate the movement of a robot wrist using intracortical
based electrodes. signals recorded from a nonhuman primate (Burrow
The viability of ECoG recordings as input signals to a et al., 1997). A similar neural network architecture was
BCI system was first demonstrated for one-dimensional also used to map the spiking activity of rats into control
control of a computer cursor (Leuthardt et al., 2004a). signals for the positioning of a one-dimensional lever
This work was based on earlier ECoG recordings in mon- (Chapin et al., 1999). Although nonlinear models have
keys that demonstrated its correlation with motion higher capacity to generalize the structure of a dataset,
parameters, including speed and direction (Moran and one comparative study reported similar performance of
Schwartz, 1999). A more recent study used ECoG a linear model and an artificial neural network when
recordings and two linear SVM classifiers running applied to the control of a remote robotic arm by
sequentially to decode three hand states: open, closed, monkeys (Wessberg et al., 2000). Also, from the nonhu-
and scissor shaped (Yanagisawa et al., 2011). More man primate literature, a Wiener filter was implemented
recently, a hierarchical linear discriminate analysis clas- for the three-dimensional control of a robotic arm
sifier was used to control a robotic arm’s individual (Carmena et al., 2003). The authors reported offline anal-
fingers from ECoG recordings (Hotson et al., 2016). ysis with different decoding algorithms like the Kalman
Although a number of studies have shown the feasibility filter, normalized least-mean squares filter, and an artifi-
of using ECoG recording to decode three-dimensional cial neural network. All of them underperformed when
92 M. VILELA AND L.R. HOCHBERG
compared with the Wiener filter. These findings sup- Moving a robotic arm in 3D space and determining its
ported the use of a linear, more interpretable, and compu- gripper state requires control signals for all independent
tational inexpensive algorithm as a solution to the movements. Therefore, the recorded neural activity has
decoding problem. As an example, the firing rates of to carry enough information to allow the decoder to dis-
monkeys implanted with microelectrode arrays were cern among all degrees-of-freedom (DoF) required for
used with an adaptive linear decoder to control a com- full control of the arm. To date, this requirement has been
puter cursor in three dimensions (Taylor et al., 2002). a challenge for most of EEG- and ECoG-based BCIs
This work made use of the closed-loop paradigm where because of the restriction of extracting neural features
visual feedback was provided to the monkeys during only from the signal frequency domain. Such restrictions
neural control of the computer cursor. Based on the neu- impose longer training times where participants need to
rons’ preferred direction, the population vector algorithm learn how to modulate the amplitude of specific fre-
was idealized and applied to decode a robotic arm end quency bands. Also, the imagery used by participants
point velocity and the gripper’s aperture (Velliste et al., does not reflect a natural muscle movement, although
2008). This algorithm sums up the preferred direction it provides a viable decoding strategy. Intracortical elec-
vectors of the neural ensemble weighted by the individ- trodes are capable of recording much richer signals,
ual neurons firing rates to provide the continuous three- which can provide enough information for high-
dimensional decoded velocities. It is likely that more dimensional decoding. This recording modality has
modern nonlinear machine learning approaches, com- shown its capability for fluid 3D control where subjects
bined with lower-cost computational power, will again can use natural motion imagery to control prosthetic/
challenge linear decoders as optimal for enabling the robotic systems, as shown in Fig. 8.2. As a reference,
complex reach-and-grasp movements intended by BCI Table 8.1 gives an overview of the three recording
users. modalities described in the preceding paragraphs,
Intracortical recordings have also been explored in accompanied by the decoding strategy employed and
human clinical trials (Kennedy et al., 2000; Hochberg DoF achieved by several BCIs.
et al., 2006; Collinger et al., 2013; Aflalo et al., 2015;
Gilja et al., 2015; Bouton et al., 2016; Ajiboye et al.,
Shared Autonomy
2017). In the first human application of an intracortical
BCI, a linear filter was used to decode the firing rate One of the latest developments in the integration of
activity collected from a microelectrode array implanted robotic/prosthetic arms to BCI systems is the idea of
in the arm/hand area of the primary motor cortex of a per- shared autonomy, which has been long appreciated in
son with C3–C4 spinal cord injury. The participant was the robotics research community (Millán et al., 2010;
able to use a computer cursor, open and close a prosthetic Allison et al., 2012; Tang and Zhou, 2018). In such sys-
hand, and use a robotic arm. The robotic arm control was tems, the control of the effector is continually shared
derived from 2D cursor control where robotic shoulder between the BCI user’s brain activity and intelligent
rotation, elbow flexion and extension, gripper opening external sensors that have some degree of awareness of
and closing were mapped onto five different targets on a the environment and/or the end result of the desired task.
computer screen. Six years later, an intracortical BCI For instance, in a reach-and-grasp task, the user could
system that allowed two people with severe paralysis to specify the target location, and an external controller
perform 3D reach-and-grasp tasks with a robotic arm would calculate the arm’s path to the target, making sure
(Hochberg et al., 2012) was demonstrated. The authors the robotic arm avoids obstacles and performs the task in
used a Kalman filter to decode the arm kinematics and minimal time. It should be noted that the idea of shared
an LDA classifier to control the grasp states of the arm’s autonomy is not to create an autonomous system but
gripper. A person with two microelectrode arrays was able rather to exempt the user from demanding portions of
to control six-dimensional movements of a robotic arm and the task. This is similar to the conscious experience of
open/close its gripper (Collinger et al., 2013). The authors an able-bodied reach and grasp, where the decision to
implemented a linear decoder to extract three-dimensional reach, for example, for a coffee cup, is explicit, but the
translation, three-dimensional orientation, and one-dimen- coordinated action of the limb muscles is performed by
sional grasp from the participant’s firing rates. This work CNS motor programs that do not require attention and
was followed by a more complex implementation where explicit conscious control. This strategy allowed, in non-
the simple one-dimensional grasp was replaced by four- human primates, the three-dimensional control of a
dimensional control of the robotic hand (Wodlinger robotic arm to be shared between the decoded brain
et al., 2015). A similar linear decoder was used, augmented activity and the information from proximity sensors
with a hand that could pinch, scoop, finger abduct, adduct, placed on the arm’s gripper (Kim et al., 2006). The sen-
and extend the thumb. The authors concluded that most sors provided extra information to avoid collision with
recorded units were tuned in all 10 dimensions. obstacles and to bias the robot arm’s motion and grasp
 BRAIN-COMPUTER INTERFACES FOR ROBOTIC CONTROL 93

Fig. 8.2. BrainGate clinical trial participant S3 performing a reach-and-grasp task for self-feeding with the DLR robotic arm. From
Hochberg, LR, Bacher, D, Jarosiewicz, B et al., (2012). Reach and grasp by people with tetraplegia using a neurally controlled
robotic arm. Nature 485 372–375. doi: 10.1038/nature11076, with permission.

state when an object was detected within a specific dis- have a wider bandwidth compared with EEG, carrying
tance; a substantial increase in performance was noted motor information over high frequency bands, but they
with the addition of the proximity sensors. Radio- are still limited by providing only frequency domain
frequency identification (RFI) tags have also been used information. It is still an open question whether the
to determine the precise location of the targets for a microelectrode recordings will remain stable and reliable
robotic arm BCI system (Úbeda et al., 2013). Using an and whether such systems will function ideally over
LDA classifier to decode the EEG signals, users steered decades. All of the challenges so far mentioned are
the robotic arm toward a target to a point the classifier had propelling the development of new wireless, fully
identified as the correct target. Then, an autonomous con- implantable systems and sensors (Borton et al., 2013;
troller finished the task by driving the arm to the target Yin et al., 2014; Lee et al., 2018) with wider coverage
and grasping it. of cortex areas (Seo et al., 2016; Diaz-Botia et al.,
ECoG recordings have also been used in shared con- 2017; Khanna et al., 2017).
trol BCI application (McMullen et al., 2014). This sys- Shared autonomy and hybrid architectures have
tem used computer vision to identify objects in the recently been suggested as possible paths to increase
robotic arm workspace, eye tracking to select the target, BCI performance when controlling robotic systems.
and ECoG signals to initiate the task. In all examples dis- Hybrid BCIs make use of additional signals collected
cussed in the preceding paragraphs, the integration of a from the user, such as EMG and eye gaze, to control spe-
shared control strategy improved the performance of cific tasks or degrees-of-freedom of the effector (Allison
BCI users in a reach-and-grasp task. et al., 2012; McMullen et al., 2014; Wang et al., 2015;
Tang et al., 2016). Object identification from the user’s
workspace captured by a video camera (Downey et al.,
CONCLUDING REMARKS AND FUTURE 2016) or proximity sensors mounted on the robotic
CHALLENGES arm are examples of additional information that can be
Significant progress has been made in neural engineering used in shared autonomy approach to BCI control of
in the past 20 years. Within the recording modalities robot arms (Kim et al., 2006; McMullen et al., 2014).
described in this chapter, EEG-based BCIs have the low- As a future alternative to these strategies, one could envi-
est cost and have been deployed for clinical applications; sion a decoding algorithm that receives input from the
however, evidence that this recording modality provides additional (nonneural) sensor. In this scenario, the
enough information for the independent and clinically decoder could, for instance, decrease the probability of
useful control of high-dimensional systems has yet to issuing a grasp command in the absence of a target near
be proven. Signals recorded using ECoG grids or strips the gripper or could increase the weight of a decoded
Table 8.1
Examples of published BCI systems used for reach and/or grasp

Publication Organism Recording modality Region of the brain Decoder strategy Effector Achieved DoF

Chapin et al. (1999) Rodent Intracortical MI and VL thalamus Artificial Neural Networks (ANN) Robotic lever 1D
Wessberg et al. (2000) Aotus trivirgatus Intracortical MI, PMd, and PP Linear filer and ANN Remote robotic arm 3D
Taylor et al. (2002) Macaca mulatta Intracortical MI and PMd Linear filter with adaptive weights Virtual cursor 3D
Carmena et al. (2003) Macaca mulatta Intracortical MI, PMd, SMA, S1 and PP Wiener filter Robotic arm 3D + grasp
Velliste et al. (2008) Macaca mulatta Intracortical MI Linear Population vector Robotic arm 3D + grasp
Hochberg et al. (2006) Human Intracortical Precentral gyrus Linear decoder Robotic arm 2D mapped onto 3D
Hochberg et al. (2012) Human Intracortical Precentral gyrus Kalman filter + LDA Robotic arm 3D + grasp
Collinger et al. (2013) Human Intracortical MI Linear filter Robotic arm 6D + grasp
Wodlinger et al. (2015) Human Intracortical MI Linear filter Robotic arm 6D + 4D grasp
Wang et al. (2013) Human ECoG Hand and arm area of the left Linear filter Virtual environment 3D
sensorimotor cortex
Fifer et al. (2014) Human ECoG Right frontal–parietal regions; LDA and threshold classifier Prosthetic arm 2D (reach and/or grasp)
right lateral occipital cortex;
dorsolateral prefrontal cortex
Hotson et al. (2016) Human ECoG Central sulcus area Hierarchical linear discriminant Prosthetic arm Individual fingers
analysis classifier
Lauer et al. (1999) Human EEG Threshold based classifier Prosthetic hand 1D grasp
McFarland et al. (2010) Human EEG Continuous linear filter Virtual environment 3D
Hortal et al. (2015) Human EEG SVM Industrial robot 2D
Meng et al. (2016) Human EEG Linear filter Robotic arm 3D + grasp
Chen et al. (2019) Human EEG Filter bank canonical correlation Robotic arm 2D
 BRAIN-COMPUTER INTERFACES FOR ROBOTIC CONTROL 95
velocity vector when this vector points toward a detected controlled muscle stimulation in a person with tetraplegia:
target on the user’s workspace. A most exciting opportu- a proof-of-concept demonstration. Lancet 389: 1821–1830.
nity for BCI-based control of robotic or prosthetic limbs https://doi.org/10.1016/S0140-6736(17)30601-3.
will be the incorporation of somatosensory information, Allison BZ, Leeb R, Brunner C et al. (2012). Toward smarter
BCIs: extending BCIs through hybridization and intelligent
so that the user, through a direct interface to the user’s
control. J Neural Eng 9: 13001–13007. https://doi.org/
nervous system, is aware of the moment and context in
10.1088/1741-2560/9/1/013001.
which a robotic effector comes into contact with an Andersen RA, Musallam S, Pesaran B (2004). Selecting the
object (Flesher et al., 2016) (Chapter 13). signals for a brain–machine interface. Curr Opin
Besides technical issues about sensors and decoding Neurobiol 14 (6): 720–726. https://doi.org/10.1016/j.conb.
algorithms, simple questions about the daily usability 2004.10.005.
of BCI systems in typical home environments are still Ang KK, Guan C, Chua KSG et al. (2010). Clinical study of
open. For instance, context and calibration problems neurorehabilitation in stroke using EEG-based motor imag-
could arise with the interchangeability of effectors, par- ery brain-computer interface with robotic feedback. Conf
ticularly if the effectors have different dimensionalities. Proc IEEE Eng Med Biol Soc2010: 5549–5552. https://
One could imagine the situation where a BCI user doi.org/10.1109/IEMBS.2010.5626782.
Barrese JC, Rao N, Paroo K et al. (2013). Failure mode anal-
changes effectors during operation, for example, from
ysis of silicon-based intracortical microelectrode arrays in
controlling a computer cursor to controlling a robotic
non-human primates. J Neural Eng 10 (6): 066014. https://
arm. Although one could apply the reduced form of a doi.org/10.1088/1741-2560/10/6/066014.
3D decoder to a 2D task, in the case of an increase in Barrese JC, Aceros J, Donoghue JP (2016). Scanning electron
dimensionality, a new calibration or a switch to a previ- microscopy of chronically implanted intracortical microelec-
ously calibrated decoder would be required. However, trode arrays in non-human primates. J Neural Eng 13 (2):
neither approaches are ideal. The inconvenience of a 026003. https://doi.org/10.1088/1741-2560/13/2/026003.
new calibration is disruptive for the goal of uninterrupted Birbaumer N, Ghanayim N, Hinterberger T et al. (1999).
BCI use, and to load an “old” calibrated decoder may be A spelling device for the paralysed. Nature 398 (6725):
detrimental to performance due to changes in neural tun- 297–298. https://doi.org/10.1038/18581.
ing (Jarosiewicz et al., 2015). Nevertheless, it is promis- Borton DA, Yin M, Aceros J (2013). An implantable wireless
neural interface for recording cortical circuit dynamics in
ing that some decoders are providing value for decoding
moving primates. J Neural Eng 10 (2): 026010. https://
over weeks (Milekovic et al., 2018).
doi.org/10.1088/1741-2560/10/2/026010.
As the neural engineering field evolves and new Bouton CE, Shaikhouni A, Annetta NV et al. (2016). Restoring
experimental paradigms are tested, fundamental ques- cortical control of functional movement in a human with
tions about the nature of volitional movements will also quadriplegia. Nature 533 (7602): 247–250. https://doi.
be addressed. The combination of neuroscience- org/10.1038/nature17435.
informed decoders and more agnostic decoders driven Bundy DT, Pahwa M, Szrama N et al. (2016). Decoding three-
by machine learning approaches will, together, likely dimensional reaching movements using electrocortico-
lead to ideal, intuitive, and rapid control over robotic graphic signals in humans. J Neural Eng 13: 026021.
effectors. New neuroscientific insights will cycle back https://doi.org/10.1088/1741-2560/13/2/026021.
and help to improve BCI systems, hopefully yielding Burrow M, Burrow M, Dugger J et al. (1997). Cortical control
of a robot using a time-delay neural network. Proceedings
ever better and clinically useful BCI-enabled control of
of international conference on rehabilitation robotics
robotic effectors for people with paralysis.
(ICORR). 83–86. Retrieved from, http://citeseerx.ist.psu.
edu/viewdoc/summary?doi¼10.1.1.596.5894.
DISCLAIMER Buzsáki G, Anastassiou CA, Koch C (2012). The origin of
extracellular fields and currents—EEG, ECoG, LFP and
The contents of this chapter are solely the responsibility spikes. Nat Rev Neurosci 13 (6): 407–420. https://doi.
of the authors and do not necessarily represent the official org/10.1038/nrn3241.
views of the Department of Veterans Affairs, or the US Carlson T, Millán JdR (2013). Brain-controlled wheelchairs: a
government. robotic architecture. IEEE Robot Autom Mag 20 (1):
65–73. https://doi.org/10.1109/MRA.2012.2229936.
REFERENCES Carmena JM, Lebedev MA, Crist RE et al. (2003). Learning to
control a brain–machine interface for reaching and grasp-
Aflalo T, Kellis S, Klaes C et al. (2015). Decoding motor ing by primates. PLoS Biol 1 (2): e42. https://doi.org/
imagery from the posterior parietal cortex of a tetraplegic 10.1371/journal.pbio.0000042.
human. Science 348 (6237): 906–910. https://doi.org/ Chao ZC, Nagasaka Y, Fujii N (2010). Long-term asynchro-
10.1126/science.aaa5417. nous decoding of arm motion using electrocorticographic
Ajiboye AB, Willett FR, Young DR et al. (2017). Restoration signals in monkey. Front Neuroeng 3: 3. https://doi.org/
of reaching and grasping movements through brain- 10.3389/fneng.2010.00003.
96 M. VILELA AND L.R. HOCHBERG
Chapin JK, Moxon KA, Markowitz RS et al. (1999). Real-time Evarts EV (1968). Relation of pyramidal tract activity to force
control of a robot arm using simultaneously recorded neu- exerted during voluntary movement. J Neurophysiol 31:
rons in the motor cortex. Nat Neurosci 2 (7): 664–670. 14–27.
https://doi.org/10.1038/10223. Fetz EE (1969). Operant conditioning of cortical unit activity.
Chaudhary U, Birbaumer N, Curado MR (2015). Brain-machine Science 163 (3870): 955–958. Retrieved from, http://
interface (BMI) in paralysis. Ann Phys Rehabil Med 58: science.sciencemag.org/content/163/3870/955/tab-pdf.
9–13. https://doi.org/10.1016/j.rehab.2014.11.002. Fifer MS, Hotson G, Wester BA et al. (2014). Simultaneous
Chen X, Zhao B, Gao X (2019). Noninvasive brain-computer neural control of simple reaching and grasping with the
interface based high-level control of a robotic arm for pick modular prosthetic limb using intracranial EEG. IEEE
and place tasks. In: 2018 14th international conference on Trans Neural Syst Rehabil Eng 22 (3): 695–705. https://
natural computation, fuzzy systems and knowledge discov- doi.org/10.1109/TNSRE.2013.2286955.
ery (ICNC-FSKD), (2015), 1193–1197. https://doi.org/ Flesher SN, Collinger JL, Foldes ST et al. (2016). Intracortical
10.1109/fskd.2018.8686979. microstimulation of human somatosensory cortex. Sci
Chestek CA, Cunningham JP, Gilja V et al. (2009). Neural Transl Med 8 (361): 361ra141. https://doi.org/10.1126/sci-
prosthetic systems: current problems and future direc- translmed.aaf8083.
tions. 2009 Annual international conference of the Frank K (1968). Some approaches to the technical problem of
IEEE engineering in medicine and biology society. chronic excitation of peripheral nerve. Ann Otol Rhinol
3369–3375. https://doi.org/10.1109/IEMBS.2009. Laryngol 77 (4): 761–771. Retrieved from, http://www.
5332822. ncbi.nlm.nih.gov/pubmed/5667220.
Churchland MM, Cunningham JP, Kaufman MT et al. Georgopoulos AP, Kalaska JF, Caminiti R et al. (1982). On the
(2012). Neural population dynamics during reaching. relations between the direction of two-dimensional arm
Nature 487 (7405): 51–56. https://doi.org/10.1038/ movements and cell discharge in primate motor cortex.
nature11129. J Neurosci 2 (11): 1527–1537. https://doi.org/citeulike-
Collinger JL, Wodlinger B, Downey JE et al. (2013). High- article-id:444841.
performance neuroprosthetic control by an individual with Georgopoulos AP, Schwartz AB, Kettner RE (1986). Neuronal
tetraplegia. Lancet 381 (9866): 557–564. https://doi.org/ population coding of movement direction. Science 233
10.1016/S0140-6736(12)61816-9. (4771): 1416–1419.
Crone N, Miglioretti DL, Gordon B et al. (1998a). Functional Gilja V, Pandarinath C, Blabe CH et al. (2015). Clinical trans-
mapping of human sensorimotor cortex with electrocorti- lation of a high-performance neural prosthesis. Nat Med 21
cographic spectral analysis. II. Event-related synchroniza- (10): 1142–1145. https://doi.org/10.1038/nm.3953.
tion in the gamma band. Brain 121 (12): 2301–2315. https:// Green AM, Kalaska JF (2011). Learning to move machines
doi.org/10.1093/brain/121.12.2301. with the mind. Trends Neurosci 34 (2): 61–75. https://
Crone N, Miglioretti DL, Gordon B et al. (1998b). Functional doi.org/10.1016/j.tins.2010.11.003.
mapping of human sensorimotor cortex with electrocorti- Hatsopoulos NG, Donoghue JP (2009). The science of neural
cographic spectral analysis. I. Alpha and beta event-related interface systems. Annu Rev Neurosci 32: 249–266. https://
desynchronization. Brain 121 (12): 2271–2299. https://doi. doi.org/10.1146/annurev.neuro.051508.135241.
org/10.1093/brain/121.12.2271. Hochberg LR, Serruya MD, Friehs GM et al. (2006). Neuronal
Daly JJ, Wolpaw JR (2008). Brain–computer interfaces in neu- ensemble control of prosthetic devices by a human with tet-
rological rehabilitation. Lancet Neurol 7 (11): 1032–1043. raplegia. Nature 442 (7099): 164–171. https://doi.org/
https://doi.org/10.1016/S1474-4422(08)70223-0. 10.1038/nature04970.
Diaz-Botia CA, Luna LE, Neely RM et al. (2017). A silicon Hochberg LR, Bacher D, Jarosiewicz B et al. (2012). Reach
carbide array for electrocorticography and peripheral nerve and grasp by people with tetraplegia using a neurally con-
recording. J Neural Eng 14 (5): 056006. https://doi.org/ trolled robotic arm. Nature 485: 372–375. https://doi.org/
10.1088/1741-2552/aa7698. 10.1038/nature11076.
Donoghue JP (2008). Bridging the brain to the world: a Horki P, Solis-Escalante T, Neuper C et al. (2011). Combined
perspective on neural interface systems. Neuron 60 (3): motor imagery and SSVEP based BCI control of a 2 DoF
511–521. https://doi.org/10.1016/j.neuron.2008.10.037. artificial upper limb. Med Biol Eng Comput 49 (5):
Doud AJ, Lucas JP, Pisansky MT et al. (2011). Continuous 567–577. https://doi.org/10.1007/s11517-011-0750-2.
three-dimensional control of a virtual helicopter using a Hortal E, Planelles D, Costa A et al. (2015). SVM-based brain–
motor imagery based brain-computer interface. PLoS machine interface for controlling a robot arm through four
One 6 (10): e26322. https://doi.org/10.1371/journal.pone. mental tasks. Neurocomputing 151: 116–121. https://doi.
0026322. org/10.1016/j.neucom.2014.09.078.
Downey JE, Weiss JM, Muelling K et al. (2016). Blending of Hotson G, McMullen DP, Fifer MS et al. (2016). Individual
brain-machine interface and vision-guided autonomous finger control of a modular prosthetic limb using high-
robotics improves neuroprosthetic arm performance during density electrocorticography in a human subject. J Neural
grasping. J Neuroeng Rehabil 13: 28. https://doi.org/ Eng 13 (2): 026017. https://doi.org/10.1088/1741-
10.1186/s12984-016-0134-9. 2560/13/2/026017.
 BRAIN-COMPUTER INTERFACES FOR ROBOTIC CONTROL 97
Humphrey DR, Schmidt EM, Thompson WD (1970). Leuthardt EC, Schalk G, Wolpaw JR et al. (2004b). A brain–
Predicting measures of motor performance from multiple computer interface using electrocorticographic signals in
cortical spike trains. Science 170 (3959): 758–762. humans. J Neural Eng 1 (2): 63–71. https://doi.org/
Retrieved from, http://science.sciencemag.org/content/ 10.1088/1741-2560/1/2/001.
170/3959/758/tab-pdf. Leuthardt EC, Miller KJ, Schalk G et al. (2006).
Jarosiewicz B, Sarma AA, Bacher D et al. (2015). Virtual Electrocorticography-based brain computer interface—
typing by people with tetraplegia using a self-calibrating the Seattle experience. IEEE Trans Neural Syst Rehabil
intracortical brain-computer interface. Sci Transl Med 7 Eng 14 (2): 194–198. https://doi.org/10.1109/TNSRE.
(313): 313ra179. https://doi.org/10.1126/scitranslmed. 2006.875536.
aac7328. Lotte F, Bougrain L, Clerc M et al. (2015).
Kennedy PR, Bakay RA (1998). Restoration of neural output Electroencephalography (EEG)-based brain-computer
from a paralyzed patient by a direct brain connection. interfaces. In: Wiley encyclopedia of electrical and
Neuroreport 9 (8): 1707–1711. electronics engineering, Wiley, p. 44.
Kennedy PR, Bakay RA, Sharpe SM (1992). Behavioral Matsushita K, Hirata M, Suzuki T et al. (2013). Development
correlates of action potentials recorded chronically inside of an implantable wireless ECoG 128ch recording device
the Cone Electrode. Neuroreport 3 (7): 605–608. for clinical brain machine interface. Conf Proc IEEE Eng
Retrieved from, http://www.ncbi.nlm.nih.gov/pubmed/ Med Biol Soc 2013: 1867–1870. https://doi.org/10.1109/
1421115. EMBC.2013.6609888.
Kennedy PR, Bakay RAE, Moore MM et al. (2000). Direct Mcfarland DJ, Sarnacki WA, Wolpaw JR (2010).
control of a computer from the human central nervous sys- Electroencephalographic (EEG) control of three-
tem. IEEE Trans Rehabil Eng 8 (2): 198–202. https://doi. dimensional movement. J Neural Eng 7 (7): 36007.
org/10.1109/86.847815. https://doi.org/10.1088/1741-2560/7/3/036007.
Khanna P, Swann N, De Hemptinne C et al. (2017). McMullen DP, Hotson G, Katyal KD et al. (2014).
Neurofeedback control in Parkinsonian patients using Demonstration of a semi-autonomous hybrid brain–
electrocorticography signals accessed wirelessly with a machine interface using human intracranial EEG, eye
chronic, fully implanted device. IEEE Trans Neural Syst tracking, and computer vision to control a robotic upper
Rehabil Eng 25: 1715–1724. https://doi.org/10.1109/ limb prosthetic. IEEE Trans Neural Syst Rehabil Eng 22
TNSRE.2016.2597243. (4): 784–796. https://doi.org/10.1109/TNSRE.2013.
Kim HK, Biggs SJ, Schloerb DW et al. (2006). Continuous 2294685.
shared control for stabilizing reaching and grasping with Mellinger J, Schalk G, Braun C et al. (2007). An MEG-based
brain-machine interfaces. IEEE Trans Biomed Eng 53 (6): brain-computer interface (BCI). Neuroimage 36 (3):
1164–1173. https://doi.org/10.1109/TBME.2006.870235. 581–593. https://doi.org/10.1016/j.neuroimage.2007.
K€ubler A, Kotchoubey B, Kaiser J et al. (2001). Brain– 03.019.
computer communication: unlocking the locked in. Meng J, Zhang S, Bekyo A et al. (2016). Noninvasive electro-
Psychol Bull 127 (3): 358–375. https://doi.org/ encephalogram based control of a robotic arm for reach and
10.1037/0033-2909.127.3.358. grasp tasks. Sci Rep 6: 38565. https://doi.org/10.1038/
LaFleur K, Cassady K, Doud A et al. (2013). Quadcopter srep38565.
control in three-dimensional space using a noninvasive Milekovic T, Sarma AA, Bacher D et al. (2018). Stable long-term
motor imagery-based brain–computer interface. J Neural BCI-enabled communication in ALS and locked-in syn-
Eng 10 (4): 046003. https://doi.org/10.1088/1741- drome using LFP signals. J Neurophysiol 120 (1):
2560/10/4/046003. 343–360. https://doi.org/10.1152/jn.00493.2017.
Lauer RT, Peckham PH, Kilgore KL (1999). EEG-based con- Millán JdR, Renkens F, Mourino J et al. (2004). Noninvasive
trol of a hand grasp neuroprosthesis. Neuroreport 10 (8): brain-actuated control of a mobile robot by human EEG.
1767–1771. Retrieved from, http://www.ncbi.nlm.nih. IEEE Trans Biomed Eng 51 (6): 1026–1033. https://doi.
gov/pubmed/10501572. org/10.1109/TBME.2004.827086.
Lebedev MA, Nicolelis MAL (2006). Brain–machine inter- Millán JdR, Rupp R, M€ uller-Putz GR et al. (2010). Combining
faces: past, present and future. Trends Neurosci 29 (9): brain-computer interfaces and assistive technologies: state-
536–546. https://doi.org/10.1016/j.tins.2006.07.004. of-the-art and challenges. Front Neurosci 4: 161. https://doi.
Lee J, Laiwalla F, Jeong J et al. (2018). Wireless power and org/10.3389/fnins.2010.00161.
data link for ensembles of sub-mm scale implantable sen- Moran DW, Schwartz AB (1999). Motor cortical
sors near 1 GHz. IEEE biomedical circuits and systems representation of speed and direction during reaching.
conference (BioCAS). 2018: 1–4. https://doi.org/10.1109/ J Neurophysiol 82 (5): 2676–2692. Retrieved from,
BIOCAS.2018.8584725. http://jn.physiology.org/content/jn/82/5/2676.full.pdf.
Leuthardt EC, Schalk G, Ojemann JG et al. (2004a). A brain– Musallam S, Corneil BD, Greger B et al. (2004). Cognitive
computer interface using electrocorticographic signals in control signals for neural prosthetics. Science 305 (5681):
humans. J Neural Eng 1 (1): 63–71. https://doi.org/ 258–262. Retrieved from, http://science.sciencemag.org/
10.1088/1741-2560/1/2/001. content/305/5681/258.full.
98 M. VILELA AND L.R. HOCHBERG
Nicolas-Alonso LF, Gomez-Gil J (2012). Brain computer signal. Nature 416 (6877): 141–142. https://doi.org/
interfaces, a review. Sensors 12 (12): 1211–1279. https:// 10.1038/416141a.
doi.org/10.3390/s120201211. Shenoy KV, Carmena JM (2014). Combining decoder design
Nuyujukian P, Albites Sanabria J, Saab J et al. (2018). Cortical and neural adaptation in brain-machine interfaces. Neuron
control of a tablet computer by people with paralysis. PLoS 84: 665–680.
One 13 (11): e0204566. https://doi.org/10.1371/journal. Shimoda K, Nagasaka Y, Chao ZC et al. (2012). Decoding
pone.0204566. continuous three-dimensional hand trajectories from epi-
Pandarinath C, Nuyujukian P, Blabe CH et al. (2017). High dural electrocorticographic signals in Japanese macaques.
performance communication by people with paralysis J Neural Eng 9 (3): 036015. https://doi.org/10.1088/1741-
using an intracortical brain-computer interface. Elife 6. 2560/9/3/036015.
https://doi.org/10.7554/eLife.18554. Simeral JD, Kim S-P, Black MJ et al. (2011a). Neural control
Perge JA, Homer ML, Malik WQ et al. (2013). Intra-day signal of cursor trajectory and click by a human with tetraplegia
instabilities affect decoding performance in an intracortical 1000 days after implant of an intracortical microelectrode
neural interface system. J Neural Eng 10 (3): 036004. array. J Neural Eng 8 (8): 25027. https://doi.org/
https://doi.org/10.1088/1741-2560/10/3/036004. 10.1088/1741-2560/8/2/025027.
Pesaran B, Musallam S, Andersen RA (2006). Cognitive neu- Simeral JD, Kim SP, Black MJ et al. (2011b). Neural control of
ral prosthetics. Curr Biol 16 (3): R77–R80. https://doi.org/ cursor trajectory and click by a human with tetraplegia
10.1016/j.cub.2006.01.043. 1000 days after implant of an intracortical microelectrode
Pfurtscheller G, Cooper AR (1975). Frequency dependence of array. J Neural Eng 8 (2): 025027. https://doi.org/
the transmission of the EEG from cortex to scalp. 10.1088/1741-2560/8/2/025027.
Electroencephalogr Clin Neurophysiol 38: 93–96. Sitaram R, Caria A, Veit R et al. (2007). FMRI brain-computer
Retrieved from, http://ac.els-cdn.com/0013469475902151/ interface: a tool for neuroscientific research and treatment.
1-s2.0-0013469475902151-main.pdf?_tid¼3b132074-813a- Comput Intell Neurosci 2007: 25487. https://doi.org/
11e7-99b7-00000aab0f01&acdnat¼1502747456_b939f35b 10.1155/2007/25487.
74927747d2e267a77f47f823. Soekadar SR, Birbaumer N, Slutzky MW et al. (2015). Brain–
Pistohl T, Schulze-Bonhage A, Aertsen A et al. (2012). machine interfaces in neurorehabilitation of stroke.
Decoding natural grasp types from human ECoG. Neurobiol Dis 83: 172–179. https://doi.org/10.1016/j.
Neuroimage 59 (1): 248–260. https://doi.org/10.1016/ nbd.2014.11.025.
j.neuroimage.2011.06.084. Tang J, Zhou Z (2018). A shared-control based BCI system: for
Pistohl T, Schmidt TSB, Ball T et al. (2013). Grasp detection a robotic arm control. In: 1st international conference on
from human ECoG during natural reach-to-grasp move- electronics instrumentation and information systems,
ments. PLoS One 8 (1): e54658. https://doi.org/10.1371/ EIIS 2017, 2018-January, 1–5. https://doi.org/10.1109/
journal.pone.0054658. EIIS.2017.8298767.
Royer AS, Doud AJ, Rose ML et al. (2010). EEG control of a Tang J, Zhou Z, Yu Y (2016). A hybrid computer interface for
virtual helicopter in 3-dimensional space using intelligent robot arm control. 2016 8th International conference on
control strategies. IEEE Trans Neural Syst Rehabil Eng information technology in medicine and education (ITME).
18 (6): 581–589. https://doi.org/10.1109/TNSRE. 365–369. https://doi.org/10.1109/ITME.2016.0088.
2010.2077654. Taylor DM, Tillery SIH, Schwartz AB (2002). Direct cortical
Schalk G, Leuthardt EC (2011). Brain-computer interfaces using control of 3D neuroprosthetic devices. Science 296 (5574):
electrocorticographic signals. IEEE Rev Biomed Eng 4: 1829–1832. Retrieved from, http://science.sciencemag.org/
140–154. https://doi.org/10.1109/RBME.2011.2172408. content/296/5574/1829.full.
Schmidt EM (1980). Single neuron recording from motor cor- Thongpang S, Richner TJ, Brodnick SK et al. (2011). A micro-
tex as a possible source of signals for control of external electrocorticography platform and deployment strategies
devices. Ann Biomed Eng 8 (4–6): 339–349. https://doi. for chronic BCI applications. Clin EEG Neurosci 42 (4):
org/10.1007/BF02363437. 259–265. Retrieved from, https://www.ncbi.nlm.nih.gov/
Schwartz AB (2016). Perspective movement: how the brain pmc/articles/PMC3653975/pdf/nihms359272.pdf.
communicates with the world. Cell 164: 1122–1135. Úbeda A, Iáñez E, Azorı́n JM (2013). Shared control architec-
https://doi.org/10.1016/j.cell.2016.02.038. ture based on RFID to control a robot arm using a sponta-
Scott SH (2003). The role of primary motor cortex in goal- neous brain–machine interface. Robot Auton Syst61:
directed movements: insights from neurophysiological 768–774. https://doi.org/10.1016/j.robot.2013.04.015.
studies on non-human primates. Curr Opin Neurobio l13: Van Dokkum LEH, Ward T, Laffont I (2015). Brain computer
671–677. https://doi.org/10.1016/j.conb.2003.10.012. interfaces for neurorehabilitation—its current status as a
Seo D, Neely RM, Shen K et al. (2016). Wireless recording in rehabilitation strategy post-stroke. Ann Phys Rehabil
the peripheral nervous system with ultrasonic neural dust. Med58: 3–8. https://doi.org/10.1016/j.rehab.2014.09.016.
Neuron 91 (3): 529–539. https://doi.org/10.1016/j.neuron. Vansteensel MJ, Pels EGM, Bleichner MG et al. (2016). Fully
2016.06.034. implanted brain–computer interface in a locked-in patient
Serruya MD, Hatsopoulos NG, Paninski L et al. (2002). Brain- with ALS. N Engl J Med 375 (21): 2060–2066. https://
machine interface: instant neural control of a movement doi.org/10.1056/NEJMoa1608085.
 BRAIN-COMPUTER INTERFACES FOR ROBOTIC CONTROL 99
Várkuti B, Guan C, Pan Y et al. (2013). Resting state changes Wodlinger B, Downey JE, Tyler-Kabara EC et al. (2015). Ten-
in functional connectivity correlate with movement recov- dimensional anthropomorphic arm control in a human
ery for BCI and robot-assisted upper-extremity training brain  machine interface: difficulties, solutions, and limi-
after stroke. Neurorehabil Neural Repair 27 (1): 53–62. tations. J Neural Eng 12 (1): 016011. https://doi.org/
https://doi.org/10.1177/1545968312445910. 10.1088/1741-2560/12/1/016011.
Velliste M, Perel S, Spalding MC et al. (2008). Cortical control Wolpaw JR, McFarland DJ (2004). Control of a two-
of a prosthetic arm for self-feeding. Nature 453 (7198): dimensional movement signal by a noninvasive brain-
1098–1101. https://doi.org/10.1038/nature06996. computer interface in humans. Proc Natl Acad Sci USA
Venkatakrishnan A, Francisco GE, Contreras-Vidal JL (2014). 101 (51): 17849–17854. https://doi.org/10.1073/pnas.
Applications of brain–machine interface systems in stroke 0403504101.
recovery and rehabilitation. Curr Phys Med Rehabil Rep 2: Wolpaw JR, Birbaumer N, Mcfarland DJ et al. (2002). Brain–
93–105. https://doi.org/10.1007/s40141-014-0051-4. computer interfaces for communication and control. Clin
Wang W, Collinger JL, Degenhart AD et al. (2013). An elec- Neurophysiol 113: 767–791. Retrieved from, www.
trocorticographic brain interface in an individual with tet- elsevier.com/locate/clinph.
raplegia. PLoS One 8 (2): e55344. https://doi.org/10.1371/ Yanagisawa T, Hirata M, Saitoh Y et al. (2011). Real-time con-
journal.pone.0055344. trol of a prosthetic hand using human electrocorticography
Wang H, Dong X, Chen Z et al. (2015). Hybrid gaze/EEG brain signals. J Neurosurg 114 (6): 1715–1722. https://doi.org/
computer interface for robot arm control on a pick and 10.3171/2011.1.JNS101421.
place task. 2015 37th Annual international conference of Yanagisawa T, Hirata M, Saitoh Y et al. (2012).
the IEEE engineering in medicine and biology society Electrocorticographic control of a prosthetic arm in para-
(EMBC). 1476–1479. https://doi.org/10.1109/EMBC. lyzed patients. Ann Neurol 71 (3): 353–361. https://doi.
2015.7318649. org/10.1002/ana.22613.
Wessberg J, Stambaugh CR, Kralik JD et al. (2000). Real-time Yin M, Borton DA, Komar J et al. (2014). Wireless neurosen-
prediction of hand trajectory by ensembles of cortical neu- sor for full-spectrum electrophysiology recordings during
rons in primates. Nature 408 (6810): 361–365. https://doi. free behavior. Neuron 84 (6): 1170–1182. https://doi.org/
org/10.1038/35042582. 10.1016/j.neuron.2014.11.010.