34th Annual International Conference of the IEEE EMBS

San Diego, California USA, 28 August - 1 September, 2012

Real-time Fusion of Gaze and EMG for a Reaching Neuroprosthesis

Elaine A. Corbett, Student Member, IEEE, Konrad P. Kording, and Eric J. Perreault, Member, IEEE


mation about the reach target is available [5–8]. We have
Abstract—For rehabilitative devices to restore functional previously shown that reaching movements can be effective-
movement to paralyzed individuals, user intent must be ly reconstructed when a single shoulder EMG is combined
determined from signals that remain under voluntary with target information obtained by tracking eye movements
control. Tracking eye movements is a natural way to [9], [10]. Eye movements are a natural, unobtrusive way to
learn about an intended reach target and, when com- learn about a subject’s intentions. However, the target in-
bined with just a small set of electromyograms (EMGs) formation obtained will be uncertain; though people almost
in a probabilistic mixture model, can reliably generate always look at a target before reaching to it, they may also
accurate trajectories even when the target information is gaze at other locations. We have shown that probabilistic
uncertain. To experimentally assess the effectiveness of mixture models can account for this and accurately recon-
our algorithm in closed-loop control, we developed a ro-
struct reaching movements even when there is high uncer-
botic system to simulate a reaching neuroprosthetic. In-
tainty about the target [11].
corporating target information by tracking subjects’
In many cases decoding algorithms have been tested by
gaze greatly improved performance when the set of
EMGs was most limited. In addition we found that online recording the neural signals as natural arm movements are
made, and then performing offline evaluations of how well
performance was better than predicted by the offline
those movements can be reconstructed. However, research-
accuracy of the training data. By enhancing the trajecto-
ry model with target information the decoder relied less ers have recently shown that offline accuracy does not nec-
essarily predict online performance [12], [13]. This is prob-
on neural control signals, reducing the burden on the
lematic as many proposed decoding algorithms have not
user.
been tested in closed-loop. Additionally, in a real neuropros-
thetic system there will be no natural reaches available to
I. INTRODUCTION train the models.

F OR people whose arms have been paralyzed by spinal

cord injury (SCI) movement can be restored using a neu-
roprosthetic known as Functional Electrical Stimulation
We have developed a robotic system to simulate a neuro-
prosthesis, thus providing a means to evaluate our approach
to combining EMG and gaze information in closed-loop
(FES), where the spinal cord is effectively by-passed and the control. This simulation was carried out by having a robot
peripheral nerves are stimulated electrically [1], or other move a subject’s arm throughout a reaching workspace
robotic assistive devices [2]. To date, most implementations based on the decoded velocity and position. We found that
of full arm reaching have involved pre-programmed patterns incorporating the gaze data produced dramatic improve-
of activation, controlled by switching mechanisms through ments in control with a single shoulder EMG channel.
respiration [3] or contra-lateral shoulder movement [4], re-
sulting in unnatural control of limited movement patterns. II. METHODS
To allow a more flexible interface with multiple degrees of A. Decoding Algorithms
freedom (DOFs), one of the most challenging problems is We compared two decoding approaches: a simple decoder
the determination of user intent from the physiological sig- that used only the EMGs as inputs and another that incorpo-
nals that remain under voluntary control. At high levels of rated target information. We tested the latter both with per-
cervical SCI, individuals have no control over a majority of fect target information and also using information found
the arm muscles, thus inferring free reaches through residual from tracking subjects’ eye movements. For both algorithms
movement or electromyograms (EMG) alone is not feasible. we employed the Kalman filter (KF) framework for decod-
A number of groups in the brain-machine-interface (BMI) ing, assuming linear dynamics and Gaussian noise:
field have shown that reconstruction of reach trajectories
from neural signals can be greatly improved when infor- ̇ ̈ (1)
where xt is the state vector at time t, zt represents
Manuscript received March 29, 2012. This work was supported by the the hand position, wt is the process noise with p(w) ~
NSF program in Cyber-Physical Systems (NSF/CNS 0939963). N(0,Q), and Q is the state covariance matrix.
E. A. Corbett is with the Department of Biomedical Engineering, To create a directional trajectory model, we added the tar-
Northwestern University, Evanston, IL 60208 USA (phone: 312-238-5828; get position to the state space (KFT), thereby linearly incor-
fax: 312-238-2208; e-mail: [email protected]). porating it into the trajectory model [7], [8]:
K. P. Kording is with the Department of Physiology and the Department
of Physical Medicine and Rehabilitation, Northwestern University, Chica- ̇ ̈ (2)
go, IL 60611 USA (email: [email protected]).
E. J. Perreault is with the Department of Biomedical Engineering and the where zT t is the vector of target positions. In all cases
Department of Physical Medicine and Rehabilitation, Northwestern Univer- the observation model was considered to be linear, with
sity, Chicago, IL 60611 USA (email: [email protected]).

978-1-4577-1787-1/12/$26.00 ©2012 IEEE 739

 Gaussian noise. Observations were generated from the cor- upper trapezius and for C5 we also included the anterior,
responding window of EMG by extracting two features from middle and posterior deltoids (Fig. 1b). Each subject per-
each channel – the RMS value and the number of zero cross- formed three experimental sessions: one without target in-
ings (above a threshold), a frequency-related feature. The formation where the KF was tested at both simulated injury
square root transform of these features was taken to obtain levels (in separate blocks); one where the KFT and mKFT
more Gaussian-like distributions. (perfect target information and eye-tracking) were tested at
For the case when the target estimates were based on eye C5; a third where the KFT and mKFT were tested at C4. The
movements, multiple potential targets needed to be consid- orders of the algorithms and simulated injury levels were
ered. To achieve this we used a probabilistic mixture model randomized across subjects.
(mKFT) over each of the potential targets [5]. The KFT
C. Training the Decoders
recursion was performed for each possible target, zT, and
a weighted sum of the outputs was taken. The weights Training data was required to estimate the decoder parame-
were proportional to the prior for that target, and the ters. Because we wanted control to be intuitive, it was im-
likelihood of the model given that target. The weights portant that the EMGs controlling the decoder correspond as
were thus initialized based on the gaze data and they closely as possible to natural reaches. However, as the target
converged to the most likely trajectory as the neural in- population would be unable to generate unassisted reaches, it
formation was integrated over the course of the reach. made sense to have the robot move along an” ideal” trajecto-
Further details of the algorithms can be found in [9–11]. ry (linear in the kinematics and target) as the subject at-
tempted to move along with the reach.
B. Experimental Setup and Protocols During training, 18 targets spanning the reachable area of
Six able-bodied subjects (3 male, 3 female) participated in the two monitors each appeared twice in random order. The
the experiments. Each subject provided informed written reach began with the HapticMaster in the original starting
consent to a protocol approved by Northwestern Univer- position and, after an audible go cue, subjects performed the
sity's Institutional Review Board. All subjects were right- reach towards the target while EMGs were recorded (Fig.
arm dominant. 2). During the reach the subject was instructed to hold on
A retractable stylus was attached to the handle of a 3 DOF and gently assist the movement. The reach was ended when
HapticMaster robot. Subjects were seated comfortably fac- stylus tip reached the touch-screen, at which time the robot
ing two touch-screen monitors that were at different distanc- returned to the start position.
es to the subject in the Z direction (Fig. 1a). The subject held For both models the parameters A, Q and the observation
the robot handle which was positioned directly in front of model and covariance matrices were estimated from training
them; the robot then moved their right arm forward through data using the maximum likelihood solution. In the case of
a reach so that the stylus would touch the monitors when the the KFT, the final recorded position of the stylus was ap-
arm was extended. The HapticMaster velocity was con- pended to the state vector for training, taking the place of the
trolled at 60Hz, with low-gain PID feedback on the position target estimate.
error to maintain positional fidelity.
The EMG signals were anti-alias and band-pass filtered
and recorded at 2400Hz. The monitor, HapticMaster and
head positions were recorded at 60Hz using an Optotrak
motion analysis system so that gaze data and positions on
the monitors could be transformed into the HapticMaster
workspace. We recorded eye movements with an ASL
EYETRAC-6 head-mounted eye tracker. All signals were
recorded simultaneously and processed at 60Hz, thus the
EMGs were divided into 16ms windows for feature ex-
traction.
Fig. 2. Sample training reach a) kinematics and b) EMG
We tested all three algorithms (KF, KFT and mKFT) with
two sets of EMGs, in an attempt to simulate the signals that
D. Real-time Decoder Evaluation
would be available at different levels of SCI. To simulate an
injury below the fourth cervical level (C4) we used just the After the models had been trained they were evaluated in
a target acquisition task. For each trial a target randomly
appeared, 1s before the go cue. The goal was to place the
stylus in center of the target. Reaches were initiated when
any EMG channel doubled relative to its level prior to the go
cue. For the C4 level, the contralateral upper trapezius was
also recorded to allow subjects to initiate reaches where they
would not normally activate the ipsilateral muscle, by shrug-
ging their left shoulder. However, it was not included as a
part of the decoder (Fig. 3). After initiation, the decoded
velocity and position were used to control the HapticMaster.
When testing the KF, only the EMGs were used as inputs
Fig. 1 a) Experimental setup; b) EMG simulation of SCI levels. to the decoder. In the case of the KFT, the state vector was
initialized with the actual location of the target. For the

740
 with the eye-tracking (mKFT) was only slightly less accu-
rate and also remained consistent across the two simulated
injury levels. For the KF, with only EMG as an input, the
simulated level of injury naturally had a large effect. The
target acquisition rate at C5 with all four EMGs was close to
1, whereas at C4 it dropped to 40%. The average target error
was roughly 5cm greater than that of the mKFT. However,
this error was effectively determined by the distribution of
targets in the workspace. As the upper trapezius was pri-
marily activated during movements in the positive Y direc-
tion subjects were able to accurately control this dimension,
but were unable to move in the X direction. This is illustrat-
ed in the target VAF, which is 0 in X and close to 1 in Y
(Fig. 4c-d).

Fig. 3. Sample testing reaches, kinematics and square-root trans-

formed RMS of EMG a) KF at C5 and b) mKFT at C4
mKFT, the gaze data from the one-second period prior to
initiation was used to estimate three potential targets with
which to initialize a corresponding mixture component (Fig.
3b). The three-dimensional location of the eye gaze was first
calculated by projecting its direction onto the monitors. The
first, middle and last samples were selected, and all other
samples were assigned to a group according to which of
the three was closest. The means of these three groups
were used to initialize three KFTs in the mixture model
and their priors were assigned proportional to the number
of samples in them. If the subject looked at multiple po- Fig. 4. Target accuracy for algorithms and simulated injury levels.
sitions prior to reaching, this method ensured with a high B.Mean and standardof
Comparison errors of a) Accuracy
Offline Target acquisition rate b) Final posi-
and Performance
probability that the correct target was accounted for by tion error, and Target VAF in c) X and d) Y directions.
one of the filters in the mixture.
After sufficient practice to produce a learning plateau, A central question in decoding research is how online and
each subject performed forty test reaches for each decod- offline performance correspond. We found that the online R2
ing model. Control performance was evaluated based on was consistently higher than the training R2 (Fig. 5). Even
the position of the stylus at the end of the reach. The tar- though subjects were in no way constrained to follow the
get acquisition rate and final distance to target were cal- “ideal” reach path, it was more closely replicated online than
culated. In addition, the proportion of target variance in reconstructing the training reaches where that path had
accounted for (VAF) was calculated separately in the X been enforced. In particular, the KF at C4 produced a wide
and Y directions, by normalizing the final distance to the range of training accuracies, as the extent to which the upper
target by the total variance of the targets. As there was trapezius was activated during training varied across sub-
greater target variance in the Y direction, this allowed us jects. However, all subjects were able to perform highly ac-
to quantify the level of control in the two dimensions. As curate control in the Y-direction, as reflected by the consist-
all of the targets were on the two monitors their positions ently high R2 values. As mentioned above, there was little
were roughly similar in the Z dimension. control in the X-direction – the high R2 values are due to the
Finally the online R 2 was evaluated between the per-
formed reach and the "ideal" reach simulated between
the start position and the target using the method for
generating training reaches. This metric was used for
comparison with the R 2 of the training data, which was
calculated using leave-one-out cross-validation to com-
pare the EMG (and final target in the case of the KFT)
based offline reconstructions to the training reaches.

III. RESULTS
A. Control Performance
Unsurprisingly, when the decoder was given perfect target
information (KFT) the reaches were very accurate, regard- Fig 5. Online R2 plotted against R2 of training data for the different
less of the quantity of EMG available (Fig. 4). Performance algorithm and simulated injury level combinations for each subject.

741
majority of the reach variance being in the Y and Z- more, even when there was sufficient neural data for effec-
directions. tive control the target information helped generate more
“ideal” trajectories, relying less on the users’ neural control
IV. DISCUSSION and thereby reducing their cognitive burden.
For people who have sustained a high-level SCI, FES con-
trol of reaching is a challenging task. The available proximal ACKNOWLEDGMENT
muscles are not sufficient to provide effective control; the The authors thank Tim Haswell and Ben Walker for their
incorporation of additional peripheral sensors, such as eye- work developing the data acquisition and robot control sys-
trackers, is therefore an obvious solution. Adding target in- tems, and Dr Nicholas Sachs for experimental assistance.
formation into the trajectory model is an intuitive way to
enhance control; the target informs us about the reach dy- REFERENCES
namics, resulting in a more well-defined trajectory. As our [1] D. T. Yu, R. F. Kirsch, A. M. Bryden, W. D. Memberg, and A. M.
training reaches were generated from a stereotyped trajecto- Acosta, “A neuroprosthesis for high tetraplegia,” J Spinal Cord
ry model as opposed to natural reaches, the uncertainty Med, vol. 24, no. 2, pp. 109–113, 2001.
about the model was further minimized. This served to de- [2] L. R. Hochberg, D. Bacher, B. Jarosiewicz, N. Y. Masse, J. D.
crease the reliance on neural control signals, requiring less Simeral, J. Vogel, S. Haddadin, J. Liu, S. S. Cash, P. van der Smagt,
and J. P. Donoghue, “Reach and grasp by people with tetraplegia
effort from the user. using a neurally controlled robotic arm,” Nature, vol. 485, no. 7398,
To effectively evaluate signal sources and algorithm ap- pp. 372–375, May 2012.
proaches it is essential that they be compared in closed-loop [3] B. T. Smith, M. J. Mulcahey, and R. R. Betz, “Development of an
control. While there are many examples in the literature of upper extremity FES system for individuals withC4 tetraplegia,”
closed-loop systems with non-human primates, usually con- IEEE Transactions on Rehabilitation Engineering, vol. 4, no. 4, pp.
264–270, 1996.
trolling a virtual interface, few of them have compared algo- [4] N. Hoshimiya, A. Naito, M. Yajima, and Y. Handa, “A multichan-
rithms explicitly. A small number of human experiments nel FES system for the restoration of motor functions in high spinal
have compared algorithms [15], or signal sources [14] in cord injury patients: a respiration-controlled system for multijoint
closed-loop. Cunningham et al used a closed-loop BMI sim- upper extremity,” IEEE Transactions on Biomedical Engineering,
ulator to explicitly compare offline and online performance vol. 36, no. 7, pp. 754–760, 1989.
[5] B. M. Yu, C. Kemere, G. Santhanam, A. Afshar, S. I. Ryu, T. H.
while changing the bin-width of a decoder, finding that Meng, M. Sahani, and K. V. Shenoy, “Mixture of trajectory models
online and offline evaluations produced strikingly different for neural decoding of goal-directed movements,” Journal of neu-
results [13]. While a larger bin-width provided smoothness rophysiology, vol. 97, no. 5, p. 3763, 2007.
in offline reconstructions, smaller bins are more useful [6] L. Srinivasan, U. T. Eden, A. S. Willsky, and E. N. Brown, “A
online as they allow faster response times for subjects’ error state-space analysis for reconstruction of goal-directed movements
using neural signals,” Neural computation, vol. 18, no. 10, pp.
corrections. We did not vary any parameter to explicitly in- 2465–2494, 2006.
fluence offline accuracy, however we did compare how well [7] C. Kemere and T. Meng, “Optimal estimation of feed-forward-
the “ideal” trajectory was replicated both on- and offline. controlled linear systems,” in IEEE International Conference on
While in this study both offline and online accuracy with Acoustics, Speech, and Signal Processing, 2005. Proceed-
target information were very high, we found that KF per- ings.(ICASSP’05), 2005, vol. 5.
[8] G. H. Mulliken, S. Musallam, and R. A. Andersen, “Decoding
formance without target information, when the subjects trajectories from posterior parietal cortex ensembles,” Journal of
could interact with the decoder, was dramatically better than Neuroscience, vol. 28, no. 48, p. 12913, 2008.
the offline accuracy predicted. This means that the im- [9] E. A. Corbett, E. J. Perreault, and K. P. Kording, “Decoding with
provements found by incorporating target information are limited neural data: a mixture of time-warped trajectory models for
less than indicated from previous offline studies [9], [10]. directional reaches.,” Journal of Neural Engineering, forthcoming.
[10] Corbett, E.A., Perreault, E.J., and Kording, K.P., “Mixture of time-
However, in the most severe cases, simulated here in the C4
warped trajectory models for movement decoding,” in Advances in
level, the target information was necessary to produce func- Neural Information Processing Systems, 2010, vol. 23.
tional movements in all directions. Furthermore, while accu- [11] E. A. Corbett, N. A. Sachs, K. Kording, and E. J. Perreault, “Deal-
rate reaches were performed in the C5 case using only EMG, ing with noisy gaze information for a target-dependent neural de-
it often required more effort from the user. Qualitatively, coder,” in Engineering in Medicine and Biology Society, EMBC,
subjects reported that the decoders incorporating target in- 2011 Annual International Conference of the IEEE, 2011, pp. 5428–
5431.
formation were much easier to control than those without; [12] S. M. Chase, A. B. Schwartz, and R. E. Kass, “Bias, optimal linear
the model with perfect target information often seemed ef- estimation, and the differences between open-loop simulation and
fortless. This is somewhat illustrated by the fact that the closed-loop performance of spiking-based brain-computer interface
online R2 is lower for the C5 KF versus mKFT (Fig. 5). algorithms,” Neural Networks, vol. 22, no. 9, pp. 1203–1213, 2009.
While the KF reaches were almost as accurate at the target [13] J. P. Cunningham, P. Nuyujukian, V. Gilja, C. A. Chestek, S. I.
Ryu, and K. V. Shenoy, “A closed-loop human simulator for inves-
(Fig. 4), the trajectories produced were less consistent. tigating the role of feedback control in brain-machine interfaces,”
Gaze information is extremely useful when predicting de- Journal of Neurophysiology, vol. 105, no. 4, pp. 1932–1949, 2011.
sired trajectories, and we have shown here that it is effective [14] E. A. Corbett, E. J. Perreault, and T. A. Kuiken, “Comparison of
and practical for use in a real-time system. Target infor- electromyography and force as interfaces for prosthetic control,” J
mation could equally be obtained by other means such as Rehabil Res Dev, vol. 48, no. 6, pp. 629–42, 2011.
[15] S.-P. Kim, J. D. Simeral, L. R. Hochberg, J. P. Donoghue, and M.
intracortical recordings or scanning the workspace. Even a J. Black, “Neural control of computer cursor velocity by decoding
small quantity of neural information can compensate for motor cortical spiking activity in humans with tetraplegia,” J. Neu-
uncertainty in the target information; a single EMG com- ral Eng., vol. 5, no. 4, pp. 455–476, Dec. 2008.
bined with the gaze here enabled accurate control. Further-

742