Abstract
Despite significant research developing myoelectric prosthesis controllers, many amputees have difficulty controlling their devices due in part to reduced sensory feedback. Many attempts at providing supplemental sensory feedback have not significantly aided control. We hypothesize this is because the feedback provided contains redundant information already provided by vision. However, whereas vision provides egocentric, position-based feedback, sensory feedback tied to joint coordinates may provide information complementary to vision. In this study, we tested if providing audio feedback of joint velocities can improve performance and adaptation to dynamic perturbations while controlling a virtual limb. While subjects performed time-controlled center-out reaches, we perturbed the dynamics of the system and measured the rate subjects adapted to this change. Our results suggest that initial errors were reduced in the presence of audio feedback, and we theorize this is due to subjects identifying the perturbed limb dynamics sooner. We also noted other possible benefits including improved muscle activation detection.
I. Introduction
For many upper-limb amputees, myoelectric prosthetic devices represent the current standard of restoring functionality. Recent studies have focused on extending myoelectric control to simultaneous movements of multiple degrees of freedom (DoFs) through pattern recognition or regression algorithms [1], [2]. However, these controllers are initially difficult to use and require a period to learn how to control the device [3]. Part of this learning is associated with making repeatable contractions [4]; however, it is likely that this learning is also partly attributed to the users recognizing the dynamical properties of the device. Reduced sensory feedback due to missing or damaged sensory organs (e.g. proprioceptors, mechanoreceptors, nociceptors) may contribute to this difficulty [5].
There have been several attempts to provide feedback via sensory substitution to improve performance, though few were successful in doing so with vision feedback present [6]. For example, Ninu et al. showed that grasping force, commonly studied in sensory substitution, can be estimated with vision alone by watching the velocity of the closing prosthesis [7]. Successful studies are able to improve performance by providing complementary sensory information not provided by vision, such as tactile information [8]. Additionally, studies suggest that providing feedback in a discrete fashion may be more beneficial for some tasks, confirming completion of a task more clearly than continuous feedback [9]. Effective sensory substitution requires not just providing a stimulus, but also consideration of how it will be interpreted by the user. For example, it is important for stimuli provided for one task to allow users to generalize their performance to other tasks [10]. Sensory substitution should also provide information not available to, or with similar or lesser variance than, the other intact senses, most notably vision [11].
Vision can be an extremely precise feedback modality, and is the most relied upon modality for amputees performing tasks [12]. Vision provides feedback in a global, egocentric reference frame [13]; therefore, less precise sensory substitutions providing feedback in the same global reference frame do not significantly improve control. However, the same sensory substitution providing feedback in a local, joint-based reference frame may provide information complementary to vision; one study suggests that joint-based velocity information is more relevant when users are less certain about control of their bodies than about the external environment [14]. Additionally, vision provides more precise feedback with position information [15], but is less precise with velocity information [16]; thus, sensory substitution encoding velocity information may also complement existing visual feedback.
The purpose of this study is to determine if continuous, local reference frame-based velocity feedback improves performance even in the presence of vision. This study is in contrast to prior works that have used discrete feedback or provided sensory substitution in global reference frames [6]. Our hypothesis was tested using a continuous joint-based, velocity-based feedback paradigm when controlling a 2-DoF myoelectric interface with control perturbations. The virtual limb was inspired by the control of a trans-humeral prosthesis consisting of an elbow and wrist. Subjects performed time-constrained center-out reaches [17] with and without audio feedback, during which the dynamics of the virtual limb were perturbed at discrete intervals. We tested the hypothesis that providing joint- and velocity-based audio feedback during these reaches would improve the rate of adaptation to perturbations to the virtual limb dynamics, and that this improved adaptation would generalize to multiple target locations.
II. Methods
A. Subjects
Ten right hand-dominant, non-amputee participants were recruited for this study, which was approved by the Northwestern University Institutional Review Board. All subjects provided informed consent before starting the study.
B. Experimental Protocol
Subjects participated in two experimental sessions, separated by at least one day: one session with no audio feedback, and one session with joint- and velocity-based audio feedback provided. The order of these sessions was randomized across subjects using balanced block randomization.
Subjects were seated in front of a computer monitor with their right arm placed in a rigid forearm brace clamped to a table, affixing the elbow and wrist positions. Four Delsys Bagnoli electromyographic (EMG) sensors were placed on the subject’s arm: over the biceps and triceps on the upper arm, and over the flexor and extensor compartments of the forearm. Reference electrodes were placed over the olecranon, and the electrode sites were wrapped with an elastic cohesive bandage [see Fig. 1(a)].
Figure 1.
Experiment Layout. (a) Experiment setup. The hand was immobilized in a forearm brace. EMG Electrodes placed over the upper arm and forearm controlled the virtual arm displayed on the screen. (b) Virtual environment. The cursor of the virtual arm would begin each trial at the home position (blue circle). One of four random targets would appear (green +), and subject would reach for the target (faded blue). During familiarization trials and the testing blocks, a ball (grey circle) would appear above the target; once the arm cursor left the home circle, the ball would drop, aligning with the target 1.5 seconds after dropping.
Subjects used arm and forearm EMG from isometric muscle contractions to control a virtual two-link arm. EMG from elbow flexion applied a counter-clockwise torque to the proximal linkage; EMG from elbow extension, clockwise torque. EMG from wrist flexion applied a counter-clockwise torque to the distal linkage; EMG from wrist extension, clockwise. Links were simulated with a length of 10 cm, a mass of 5 kg, and a damping coefficient of 1.5 Ns/m.
During the audio feedback session, subjects wore a pair of noise-canceling headphones (Bose Corporation, Framingham, MA). Audio feedback consisted of two pitches whose volumes were proportional to the speed of the two virtual linkages; the speed of the proximal linkage determined the amplitude of a 200 Hz pitch, and the speed of the distal linkage determined the amplitude of a 300 Hz pitch.
Subjects completed the following experimental tasks [see Table I]:
TABLE I.
EXPERIMENTAL TASKS
| Experimental Protocol |
Free Training | Familiarization | Testing Blocks | ||||
|---|---|---|---|---|---|---|---|
| Baseline |
No perturbation |
Perturbation |
Left generalization |
Right generalization |
|||
| Time-Constrained | No | Yes | Yes | ||||
| Trials | 100 (random targets) | 100 (random targets) | 40 (random targets) | 20 (far-right target) | 20 (far-right target) | 20 (top-left target) | 20 (top-right target) |
| Perturbation | No | No | No | No | Yes | Yes | Yes |
1). System Tuning
EMG gains and dead zones were tuned during a free exploration session lasting only a few minutes, where subjects were permitted to control the virtual arm and explore the workspace with no objective, allowing users to become familiar with the dynamics of the system.
2). Free Training
Subjects completed 100 free training trials; during free training, subjects performed center-out reaches towards one of four 2 cm diameter targets located 14 cm from the home position (25 reaches towards each target, in randomized order) [See Fig. 1(b)]. When the cursor of the arm stopped within the target, the arm was reset to the starting position and a new target was presented.
3). Familiarization
Subjects completed 100 familiarization trials introducing them to the protocol of the testing blocks. The protocol was similar to free training (25 reaches towards each of four targets); however, subjects were instructed to reach the target at 1.5 seconds after leaving the 5 cm diameter home circle [17]. This time-constrained task was used to ensure similar movement profiles across trials. A ball was shown above each target, and began dropping at a constant speed when the cursor left the home circle [see Fig. 1(b)]. This ball would align concentrically with the target at 1.5 seconds, thus indicating when the subject was to reach the target. If the subject moved for longer than 2 seconds, the trial was marked as timed out. Subjects were also instructed to complete the reach in a single fluid movement; if the cursor speed dropped below 1 cm/s, the trial ended, the arm was reset to the starting position, and a new target was presented. After completing 100 familiarization trials, if subjects did not stop within a ±0.25 second time window for at least 40% of the trials, or if they desired additional practice, they completed a second set of 100 familiarization trials.
4). Testing Blocks
Subjects performed 40 baseline reaches towards the four targets (10 reaches towards each target, in randomized order) to provide a baseline performance, before making 20 reaches towards the far-right target (no perturbation trials). The dynamic properties of the simulated arm were then perturbed by reducing the damping coefficient of each linkage to 0.5 Ns/m; this perturbation was used to promote adaptation to an intrinsic disturbance [14]. Subjects performed 20 additional reaches towards the far-right target with the new dynamics (perturbation trials). Following this, subjects performed 20 reaches towards the top-left target (left generalization) and 20 reaches towards the top-right target (right generalization), for 120 total trials.
C. Performance Metrics
Two performance metrics were calculated for each trial: Euclidean distance between the cursor and the target at 1.5 seconds (when the timing ball was concentrically aligned with the targets), and the average cursor speed during the 1.5 seconds of movement. Both metrics were adjusted by subtracting the average from the baseline trials to account for varying accuracy and speeds to different targets. From these performance metrics, adaptation during perturbation, left generalization, and right generalization trials were calculated for each subject by fitting the data to an exponential decay function (ae−λx), where a, the exponential decay gain, represents the overall magnitude of error, and λ, the exponential decay rate, represents the adaptation rate. Additionally, experiments were video- and screen captured, allowing for subsequent observation of subject performance.
III. Results
Sample data from a representative subject are shown in Figure 2. Reaches towards the far-right target during no perturbation trials (black) are consistent and accurate. However, when movement dynamics are perturbed (grey), movement becomes more erratic [see Fig. 2(a)]. This can be seen in the increase in distance from the target and average movement speed immediately after trial 60 [see Fig. 2(b)]. However, subjects typically quickly adapted to this change and restored their performance to baseline levels. Furthermore, while we expected to see an increase in error during initial reaches towards both generalization targets, these initial reaches were often near baseline performance.
Figure 2.
Sample data from representative subject. (a) Individual subject movement profiles during testing block, starting at trial 41. Subject was provided audio feedback in this testing block. Trace colors represent no perturbation (black), perturbation, left generalization, and right generalization (grey) trials. (b) Individual subject performance during testing blocks. Vertical dashed lines separate between testing block subsections. Horizontal dashed lines indicate mean ± standard deviation for the corresponding target during baseline trials. Example exponential decay curves are fit to the data.
A. Euclidean Distance
During no perturbation trials, Euclidean distance was unchanged from the baseline trials, as expected [see Fig. 3(a)]. After the system dynamics were perturbed, there was a trend to an increase in Euclidean distance error during the first few perturbed trials, especially when audio-feedback was not provided. This is supported by the exponential fit gains, which are lower on average with audio feedback present [see Table II]. This increase tapered to baseline values within the first few trials. There were no clear trends during either of the generalization blocks, which may indicate that subjects were capable of adapting their control over the entire movement space.
Figure 3.
Baseline-adjusted performance during testing blocks. Baseline values are subtracted from raw metrics. Bold lines represent trial mean and dashed lines represent trial standard error, averaged across subjects. No perturbation and perturbation trials are reaches towards the far-right target. Left and right generalization trials are reaches towards the top-left and top-right target, respectively. (a) Baseline-adjusted Euclidean distance during testing blocks. (b) Baseline-adjusted cursor speed during testing blocks.
TABLE II.
EXPONENTIAL DECAY COEFFICIENTS
| Testing Block Trials | No perturbation | Perturbation | Left generalization | Right generalization | |||||
|---|---|---|---|---|---|---|---|---|---|
| Mean | SD | Mean | SD | Mean | SD | Mean | SD | ||
| Euclidean Distance | |||||||||
| Gain (a) | No Audio | 0.0309 | 0.2546 | 3.4666 | 3.0812 | 1.4239 | 1.7595 | 0.7220 | 0.9463 |
| Audio | −0.5955 | 0.8693 | 1.3338 | 2.4776 | 0.9947 | 1.5277 | 1.0023 | 1.2965 | |
| Decay (λ) | No Audio | 0.1361 | 0.1695 | −0.1435 | 0.2251 | −0.4287 | 0.7613 | 0.0394 | 0.0426 |
| Audio | 0.2111 | 0.7579 | 2.3157 | 5.1995 | −0.0146 | 0.0677 | −0.0442 | 0.1242 | |
| Average Speed | |||||||||
| Gain (a) | No Audio | −0.0524 | 0.6561 | 6.8426 | 6.3346 | 2.8922 | 4.2834 | 0.8228 | 1.5560 |
| Audio | −0.3164 | 0.7877 | 2.3975 | 3.2656 | 1.1424 | 1.3870 | 0.5328 | 2.0675 | |
| Decay (λ) | No Audio | 0.2158 | 0.3185 | −3.4349 | 7.7524 | 0.0644 | 0.7395 | 0.0390 | 0.1184 |
| Audio | 0.0060 | 0.0663 | 0.4467 | 1.1598 | 0.0207 | 0.0546 | −0.0485 | 0.0598 | |
B. Average Cursor Speed
As with Euclidean distance, during no perturbation trials the average cursor speed was unchanged from baseline trials [see Fig. 3(b)]. After the system dynamics were perturbed, cursor speed increased due to the reduced damping term in the dynamics of the limb. This speed increase appears to be greater in the absence of audio feedback, as shown both in the figure and in the increased exponential fit gains [see Table II]. Similar to Euclidean distance error, these increases lessened over time, generally returning to baseline levels after a few trials. However, unlike trends for Euclidean distance, there appeared to be a second spike in cursor speed during initial reaches towards the left generalization target, with similar adaptation profiles as subjects adjusted their movements; furthermore, this error spike appears smaller during audio feedback blocks. There was no clear increase in speed during right generalization trials.
IV. Discussion
Two of the most desirable features of trans-humeral prostheses are the simultaneous and proportional control of multiple joints to perform coordinated movements, and a reduction in visual attention required to perform certain functions [18]. Sensory feedback and proprioception of the prosthetic limb are key components to addressing these limitations, but restoring these senses remains a major challenge facing myoelectric prostheses [5]. Closed-loop control for prosthetic devices is a vital part of correcting for errors, and plays an even greater role in learning unintuitive or arbitrary control mappings [19]. If sensory feedback provides the same information as intact senses (such as vision) but with greater uncertainty, this redundant feedback has little effect on the final state estimate [20].
In this study, we investigated a joint- and velocity-based feedback paradigm’s effect on subjects’ myoelectric control of a 2-DoF virtual limb to determine if sensory feedback provided in a local reference frame complemented visual feedback provided in a global reference frame. The dynamics of the virtual limb were perturbed during use to calculate performance impact and adaptation to the new dynamics (perturbation trials), and to determine if this adaptation was in a local or global frame (generalization trials).
These preliminary results suggest that, while subjects were able to adapt both their movement errors and movement speed to the perturbed dynamics of the virtual limb over time and return to baseline performance, initial errors were smaller when audio feedback was present. We theorize that this reduction is because the feedback was providing joint-based and velocity-based, rather than Cartesian-based, information continuously throughout the movement, allowing subjects to identify earlier within the trial that the dynamics were perturbed and facilitated earlier adaptation. This hypothesis is supported by the increase in average speed over the first few perturbation trials; because vision is relatively imprecise in determining speed [16] compared to position [15], providing separate channel of feedback encoding the velocity of the virtual limb should improve subjects’ control over limb velocity. Providing continuous joint- and velocity-based feedback also serves to provide, in essence, an efference copy of control inputs resulting in limb movements. By directly providing this efference copy through a different sensory modality, our feedback paradigm enables subjects to develop a better feedforward model of the prosthesis control [21], ultimately resulting in an improved ability to adapt to perturbations and to generalize this improvement across the entire workspace.
When asked to provide subjective feedback, several subjects commented that the audio feedback helped them to recognize when they were unintentionally contracting their muscles, allowing them to relax their muscles to prevent the virtual arm from making unintended movements, particularly between trials while waiting for the target to appear. These comments appear to support the results of Cipriani et al. [9], who found that humans are capable of integrating feedback of discrete events, such as finger contact with grasped objects, into their sensorimotor control. One can argue that the binary state of muscle activation (below- or above movement threshold), and the corresponding continuous audio feedback (below- or above hearing threshold) are discrete events, and thus subjects may have been incorporating these discrete events in their control of the virtual limb during this study. In addition to the tested hypothesis, there are several other possible benefits of sensory feedback that were not investigated during this study. One such benefit is prosthesis embodiment. Studies have shown that providing sensory feedback with prosthesis use, regardless of modality, can improve the user’s embodiment of the prosthesis and make them feel more connected to the device [8], [22], [23].
We originally expected that the perturbation would influence the generalization trial performances more than what was observed. We believe that this was likely due to our choice of perturbation. The change in damping uniformly affected the workspace. If we were to have implemented a curl field or other external perturbation [10], we may have found greater differences at the generalization targets. We also did not expect that subjects would adapt before the first trial was completed. However, upon examining video capture of subjects performing the task, the within trial adaption was apparent.
Sensory feedback remains an expansive field of study with many challenges yet to be overcome. However, addressing these challenges will give us an improved understanding of how humans incorporate multiple sources of sensory inputs, ultimately leading to improved prosthetic devices capable to restoring greater functionality and quality of life.
Acknowledgments
Research supported by NSF-NRI 1317379. E. J. Earley was supported by NIH grant T32 HD07418.
Contributor Information
Eric J. Earley, Department of Biomedical Engineering, Northwestern University, and the Center for Bionic Medicine, Shirley Ryan AbilityLab, Chicago, IL 60611.
Kyle J. Kaveny, Center for Bionic Medicine, Shirley Ryan AbilityLab, Chicago, IL 60611
Reva E. Johnson, Department of Mechanical Engineering, Valparaiso University.
Levi J. Hargrove, Center for Bionic Medicine, Shirley Ryan AbilityLab, and the Departments of Physical Medicine & Rehabilitation and Biomedical Engineering, Northwestern University, Chicago, IL 60611.
Jon W. Sensinger, Institute of Biomedical Engineering, University of New Brunswick, Fredericton, NB Canada.
REFERENCES
- [1].Young AJ, Smith LH, Rouse EJ, and Hargrove LJ, “Classification of simultaneous movements using surface EMG pattern recognition,” IEEE Trans. Biomed. Eng, vol. 60, no. 5, pp. 1250–1258, 2013. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [2].Hahne J, Biessmann F, Jiang N, Rehbaum H, Farina D, Meinecke F, Muller K, and Parra L, “Linear and non-linear regression techniques for simultaneous and proportional myoelectric control,” IEEE Trans. Neural Syst. Rehabil. Eng, vol. 22, no. 2, pp. 1–1, 2014. [DOI] [PubMed] [Google Scholar]
- [3].Kuiken T, Miller L, Turner K, and Hargrove L, “A comparison of pattern recognition control and direct control of a multiple degree-of-freedom transradial prosthesis,” IEEE J. Transl. Eng. Heal. Med, vol. 2372, no. c, pp. 1–1, 2016. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [4].He J, Zhang D, Jiang N, Sheng X, Farina D, and Zhu X, “User adaptation in long-term, open-loop myoelectric training: implications for EMG pattern recognition in prosthesis control.,” J. Neural Eng, vol. 12, no. 4, p. 46005, 2015. [DOI] [PubMed] [Google Scholar]
- [5].Childress DS, “Closed-loop control in prosthetic systems: Historical perspective,” Ann. Biomed. Eng, vol. 8, no. 4–6, pp. 293–303, 1980. [DOI] [PubMed] [Google Scholar]
- [6].Antfolk C, D’Alonzo M, Rosén B, Lundborg G, Sebelius F, and Cipriani C, “Sensory feedback in upper limb prosthetics.,” Expert Rev. Med. Devices, vol. 10, no. 1, pp. 45–54, 2013. [DOI] [PubMed] [Google Scholar]
- [7].Ninu A, Dosen S, Muceli S, Rattay F, Dietl H, and Farina D, “Closed-loop control of grasping with a myoelectric hand prosthesis: Which are the relevant feedback variables for force control?,” IEEE Trans. Neural Syst. Rehabil. Eng, vol. 22, no. 5, pp. 1041–1052, 2014. [DOI] [PubMed] [Google Scholar]
- [8].Schiefer M, Tan D, Sidek SM, and Tyler DJ, “Sensory feedback by peripheral nerve stimulation improves task performance in individuals with upper limb loss using a myoelectric prosthesis.,” J. Neural Eng, vol. 13, no. 1, p. 16001, 2016. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [9].Cipriani C, Segil JL, Clemente F, Richard RF, and Edin B, “Humans can integrate feedback of discrete events in their sensorimotor control of a robotic hand,” Exp. Brain Res, vol. 232, no. 11, pp. 3421–3429, 2014. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [10].Shadmehr R and Mussa-Ivaldi F. a, “Adaptive representation of dynamics during learning of a motor task,” J. Neurosci, vol. 14, no. 5, pp. 3208–3224, 1994. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [11].Ernst MO and Banks MS, “Humans integrate visual and haptic information in a statistically optimal fashion.,” Nature, vol. 415, no. 6870, pp. 429–433, 2002. [DOI] [PubMed] [Google Scholar]
- [12].Sobuh MMD, Kenney LPJ, Galpin AJ, Thies SB, McLaughlin J, Kulkarni J, and Kyberd P, “Visuomotor behaviours when using a myoelectric prosthesis,” J. Neuroeng. Rehabil, vol. 11, no. 1, pp. 1–11, 2014. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [13].Scheidt RA, Conditt MA, Secco EL, Ferdinando A, Kapreli E, Athanasopoulos S, Gliatis J, Papathanasiou M, Peeters R, Strimpakos N, Van Hecke P, Gouliamos A, Sunaert S, and Mussa-ivaldi FA, “Interaction of Visual and Proprioceptive Feedback During Adaptation of Human Reaching Movements,” J. Neurophysiol, no. January 2005, pp. 3200–3213, 2005. [DOI] [PubMed] [Google Scholar]
- [14].Berniker M and Körding K, “Estimating the sources of motor errors for adaptation and generalization,” Nat. Neurosci, vol. 11, no. 12, pp. 1454–1461, 2008. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [15].Ross HE, “Context effects in the scaling and discrimination of size,” Fechner Day 2003, pp. 257–262, 2003. [Google Scholar]
- [16].Stocker A. a and Simoncelli EP, “Noise characteristics and prior expectations in human visual speed perception.,” Nat. Neurosci, vol. 9, no. 4, pp. 578–585, 2006. [DOI] [PubMed] [Google Scholar]
- [17].Schmidt RA, Zelaznik H, Hawkins B, Frank JS, and Quinn JT Jr, “Motor-output variability: a theory for the accuracy of rapid motor acts.,” Psychol. Rev, vol. 86, no. 5, p. 415, 1979. [PubMed] [Google Scholar]
- [18].Atkins DJ, Heard DCY, and Donovan WH, “Epidemiologic Overview of Individuals with Upper-Limb Loss and Their Reported Research Priorities,” JPO J. Prosthetics Orthot, vol. 8, no. 1, pp. 2–11, 1996. [Google Scholar]
- [19].Ison M, Antuvan CW, and Artemiadis P, “Learning efficient control of robots using myoelectric interfaces,” Proc. - IEEE Int. Conf. Robot. Autom., pp. 2880–2885, 2014. [DOI] [PubMed] [Google Scholar]
- [20].Kording KP and Wolpert DM, “Bayesian integration in sensorimotor learning,” Nature, vol. 427, no. 6971, pp. 244–247, 2004. [DOI] [PubMed] [Google Scholar]
- [21].Kawato M, “Internal models for motor control and trajectory planning,” Curr. Opin. Neurobiol, vol. 9, no. 6, pp. 718–727, 1999. [DOI] [PubMed] [Google Scholar]
- [22].Dosen S, Markovic M, Wille N, Henkel M, Koppe M, Ninu A, Frömmel C, and Farina D, “Building an internal model of a myoelectric prosthesis via closed-loop control for consistent and routine grasping,” Exp. Brain Res, 2015. [DOI] [PubMed] [Google Scholar]
- [23].D’Alonzo M, Clemente F, and Cipriani C, “Vibrotactile stimulation promotes embodiment of an alien hand in amputees with phantom sensations.,” IEEE Trans. Neural Syst. Rehabil. Eng, vol. 4320, no. c, pp. 450–457, 2014. [DOI] [PubMed] [Google Scholar]