Electroencephelograph Based Brain Machine

Interface for Controlling a Robotic Arm

Wenjia Ouyang, Kelly Cashion, Vijayan K. Asari
Department of Electrical and Computer Engineering,
University of Dayton, Dayton, OH 45410, USA

Abstract—A brain machine interface (BMI) facilitates

the control of machines through the analysis and I. INTRODUCTION
classification of signals directly from the human brain.
Using an electroencephalograph (EEG) to detect Brain computer interfacing (BCI), also known as brain
neurological activity permits the collection of data machine interfacing (BMI), is a field of research that has been
representing brain signals without the need for invasive explored throughout the last few decades. It deals with the
technology or procedures. A 14-electrode EPOC headset manipulation of computers and machines by using thought as
produced by the Emotiv Company is used to capture live the control mechanism. Initial research in the field used
data, which can then be classified and encoded into invasive technology, attaching electrodes inside the skull,
control signals for a 7-degree-of-freedom robotic arm. often on the brain itself, in order to read the electric signals
The collected data is analyzed in using an independent produced by the human brain. Such systems require surgery
component analysis (ICA) based feature extraction and a in order to connect the subjects to the sensors. Trends now
neural network classifier. The collected EEG data is have moved toward systems that make use of non-invasive
classified into one of four control signals: lift, lower, rotate physiological sensors such as electroencephalographs (EEG)
clockwise, and rotate counter-clockwise. Additionally, the [1]. EEG systems use a number of electrodes to read electrical
system watches the collected data for electromyography signals on the scalp caused by brain activity. This number
(EMG) signals indicative of movement of the facial can vary from only a single electrode to as many as 256 or
muscles. Detections are used to incorporate two additional more.
control signals: open and close. A personal set of EEG
data patterns is trained for each individual, with each BCI systems come in two distinct varieties, synchronous
control signal requiring only a few minutes to train and asynchronous. Synchronous systems use cues to indicate
initially. EMG signal detections are measured against a windows of time that the system looks for a particular signal.
generic threshold for all users. Once a user has trained It is only useful for the user to attempt to generate a particular
their personal data into the system any positive detections thought pattern recognized by the system during that time. At
trigger a signal to the interfaced robotic arm to perform a any time outside these specific windows, the system makes no
corresponding, discrete action. Currently, subjects are attempt at classification. Asynchronous systems do not limit
able to repeatedly execute two EEG commands with the system to discrete windows, instead they aim to recognize
accuracy within a short period of time. As the number of and respond appropriately to a particular thought pattern
EEG based commands increases, the training time regardless of when it is detected [2]. While synchronous
required for accurate control increases significantly. systems are easier to implement, asynchronous systems fulfill
EMG based control is almost always immediately the needs of a much wider array of applications.
responsive. In order to extend the range of available
controls beyond a few discrete actions, this research Another variable characteristic of BCI systems is the
intends to incorporate and refine the algorithmic steps of particular EEG signal, or other brain signal response, used to
classification and detection to shift an increased prompt specific responses from the rest of the system. Some
percentage of the burden of training onto the computer. systems are controlled by evoked potentials, most commonly
the steady-state visual evoked potential (SSVEP) or P300
Keywords--BCI, Robotic Arm, EEG, ICA, MLP, Emotiv
evoked potentials. Alternatively, some systems use
endogenous brain signals, such as movement related
potentials, to communicate intention [1]. It is also possible to

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY KHARAGPUR. Downloaded on August 16,2021 at 05:50:53 UTC from IEEE Xplore. Restrictions apply.
 use electromyography (EMG) and electrooculography (EOG) It is wireless and utilize a proprietary USB dongle to
signals as control mechanisms. EMG signals are generated by communicate using the 2.4GHz band.
muscle movement and EOG signals are generated by eye
B. System Software and Control Commands
movement. Often, these signals are considered noise artifacts
and their removal from pure EEG signals is desirable. The Research Edition SDK provided with the EPOC
However, the same distinguishing characteristics used to headset offers a variety of solutions for creating an interface
recognize and extract these artifacts with noise removal with the robotic arm. In addition to a robust API, the included
algorithms are also easily recognized features used to classify software interfaces with the EPOC headset in a useful way.
the signals as part of BCI control [3]. Additionally, a program is needed to control the robotic arm.
The necessary software is (1) the Emotiv Control Panel, (2)
The possible uses for a BCI system are as wide-ranging as the EmoKey application, and (3) the CytonControl
the applications of computer programs and computer application.
controlled devices. One of the most noteworthy applications
of BCIs is in the field of medicine. BCIs offer tools for the The Emotiv Control Panel manages the connection to the
disabled to interact with the world, even if they are suffering headset and processes EEG data in real time according to a
from severe nerve damage between their brain and original number of suites. Of particular relevance is the Cognitiv suite,
limbs. In the case of a lost or paralyzed limb, BCIs offer the which handles the recognition of trained commands from pure
potential for patients to use a robotic limb, controlled with EEG data. The tab dealing with the Cognitiv suite presents a
their natural thought patterns, to interact with the world. virtual block and allows the user to establish which
commands to use and train those commands as well as a
neutral state. An additional feature of the Cognitiv suite tab is
In this research, a BCI is implemented by using a Robai
the ability to adjust the sensitivity to a certain command.
Cyton Veta robotic arm, the Emotiv headset, and several
Qualitative analysis has shown that adjusting the sensitivity of
software applications included with the Emotiv system. The
EEG signals are used to control the robotic arm. Initially, the all commands equally towards a higher sensitivity usually
system is developed using the provided Emotiv software to provides better response times.
handle not only interfacing but also to solve the basic
classification problem. After this proof of concept, a new
approach is used to handle feature extraction and
classification of the EEG data.

II. EMOTIV BASED SYSTEM

A. Hardware Setup
The initial implementation of the BCI takes advantage of
the control patterns already recognized by the Emotiv
software, encoding them as commands for the robotic arm. In
this research, an Emotiv EPOC headset, Emotiv Research
Edition SDK, and Robai’s Cyton Veta 7-Degree-of-Freedom Fig. 2: Emotiv Control Panel Cognitive suite tab
robotic arm are used to create the brain-machine interface, as
shown in Fig. 1.
The EmoKey software facilitates mapping specific
commands recognized by the Emotiv suites—such as the
Cognitiv suite—to keystrokes in a given application window.
The control panel interfaces with the EmoKey which has
appropriate keystrokes mapped to each of the trained
commands corresponding to an appropriate action performed
by the robotic arm. For each action a threshold value is set
that must be achieved before a keystroke is performed. An
Fig. 1: Equipment setup
appropriate threshold varies by user, but most often a
confidence of 0.3 (out of 1) is used as the threshold value.
The robotic arm has a 60 cm reach in all directions and is Causes for reducing the threshold include stress or tiredness
kept in a marked zone separate from the user. It is connected in the subject or excess noise in the environment. In both
to the computer via a USB cable. The EPOC headset used in cases, the added noise makes higher power thresholds more
this research contains 14 EEG sensors, a rechargeable 12 hour difficult to reach.
lithium battery, and has an effective bandwidth of 0.16-43Hz.
An important piece of software, not provided by Emotiv,
is the CytonControl application. This program manipulates

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY KHARAGPUR. Downloaded on August 16,2021 at 05:50:53 UTC from IEEE Xplore. Restrictions apply.
 the robotic arm by using specific keystrokes as input, each The majority of subjects are able to train with the system
corresponding to one of six possible commands. The six and manage one to two Cognitiv control signals without
commands are grab, release, lift, lower, rotate left, and rotate excessive difficulty. Typically the commands chosen are lift
right. Several provisions are made as part of the CytonControl and lower. However, adding the additional commands for
software to prevent the robotic arm from attempting to rotation reduce the accuracy of command execution for all the
execute redundant actions or collide with known obstacles. control signals to an almost unusable degree without
The current state of the robotic arm is checked before each extensive time practicing with the system. The EMG based
command execution to insure that the next command it has grab and release controls are easily performed by the majority
been sent is valid. For example, lift is only executed if the of subjects.
system is in the lowered position. Rotations occur unless the
robotic arm has reached the limit of its range of motion. III. A NEW LEARNING SYSTEM
Before these changes were implemented, seemingly random
3-5 second pauses would occur where the system would be With the Emotiv based system, training is easily achieved
unresponsive. This was caused by the delays implemented in within a time span of 30 minutes for one or two actions to
the code to guarantee the robotic arm completes a motion achieve a reasonable level of accuracy. However, for each
before accepting the next command. action added it becomes more difficult to retain accuracy. In
order to gain a high efficiency and accuracy when training the
The keystrokes from the EmoKey program are sent to the
robotic arm, a new learning system is designed in this
CytonControl program which in turn sends command to the
research, as shown in Fig. 4.
robotic arm. Fig. 3 shows that Rule 3 is corresponding to the
action lift and the trigger conditions of it is when power is
greater than 0.2.

Fig. 4: The schematic diagram for action encoding

This new learning system includes two distinct phases:

feature extraction and classification. Data collection is
performed using the TestBench application, provided with the
EPOC headset. This software records and displays the EEG
data directly from the headset. Raw data is broken into
10~20s segments of EEG signal from EPOC headset with 14
channels (electrodes) each. Each segment corresponds to a
specific command.
Fig. 3: The interface of EmoKey
In order to acquire exact information from the EEG,
In addition to the Cognitv suite commands used, control Independent Component Analysis (ICA) is used to preprocess
for the grab and release function of the robotic arm is tied to the original data and remove artifacts. Outputs of the ICA
detection of EMG signals through the Emotiv headset. These algorithm are mutually independent components. Therefore,
signals indicate muscle movement, and are both distinctive dominant features of EEG are extracted by this method. After
and similar across the majority of subjects. For grab and that, a classification algorithm, Multilayer Perceptron (MLP)
release, clenching the jaw is used as the command signal. is applied. The processed data is encoded into appropriate
actions for the robotic arm.
From the Cognitiv suite, lift, drop, rotate left, and rotate
right are mapped to the robotic arm’s lift, lower, rotate left, IV.THEORETICAL APPROACH
and rotate right commands. A. Feature Extraction
C. Initial Results. The separation effectiveness of ICA algorithms are related
to statistical properties of the mixing signals on which they
After training with the robotic arm with 4 actions are applied. The FastICA algorithm is used in this research,
frequently over the course of a month, adequate results can be which is a computationally highly efficient method for
achieved. However, users may need to think a command for a performing the estimation of ICA. It uses a fixed-point
relatively long time (more than 30s) in order to obtain the iteration scheme developed by Aapo Hyvärinen, which has
desired response. Misclassifications remain a serious problem been found in independent experiments to be 10-100 times
though are reduced with increased training time. Currently a faster than conventional gradient descent methods for ICA [4,
large percentage of the burden of learning rests on the user to 5].
practice consistent command thoughts.

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY KHARAGPUR. Downloaded on August 16,2021 at 05:50:53 UTC from IEEE Xplore. Restrictions apply.
 In this research, the input data, represented by X, is a linear perceptron and can distinguish data that is not linearly
14×1000 data matrix, where each row includes EEG signals separable.
from one electrode (channel). The first step of FastICA
algorithms is to remove the sample mean and whiten the data. In this research, the processed data (a matrix of 14*1000)
with ICA representing one action is the input to the multilayer
network. The output indicates which of the action (drop, lift,
−1/ 2
rotate left, rotate right and neutral) is thinking when collecting
X =D ET ( X − X ) (1)
this input data. A fully connected network structure of MLP is
where shown in Fig. 5.
E{ XX T } = EDE T (2)
X is the sample mean, E is the matrix of eigenvectors and
D is the diagonal matrix of eigenvalues. The output whitening
data X contains linear transformation, so it is uncorrelated
unit variance data.

Next, the fixed-point algorithm uses an iterative approach

to find the direction for the weight vector W. As Hyvärinen
concludes, power 3 is used as the contrast function in this Fig. 5: Network Structure of MLP
algorithm, that is
In between the input layer and the output layer, it is the
3 hidden layers of the network. In addition to the predictor
g (u ) = u (3)
variables, there is a constant input, called the bias that is fed
and to each of the hidden layers. The bias is multiplied by a
g ' (u ) = 3u 2 (4) weight and added to the sum going into the next nodes. At the
'
where g (⋅) denote the first derivative of the function g (⋅) . beginning, input the data and the weights to the system. Vji
represents the weight between the input layer and the hidden
The initial weight vector W is random. Then, iterate layer, which is a random matrix with the range between -1 to
W + = E { Xg (W T X )} − E {g ' (W T X )W } (5) 1.
W* =W+ / W+ (6)
until convergence is achieved. The value from each input is multiplied by Vji, and the
resulting weighted values are added together producing a
Hyvärinen states that the FastICA has advantages: (1) it is combined value net_pl, as
parallel, (2) distributed, (3) computationally simple, and (4)
requires little memory space. Stochastic gradient methods N
seem to be preferable only if fast and adaptive performance in net _ pl = ∑ vli x pi − v0 (7)
a changing environment is required. Independent components i =1

of any non-Gaussian distribution are directly found by using and

any nonlinear function g. This is in contrast to many 1
algorithms, where some estimate of the probability u _ pl = (8)
1 + e− net _ pl
distribution function has to be available and the nonlinearity u_pl is the output from the hidden layer, and is distributed to
must be chosen accordingly [4]. The Matlab package using in the output layer. Next, the value from each hidden layer is
this research was programmed by Hugo Gavert, Jarmo Hurri, multiplied by another weight Wjl, which is initially randomly
Jaakko Sarela, and Aapo Hyvarinen and download at generated, with the range between -1 to 1. And the resulting
www.cis.hut.fi/projects/ica/fastica/. weighted values are added together producing a combined
value net_pj, as
B. Classification
An MLP network is used to classify the data in order to N

control the system. It is a feed forward artificial neural net _ pj = ∑ w jl u pl − w0 (9)

l =1
network model that maps sets of input data onto a set of
appropriate output. An MLP consists of multiple layers of The net_pj is fed into a transfer function, which is contributed
nodes in a directed graph, with each layer fully connected to to the final output, represented by y_pj, of the entire network
the next one. Except for the input nodes, each node is a
neuron with a nonlinear activation function. MLP utilizes a 1
supervised learning technique called back propagation for y _ pj = − net _ pj
(10)
1+ e
training the network. MLP is a modification of the standard

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY KHARAGPUR. Downloaded on August 16,2021 at 05:50:53 UTC from IEEE Xplore. Restrictions apply.
 Next, according to the error, adjust weights iteratively, as format, EDF, which is compatible with many EEG analysis
shown following: programs such as EEGLab, an open source project for EEG
data processing in Matlab. Following the initial information
Δw jl = η ⋅ (d − y _ pj ) ⋅ y _ pj ⋅ (1 − y _ pj ) ⋅ u _ pl (11) line, each successive row in the data file corresponds to one
data sample. Successive rows correspond to successive time
w jl (t + 1) = w jl (t ) + Δw jl (t ) (12) slices. Each column of the data file corresponds to an
M individual sensor or other information tag.
δ pl = ∑ w jl ( d − y _ pj ) (13)
j =1

Δvli = η ⋅ δ pl ⋅ u _ pl ⋅ (1 − u _ pl ) ⋅ x _ pi (14)
vli (t + 1) = vli (t ) + Δvli (t ) (15)
where η is called the learning rate parameter.

Finally, check the mean square error E. When E is smaller

than a particular small value, stop to adjust the weights. The E
is supposed to decrease exponentially. When error E is stable
and small enough, weights Wij and Vij are applied to classify Fig. 7: Locations of electrodes
the data.
In this research, 150 EEG signal samples with 5 actions
V. EXPERIMENTAL APPROACH (drop, lift, rotate right, rotate left and neural) are collected by
the TestBench for testing our action encoding system, each
A. Data Collection with 30 samples. First, open TestBench and click the Save
An additional piece of software provided with the Emotiv Data button. After thinking of the given command for 10s,
SDK, Testbench, can independently collect data packets from data is saved. In the TestBench, data can be saved as an EDF
the USB device and processes them to display, analyze, record file, and covert to CSV file. Each CVS file includes 1000
and play back time-dependent EEG signals, as shown in Fig. 6. successive voltage values for each channel. That is, each data
is treated as a 2-D matrix of 14*1000.
B. Testing the New Learning System
The FastICA algorithm is used to extract features from
original EEG data. The user interface is shown in Fig. 8.
First on data segment is loaded. This includes 14 mixed
EEG signals from 14 electrodes, and each channel includes
1000 voltage sample values. Then, run the ICA algorithm,
which runs fixed-point iteration method of ICA. The reduced
dimension is 14 and Eigen values are shown in Fig. 9.
In this way, ICA data is obtained. Fig. 10 shows signals
from channel 11 to 14 (FC6, F4, F8, AF4) when the user
Fig. 6: Emotiv TestBench thinking of action lift, and the left section is ICA data, while
the right section is original data. From Fig. 7 above, it is easy
This TestBench involves EEG Suite, FFT Suite, Gyro to find these 4 channels located closely together (all in the top
Suite and Data Packets Suite. The EEG Suite is used in this right section of head). The original signals from these channels
research. It reports real time changes in the subjective actions are similar, as shown in the right side of Fig. 10. However,
experiences by the users. EEG shows brainwave signals of 14 after applying ICA to data, as the left side of Fig. 10 shown,
channels (AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, signals from these 4 channels of processed data are very
F8, and AF4). The correspondence locations of these channels distinct. This is because the features of original EEG signals
are shown in Fig. 7. are successfully extracted by ICA method, removing the noise
In EEG Suite, the users can choose to display one or more interference in each of those channels.
channels, and if users choose to display one channel, In order to reduce the size of data, 10 nearby samples from
AutoScale button can be used. The main function of a channel are replaced by its average value. Therefore, original
AutoScale button is automatic alignment of the upper and
1000 values from a channel are replaced by 100 average
lower limits consistent with the current channel values. The
Sampling Frequency displays when the program implements. values. Finally, a 3-D matrix of 14*100*150 (using 150 EEG
For EPOC headset, this is fixed at 128 samples per second. data segments for different commands, each one has 14 signals
Data can be collected by using the Save Data button. In the from 14 channels, and each signal has 100 sample values) is
child window, users can choose the path to save data and type created as the input database.
name file. Data is stored by Testbench in a standard binary

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY KHARAGPUR. Downloaded on August 16,2021 at 05:50:53 UTC from IEEE Xplore. Restrictions apply.
 hidden nodes L, the mean square error steadies at a lower
value with less oscillation, as ŋ=0.5, when L=50, mean square
error is 2.18, almost 3 times lower than the mean square error
when L=20. With the decreasing value of learning rate
parameter ŋ, convergence characteristics improve and the
mean square error steadies at a lower level. That is, the more
hidden layer nodes and the lower the learning rate parameter
that are used, the lower mean square error and the better
testing performance are produced. However, using too many
hidden nodes wastes time on training without a relevant
decrease the percentage error significantly. Therefore, in this
research, the hidden layer of MLP uses 40 nodes, and the
learning rate used is 0.5. The learning curve is shown in Fig.
11 below.
Fig. 8: the user interface of FastICA

Fig. 9: Eigen values

Fig. 11: Testing result

TABLE 1
THE FINAL CONVERGENT MEAN SQUARE ERROR

L ŋ Mean Square Error

20 0.2 5.23
20 0.5 6.07
Fig. 10: Features extraction by FastICA
20 0.8 16.21
30 0.2 2.44
Next, classification is through MLP. The first 130 data 30 0.5 2.77
segments are used for training the system. The remaining 20
data are used for testing. Each segment data is a 14*100 matrix, 30 0.8 4.13
so the number of input layers is 1400. There are two important 40 0.2 2.11
parameters in MLP, the number of hidden layer, represented 40 0.5 2.25
by L, and the learning rate, represented by ŋ. Different
combination of L and ŋ produce different learning curve, 40 0.8 3.05
which is convergence characteristics that illustrates the mean 50 0.2 2.02
square error versus iterations (one iteration applies all the 50 0.5 2.18
training inputs once to the network and compute the mean
square error), and the final convergent mean square error 50 0.8 3.18
effects the testing performance. Table 1 shows the final
convergent mean square error with different combination of L
and ŋ.
By comparing the final convergent mean square error in
Table 1, it is easy to see that with the increasing number of the

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY KHARAGPUR. Downloaded on August 16,2021 at 05:50:53 UTC from IEEE Xplore. Restrictions apply.
 match to a particular command. Thus, it will help actions to be
TABLE 2 more accurately represented when collecting data, leading to
DETAILS OF TESTING DATA easier classification.
Mark 1 2 3 4 5 Moreover, a better approach is not choosing methods for
rotate rotate the feature extraction and classification separately, but
Action drop lift right left neural combining them, which means choosing features which
amount optimizes the classifier’s objective. The fixed-point ICA
among maybe a good method to extract features of EEG signals, but
testing data 4 3 6 3 4 these ICA signals maybe not good to be used as the input to
MLP. Because MLP is sensitive to overtraining especially
with noisy data, so a good deal of care must go into the
selection of specific architectures and regularization.
REFERENCES
[1] Blankertz, Benjamin, Guido Dornhege, et al. Boosting Bit Rates and
Error Detection for the Classification of Fast-Paced Motor Commands
Based on Single-Trial EEG Analysis. IEEE Trasactions on Neural
Systems and Rehabilitation Engineering. 11.2 (2003): 127-31. Print.
[2] Shenoy, Pradeep, Matthias Krauledat, et al. "Towards adaptive
classification for BCI." Journal of Neural Engineering. 3. (2006): R13-
R23. Print.
[3] Cunnigham, Patrick; Lang, Keith; Mearn, Brian; Russel, Lee; and
Sanchez, Stephen, “EEG brain-computer interface project” (2007).
Capstone Design Program: Electrical and Computer Engineering. Paper
2.
[4] Hyvärinen. Fast and Robust Fixed-Point Algorithms for Independent
Component Analysis. IEEE Transactions on Neural Networks
Fig. 12: Testing result 10(3):626-634, 1999.
[5] Hyvärinen and E. Oja. Independent Component Analysis: Algorithms
The last 20 samples in our database are used for testing. and Applications. Neural Networks, 13(4-5):411-430, 2000.
They include 4 actions and a neural status. Details are shown [6] M.D. Eggers and T.S Khon. Learning Algorithms for the Multilayer
in Table 2. Perceptron. Technical Report 813.October 1988.
[7] V.Matic, W. Deburchgraeve and S. Van Huffel. Comparison of ICA
The percentage error for the testing results is shown in Fig. Algorithms for ECG artifact removal from EEG singals. IEEE-EMBS
12. Action 1 to 5 represent drop, lift, rotate right, rotate left Benelux Chapter Symposium. November 9-10, 2009
and neutral, respectively. The testing data involves 4 drop
signals, 3 lift signals, 6 rotate right signals, 3 rotate left signals
and 4 neural signals. The majority of classification errors are
below 30% for these commands, which means the full learning
system can efficiently recognize EEG signals. Lift performs
the best, which has no error in testing. Rotate left is the most
difficult to be recognized, and rotate right is also challenging.
This is likely a result of the similarity between the two
commands. A focus on left and right rather than rotation might
produce more distinct features.
VI. CONCLUSIONS
In a summary, the new implementation of the BCI system
addressed here has proven to be effective for identifying
actions from the EEG brainwave signals. It will enhance the
successful controlling rate to the robotic arm after converting
these Matlab code into C++ and adding them to the main
control file CytonControl.cpp. Future work involves acquiring
more EEG data and adding to the database for training and
testing the MLP network. At the same time, the quality of
original EEG data should be improved. Subjects can practice
using consistent thoughts for control signals by using the
software provided with the Emotiv system. By running a demo
program or practicing manipulating the cube with the
Cognitive suite before collecting original data, users can
practice thinking of consistent commands. This will allow
users to get accustomed to thinking of the same thought to

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY KHARAGPUR. Downloaded on August 16,2021 at 05:50:53 UTC from IEEE Xplore. Restrictions apply.