Biomedical Signal Processing and Control 65 (2021) 102351

Contents lists available at ScienceDirect

Biomedical Signal Processing and Control

journal homepage: www.elsevier.com/locate/bspc

Toward asynchronous EEG-based BCI: Detecting imagined words segments

in continuous EEG signals
Tonatiuh Hernández-Del-Toro ∗, Carlos A. Reyes-García, Luis Villaseñor-Pineda
Biosignal Processing and Medical Computing Lab, Instituto Nacional de Astrofísica, Óptica y Electrónica, Mexico

ARTICLE INFO ABSTRACT

Keywords: An asynchronous Brain–Computer Interface (BCI) based on imagined speech is a tool that allows to control an
Imagined speech external device or to emit a message at the moment the user desires to by decoding EEG signals of imagined
Asynchronous BCI speech. In order to correctly implement these types of BCI, we must be able to detect from a continuous
Signal processing
signal, when the subject starts to imagine words. In this work, five methods of feature extraction based on
wavelet decomposition, empirical mode decomposition, frequency energies, fractal dimension and chaos theory
features are presented to solve the task of detecting imagined words segments from continuous EEG signals
as a preliminary study for a latter implementation of an asynchronous BCI based on imagined speech. These
methods are tested in three datasets using four different classifiers and the higher 𝐹1 𝑠𝑐𝑜𝑟𝑒𝑠 obtained are 0.73,
0.79, and 0.68 for each dataset, respectively. This results are promising to build a system that automatizes the
segmentation of imagined words segments for latter classification.

1. Introduction windows. This problem can be solved by being able to detect the onset
and ending of Imagined Words Segments (IWS) in continuous EEG
A Brain–Computer Interface (BCI) is a tool that allows the use signals. The detection of the onset and ending of IWS in continuous
of brain signals to control an external device or to emit a message.
EEG signals can lead to the detection of certain words (like up, down,
There exist several ways of implementing a BCI: (i) by the brain signal
acquisition technique, (ii) by the neuroparadigm used to control the left, right and select) that are relevant to the BCI actions such as moving
BCI, and (iii) by their response to trigger an order, being synchronous, a PC pointer, or inside an iconic navigation interface with symbols to
or asynchronous [1]. There are a wide gamma of brain signal acquisi- perform specific actions like commanding the driving of a wheelchair
tion techniques, we make use of non-clinical Electroencephalography with a defined vocabulary [5,6]. Despite this problem has not yet been
(EEG) recording headsets that have shown to be capable to build solved, some approaches have been proposed as to detect the onset of
BCI as the EMOTIV EPOC+ [2] and the BrainVision [3] due to its
similar neuroparadigms like imagined sound production [7–10].
non-invasiveness, fast setup, wearability, portability and low cost in
In this work, five methods of feature extraction are presented to
comparison with clinical EEG devices and other techniques (functional
Magnetic Resonance Imaging, functional Near-Infrared Spectroscopy). solve the task of detecting IWS from continuous EEG signals as a prelim-
Common neuro-paradigms used focus mainly on Evoked Potentials, inary study for a latter implementation of an asynchronous BCI based
Motor Imagery, and more recently the Imagined Speech, where the on imagined words that controls a PC pointer. Our work experiments
Imagined Speech is defined as ‘‘pronouncing internally to oneself a with three treatments of the EEG signal: (i) wavelet decomposition,
syllable, word or phrase without producing any sound or moving any (ii) empirical mode decomposition, and (iii) only noise cleaned signal.
articulation’’[4].
From these treated signals, five feature sets based on energies of the
An asynchronous BCI based on imagined speech has the advantage
over other BCIs that is more direct, and is not restricted to image frequency bands, measures of chaos theory and fractal dimensions
abstract body movements, also that it is the most natural way of are proposed. This work is focused on finding the best features sets
communication of human beings. In order to build an asynchronous to identify IWS from continuous EEG signals. Using four different
BCI that takes EEG signals as input and uses imagined speech as neu- classifiers, these methods are tested on three datasets that contain IWS
roparadigm, we must be able to detect when the user starts to imagine in continuous EEG signals.
the word, otherwise, the BCI would only act in certain predefined

∗ Corresponding author.
E-mail addresses: [email protected] (T. Hernández-Del-Toro), [email protected] (C.A. Reyes-García), [email protected] (L. Villaseñor-Pineda).

https://doi.org/10.1016/j.bspc.2020.102351
Received 25 May 2020; Received in revised form 19 October 2020; Accepted 15 November 2020
Available online 30 November 2020
1746-8094/© 2020 Elsevier Ltd. All rights reserved.
T. Hernández-Del-Toro et al. Biomedical Signal Processing and Control 65 (2021) 102351

2. Related work

Although the problem of identifying IWS from continuous EEG

signals has not been solved yet, there is no known baseline to whom
compare metrics. This research is based on some past works that lead
the way into solving this problem.
Using a Time Error Tolerance Region, in [11] was tried to identify
the onset of linguistic segments in continuous EEG signals. There have
been studies on movement imagery where it was tried to identify from
an EEG signal, the onset of the imagined movement in asynchronous
BCIs [12]. These works aim to predict the onset by analyzing the
Movement Related Cortical Potentials [13,14]. These studies have been
promising and have lead to the building of imagined movement-based
BCIs. However, the speech is generated in Broca’s and Wernicke’s areas
and there is not a direct relation between imagined speech and real
speech because the imagined speech is detected over all the brain [6]. Fig. 1. Placement of the 14 electrodes used in datasets 1 and 2. This same electrodes
This leads us to the use of other features for detecting imagined speech are selected in the third dataset to have the same channels.
in EEG. There also have been studies where the neuroparadigm to
identify is the onset of imagined sound production [7–10]. In these
works, the subject imagines a high pitch sound as well as a siren-like
sound, and this is used to trigger an action in an asynchronous BCI. In
the study of imagined speech, there have been several works that aim to
classify words or vowels [5,15–17]. These works have shown promising
results in this field. However, all of them deal with signals which have
been already segmented by hand into imagined speech segments. It is
necessary to deal with the continuous signal in order to detect the IWS.
The detection of IWS can be seen as a pre-processing of signal in the
sense that solving this problem will lead to an automatic segmentation
of imagined speech segments in signals and with this, the classification
of the word can be made.

3. Materials and methods

Fig. 2. Protocols of recording trials in each of the three datasets and the duration of
All signal cleaning, trial segmentation and feature extraction meth- each stage. The cross means that the subject must stare at the screen paying attention.
ods were implemented in Matlab. The machine learning part was
implemented in Python using the Sklearn libraries.
Table 1
Description of the differences in the 3 datasets.
3.1. Datasets Dataset No. Subjects No. of IW Trials per word No. Trials
1 27 5 20 100
Three datasets containing recordings of IWS in continuous EEG 2 27 5 32 160
signals are used in this work. Datasets 1 and 2 were recorded using the 3 20 4 40 160
EMOTIV EPOC+ [18] headset, which has 14 channels and a sampling
rate of 128 Hz using the recording software TestBench. Dataset 3 was
recorded with the BrainVision [19] headset which has 64 channels and
a sampling rate of 500 Hz using the recording software BrainVision the subjects imagine, appear randomly, and were recorded in only one
Professional Recorder. Although in this third dataset the sampling rate session. Each subject has recordings of 160 trials, each word appeared
was 500 Hz, the recordings were down-sampled to 128 Hz, and only randomly 32 times. Each trial consists of an idle state given by a cross
the 14 channel used in dataset 1 and 2 were selected to have the fixation, followed by the presentation of the word, then followed by the
same sample rate and channels on the three datasets, and with this, imagined word and ended by a black screen (idle state).
to have a better comparison of results. The reference electrodes used The third dataset consists of 20 subjects that imagine four words
for recording the three datasets were P3 and P4. In Fig. 1 are shown in Spanish which mean (up, down, left, right). Similarly to the second
the 14 electrodes used in the three datasets. dataset, the words appear randomly 40 times each one in one session.
In the three datasets, all subjects were informed about the process Each subject has recordings of 160 Trials. Each trial consists of an idle
of recording the signals, and the privacy details in which the data can state given by a cross fixation, followed by a picture of a house whose
only be used for research purposes. Also, we collected the documents position indicates the word to imagine, followed by a screen showing a
of informed consent that each subject read and signed before the text that reads ‘‘The house was’’, then followed by the word imagination
recordings. and followed by a black screen (idle state).
The first dataset consists of 27 subjects that imagine five words in The protocols of recording trials in each dataset can be seen in
Spanish which mean (up, down, left, right and select). Each subject has Fig. 2, as well as the average time of each stage in the train of stimuli.
recordings of 5 sessions, 20 trials per session, each trial consists of an All three datasets were recorded from male and female students
idle state followed by an imagined word, followed by another idle state. between ages from 20 to 30 years old. In Table 1 is summarized the
This dataset contains 100 Trials per subject and is well described in [5], description of each dataset.
it has been also used for imagined word classification [15]. For dataset 1, the imagined word will be taken as an IWS while the
The second dataset consists of 27 different subjects that imagine five idle states will be taken as Idle State Segments (ISS). For dataset 2, the
words in Spanish which mean (up, down, left, right and select). The states of cross fixation as well as the idle state at the ending will be
main difference with respect to the first dataset is that the words that taken as ISS. The word presented and the imagined word will be taken

2
T. Hernández-Del-Toro et al. Biomedical Signal Processing and Control 65 (2021) 102351

Fig. 3. Example of a trial for all three datasets. It contains an ISS followed by an IWS
followed by another ISS.

as IWS. For dataset 3, the states of cross fixation, the picture shown, and
the idle state at the ending, will be taken as ISS. The text presented and
Fig. 4. Segmentation of signal when the trial belongs to the training set. The signal
the imagined word are taken as IWS. In this sense, for every dataset,
instances of ISS are taken from two parts: (i) moving a window from the start of the
each trial will be composed by and ISS followed by an IWS followed by trial to the onset of the IWS and (ii) moving a window from the ending of the IWS
another ISS as shown in Fig. 3. to the ending of the trial. The signal instances of IWS are obtained moving a window
In this scheme, all the samples prior and after the IWS are treated from the start of the IWS to the end of the IWS. In this sense, each signal instance
as ISS and the samples in between the imagined speech segment are contains only samples that belong to their class and there is no signal instance with
samples of ISS and IWS simultaneously.
treated as IWS, no matter which word the subject imagines, since the
primary focus of the research is to identify the IWS within the trial and
not to identify the word itself.

3.2. Signal cleaning

To reduce noise and to remove artifacts from the signals, the Com-
mon Average Reference (CAR) method is applied. This method is
chosen because it has also been used in other works that deal with noise
and artifacts in EEG signals [5,6,15,16,20,21].
Fig. 5. Segmentation of the signal when the trial belongs to the test set. In this scheme,
3.3. Cross validation scheme as there is no label for the signal instances, a window will be moving from the start
of the trial to the end of the trial and all signal instances will be taken no matter if
Imagined speech is a task that is subject dependent done and thus, they have mixed samples of ISS and IWS simultaneously.
using a global classifier would lead to poor performance. This property
of imagined speech being subject dependent has also been used in a
manner in which imagined words states and rest states of each subject 3.4.2. Testing set trial segmentation
are used as a biometric system for subject identification [21–23]. Due If the trial belongs to the testing set, although we know the actual
to this behavior, in this work, for the three datasets, an individual
onsets and ending markers, we are not going to use them and the
classifier was trained for each subject. The set of all trials of each
extraction of signal instances will be continuous over all the trial. From
subject was divided into a training trials set, and a test trials set, taking
the beginning, a window of 0.5 s will extract a signal instance and
75% of trials for training the models and 25% of trials for testing
will be moved 0.1 s each time until it reaches the end of trial yielding
the models. This procedure was repeated 4 times and all results were
to a set of signal instances to be classified after by the model. In this
averaged to obtain a 4-fold validation scheme.
scheme, some signal instances will have samples of ISS and samples of
3.4. Trial segmentation IWS simultaneously as illustrated in Fig. 5

The segmentation for each trial is made with a window of 0.5 s that 3.5. Feature extraction
is moving with an overlap of 0.1 s. This window will output a signal
instance. Also, there will be two types of methods to segment the trials, To extract features from the signal instances, first the signal is
these types will depend if the trial belongs to the training set or to decomposed and then, the features are extracted. We decompose signals
the testing set. For the signal instances extracted either for the training in three ways: (i) Discrete wavelet transform, (ii) Empirical Mode
set or the testing set, each signal instance will be an 64 × 14 matrix decomposition, and (iii) Only cleaned signal. The features used in this
where 64 denotes the number of samples and 14 denotes the number work contain information about the frequency of signals because we
of channels. know that many of the information of EEG is encoded within the
frequency domain, and also the features used contain information about
3.4.1. Training set trial segmentation fractal dimensions, and chaos theory. This is due to the natural chaotic
If the trial belongs to the training set, we know a priori the actual and nonlinear behavior of EEG signals.
onset and ending of the IWS, this markers will help us to determine the
number of signal instances of ISS and the number of signal instances of 3.5.1. Discrete wavelet transform
IWS. From the beginning of the trial, a window of 0.5 s extracts a signal The discrete wavelet transform (DWT) [24] of a signal is a method
instance of ISS, then the window is moved by 0.1 s and another signal of decomposing a signal into 𝑗 sets called detail coefficients 𝑤𝑗 (𝑟) and
instance of ISS is extracted, this is repeated until the end of the window an approximation coefficient 𝑎(𝑟). This coefficients give information
reaches the onset of the IWS. Then the window of 0.5 s now begins about how each frequency band varies in time. For a signal 𝑋(𝑡), the
from the onset of the IWS, and from this window, a signal instance of
DWT is defined as:
IWS is extracted, again the window is moved 0.1 s and another signal
instance of IWS is extracted until the end of the window reaches the ∑
𝑚
𝐷𝑊 𝑇 (𝑛, 2𝑗 ) = 𝑋(𝑡)𝛹2∗𝑗 (𝑡 − 𝑛) (1)
ending of the IWS. Then, the window now starts from the ending of 𝑡=0
the IWS, and from this, a signal instance of ISS is extracted, then the 1
window is moved 0.1 s and another signal instance of ISS is extracted with 𝛹2𝑗 (𝑛) = √ 𝛹 ( 2𝑛𝑗 ), where 𝛹 is the mother wavelet and 𝐷𝑊 𝑇 (𝑛, 2𝑗 )
2𝑗
until the end of the moving window reaches the end of the trial. This is the discrete wavelet transform at points 𝑛 and 2𝑗 . One advantage of
process is shown in Fig. 4 this method is that 𝐷𝑊 𝑇 can be implemented recursively with low

3
T. Hernández-Del-Toro et al. Biomedical Signal Processing and Control 65 (2021) 102351

pass and high pass filters until a desired level 𝑗 of decomposition is 3.5.5. Higuchi fractal dimension
reached. With the DWT applied over a signal using a mother wavelet The Higuchi Fractal Dimension (HFD) [25,27] estimates the fractal
𝛹 and a number of decomposition levels 𝑗 we can get a set of detail dimension of a signal in the time domain. For a signal 𝑋(𝑡), the HFD is
coefficients 𝑤𝑗 (𝑟) and an approximation coefficient 𝑎(𝑟). In order to obtained as follows: First, from the original time series 𝑋(𝑡), calculate
simplify the notation, the detail coefficients and the approximation a new set of time series 𝑋𝑘𝑚 as:
coefficients are denoted as 𝑤𝑗 (𝑟) of the 𝑗th decomposition level, and
𝑁 −𝑚
the approximation coefficient will correspond to the level 𝑗 + 1. In this 𝑋𝑘𝑚 = [𝑋(𝑚), 𝑋(𝑚 + 𝑘), … , 𝑋(𝑚 + [ ] ⋅ 𝑘)] (5)
𝑘
work, 4 decomposition levels were calculated, and the approximation
level was also used for feature extraction. With this, we obtain 5 sets for 𝑚 = 1, 2, … , 𝑘 and 𝑘 = 1, 𝑘𝑚𝑎𝑥 . 𝑘𝑚𝑎𝑥 is usually defined between 6
of coefficients per channel signal. The wavelet mother used was bior2.2 and 16, in this work is fixed as 10. With the obtained new series, we
because it has shown desirable results in previous work that deal with calculate the length of the curve 𝐿𝑚 (𝑘) of 𝑋𝑘𝑚 as:
imagined speech [16,21].
⎛[ 𝑁−𝑚 ] ⎞( )
1⎜ ∑ ⎟ 𝑁 −1
𝑘
|𝐴(𝑚, (6)
𝑘 ⎜⎜ 𝑖=1 ⎟
3.5.2. Empirical mode decomposition 𝐿𝑚 (𝑘) = 𝑖, 𝑘)| 𝑁−𝑚
The Empirical Mode Decomposition (EMD) is a process where a ⎟ [ 𝑘 ]𝑘
⎝ ⎠
signal is decomposed into a finite set of components called Intrinsic
where 𝐴(𝑚, 𝑖, 𝑘) = 𝑋(𝑚 + 𝑖𝑘) − 𝑋(𝑚 + (𝑖 − 1)𝑘), and 𝑁 is the length of the
Mode Functions (IMF) [25]. The IMFs of a signal 𝑋(𝑡) are obtained
original time series. The 𝐿𝑚 (𝑘) is averaged over 𝑚 for each 𝑘, obtaining
through the following process:
with this
1. Calculate lower envelope 𝑙(𝑡) and upper envelope 𝑢(𝑡) of 𝑋(𝑡).
1 ∑
𝑘
2. Calculate the mean envelope 𝑚(𝑡) = 𝑢(𝑡)−𝑙(𝑡) 𝐿(𝑘) = 𝐿 (𝑘) (7)
2 𝑘 𝑚=1 𝑚
3. Extract the mean from the signal to obtain ℎ(𝑡) = 𝑋(𝑡) − 𝑚(𝑡)
4. Verify if ℎ(𝑡) fulfills the two conditions of an IMF: Finally, the HFD is estimated as the slope of the method of least-
squares from the plot of ln(𝐿(𝑘)) against ln(1∕𝑘).
• The number of extrema and of zero crossings must be ln(𝐿(𝑘))
either equal or differ at most by one. 𝐻𝐹 𝐷 = (8)
ln(1∕𝑘)
• At any point, the mean value of the envelope defined by
the local maxima and the envelope of the local minima is 3.5.6. Katz fractal dimension
zero. The Katz Fractal Dimension (KFD) [28] gives an estimation of the
If these two conditions are fulfilled, it means that ℎ(𝑡) is an fractal dimension by analyzing the waveform of the time series. For a
IMF. Then ℎ(𝑡) is extracted from 𝑋(𝑡) and the process is repeated signal 𝑋(𝑡), the KFD is defined as:
for the next IMF with the resulting signal 𝑋 ′ (𝑡) = 𝑋(𝑡) − ℎ(𝑡). log(𝑚)
𝐾𝐹 𝐷 = (9)
Otherwise, If ℎ(𝑡) does not fulfills the two conditions of an IMF, log(𝑚) + log( 𝐿𝑑 )
then the process is repeated with ℎ(𝑡) until it is an IMF.
with 𝑚 the number of samples in the signal, 𝐿 the total length of
Depending on the information and noise in the signal, some signals the signal, this is:
will have more IMFs than others, in order to have a constant set of 𝑚 √
∑
IMFs from all signals, we will select the two signals with the smaller 𝐿= 1 + (𝑋(𝑡 − 1) − 𝑋(𝑡))2 (10)
Minkowski distance because they are the IMFs that contain more 𝑡=2
information about the signal [26]. For an IMF ℎ(𝑡) obtained from a And 𝑑 the planar distance of the waveform, defined as the distance
signal 𝑋(𝑡), the Minkowski distance is defined as:
from the first point to the farthest point in the signal
(𝑚 )1∕2 √
∑
𝐷𝑀 = |𝑋(𝑡) − ℎ(𝑡)|2
(2) 𝑑 = max{ (𝑡 − 1)2 + (𝑋(𝑡) − 𝑋(1))2 } (11)
𝑡
𝑡=1

To maintain consistency with the detail coefficients obtained 3.5.7. Generalized hurst exponent (GHE)
through DWT in the feature extraction process, the two selected IMFs The Generalized Hurst Exponent (GHE) [29–31], denoted as 𝐻(𝑞)
will be named as 𝑤𝑗 (𝑟) ∶ {ℎ1 (𝑟), ℎ2 (𝑟)}. With the signal decomposed in is used in time series analysis and fractal analysis as a measure of the
sets 𝑤𝑗 (𝑟) either with EMD or DWT, features can now be calculated. scaling properties by analyzing the 𝑞𝑡ℎ-order moments of the distribu-
tion of the increments. For a time series 𝑋(𝑡), the GHE 𝐻(𝑞) can be
3.5.3. Instantaneous energy obtained from the relation:
The Instantaneous Energy (IE) [24] extracted from the detail coef- ( )𝑞𝐻(𝑞)
𝜏
ficients obtained through DWT or from the IMFs obtained from EMD 𝐾𝑞 (𝜏) ∼ , (12)
𝜈
reflects the energy distribution of each band. For a set 𝑤𝑗 (𝑟) obtained
with
through a decomposition process of a signal, the IE is defined as:
( ) ⟨|𝑋(𝑡 + 𝜏) − 𝑋(𝑡)|𝑞 ⟩
𝐾𝑞 (𝜏) = , (13)
1 ∑ 2
𝑚
𝐼𝐸𝑗 = log10 𝑤 (𝑟) (3) ⟨|𝑋(𝑡)|𝑞 ⟩
𝑚 𝑟=1 𝑗
given from 𝑋(𝑡), with 𝑡 = 𝜈, 2𝜈, … , 𝑘𝜈, 𝑇 . (Observation period 𝑇 and
where 𝑚 are the number of samples in the set 𝑤𝑗 (𝑟). time resolution 𝜈). For 𝑞 = 1, the GHE is closely related to the original
Hurst exponent, which measures how chaotic or unpredictable a time
3.5.4. Teager energy series is.
The Teager Energy (TE) [24] reflects the variations in both ampli-
tude and frequency of the signal. For a set 𝑤𝑗 (𝑟) obtained through a 3.6. Feature sets
decomposition process of a signal, the TE is defined as:
[ 𝑚−1 ] Once we have a signal instance extracted from the signal, the
1 ∑| 2 |
𝑇 𝐸𝑗 = log10 |𝑤𝑗 (𝑟) − 𝑤𝑗 (𝑟 − 1) ⋅ 𝑤𝑗 (𝑟 + 1)| (4) features will be extracted from each channel, and those features will
𝑚 𝑟=1 | |
be concatenated in an instance vector. Features are calculated per each
where 𝑚 are the number of samples in the set 𝑤𝑗 (𝑟). one of the following 5 feature sets:

4
T. Hernández-Del-Toro et al. Biomedical Signal Processing and Control 65 (2021) 102351

Fig. 6. Procedure for extracting a feature vector from a signal instance. First each
signal for each channel is decomposed and from the decomposition, the features are
Fig. 7. Procedure for converting the overlapped segments into segments of smaller
calculated. The final feature vector is made by concatenation of the features obtained
window size, and procedure of error correction using first neighbors in the predicted
for each channel.
vector to correct false positives and false negatives. Each square indicates a predicted
segment. The white squares indicate that the segment belongs to an ISS, the dark
squares indicate that the segment belongs to and IWS.
1. IE of the detail and approximation coefficients obtained with the
DWT of each channel.
2. TE, IE, HFD, KFD, and GHE with 𝑞 = 1, 2. All of them extracted Although some hyper-parameters were tuned, not many combinations
from the IMFs obtained through EMD of each channel. were tried, since the primordial objective of this work was to test and
3. GHE for 𝑞 = 1, 2 with the only noise cleaned signal for each to compare different feature extraction methods and not the classifiers.
channel.
3.8. Window reduction and error correction
The protocol of recording the signals is different in each dataset,
this can lead to a scenario where each of the feature set mentioned
After we have our trial classified, we obtain a vector of zeros and
above performs differently for each subject in each dataset, in order to
ones. We must remember that this vector contains information about
exploit the information provided by each feature, two more feature sets
overlapped windows. Before getting our final prediction we must turn
are defined
our vector of 0.5 s sized windows and with an overlap of 0.1 s into
4. All the first three feature sets, where each instance is concate- a vector of 0.1 s sized windows with no overlap. We obtain this by
nated as (1, 2, 3). making a majority vote with every 5 samples. After we do this, we
5. Principal Component Analysis (PCA) applied to the feature set also apply an error correction of first neighbors in order to delete false
no. 4 preserving 90% of the information. positives or false negatives. This whole procedure is illustrated in Fig. 7

The PCA is introduced to be able to exploit all the information in 3.9. Evaluation metrics
features but to reduce the training time of models. The feature sets
based on chaos theory and fractal dimensions are selected because We can compare the corrected predicted trial with the actual trial
they have been used in studying EEG signals due to its natural non- using the 𝐹1 𝑠𝑐𝑜𝑟𝑒 [36]. The 𝐹1 𝑠𝑐𝑜𝑟𝑒 gives a trade-off of the 𝑃 𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛
linear a chaotic behavior [11,32–34] and in studying epilepsy in EEG and 𝑅𝑒𝑐𝑎𝑙𝑙 by penalizing if either one or another is low. It is defined
signals [25,35]. The whole procedure to extract an instance from a as the harmonic mean of 𝑃 𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 and 𝑅𝑒𝑐𝑎𝑙𝑙:
segment of a signal is explained in Fig. 6. 𝑃 𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 ⋅ 𝑅𝑒𝑐𝑎𝑙𝑙
After this procedure, each signal instance of the trial will be con- 𝐹1 𝑠𝑐𝑜𝑟𝑒 = 2 ⋅ (14)
𝑃 𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑅𝑒𝑐𝑎𝑙𝑙
verted into a feature vector of size 1 × 𝑚. The whole trial to predict
With the 𝐹1 𝑠𝑐𝑜𝑟𝑒, we can compare the corrected predicted trial with
will be converted into a matrix of size 𝑛 × 𝑚 where 𝑛 is the number
the actual trial as vectors containing clusters of 0′ 𝑠 and 1′ 𝑠 denoting
of segments extracted from the trial and 𝑚 is the number of features
ISS and IWS, respectively. Ideally we would like to have our predicted
extracted.
vectors as a cluster of 0′ 𝑠 followed by a cluster of 1′ 𝑠 followed by
another cluster of 0′ 𝑠 since that is the construction of every trial.
3.7. Classifiers
However, this will not always happen and sometimes, the cluster of
1′ 𝑠 will be moved to the left or right, or will be shrunk or enlarged. In
All features are normalized using the standard score 𝑧. The classi-
other worse cases, we will have two clusters of 1′ 𝑠 indicating that there
fication part was programmed in Python using sklearn libraries. Four
are two IWS in the trial, which is wrong. The 𝐹1 𝑠𝑐𝑜𝑟𝑒 can take these
classifiers are used in this work:
details into account and give us a good metric to evaluate our results.
• Random Forest: A classifier based on decision trees. For this With an average 𝐹1 𝑠𝑐𝑜𝑟𝑒 of 0.75, the system is capable to detect this
classifier, the number of trees was 100 because was the one that segments of imagined words for latter classification, we would expect
performed best from the possibilities {5, 10, 50, to have this results in all three dataset.
√ 100, 500}. The
number of features per tree in this model was 𝑚 where 𝑚 is the
number of total features. 4. Results
• k Nearest Neighbors: A classifier based on the distance of the
instances in the feature vector space. For this classifier, the num- For each subject, and individual classifier was trained and the
ber of neighbors was 50 because was the one that performed predicted trials were evaluated using as metrics the 𝑃 𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛, 𝑅𝑒𝑐𝑎𝑙𝑙
best from the possibilities {5, 10, 50, 100, 500}. The Minkowski (𝑆𝑒𝑛𝑠𝑖𝑡𝑖𝑣𝑖𝑡𝑦), and 𝐹1 𝑠𝑐𝑜𝑟𝑒 with the different feature sets and classifiers.
distance is chosen to measure the distances between instances.
• Support Vector Machine: A classifier where a decision boundary 4.1. Feature sets 1, 2 and 3
is calculated. For this classifier, the default options given by
sklearn were selected, using the rbf as kernel. The regularization For the three datasets, the results for the first three feature sets are
parameter is set to 𝐶 = 1. shown in Fig. 8. We can see that for each dataset, different feature sets
• Logistic Regression: A classifier based on probability. For this perform differently. For the first dataset, the first feature set performs
classifier, the default options given by sklearn were selected, better in the four classifiers, although many outliers are observed on
using the l2 as penalty. this dataset. The second dataset presents higher 𝐹1 𝑠𝑐𝑜𝑟𝑒𝑠 when using

5
T. Hernández-Del-Toro et al. Biomedical Signal Processing and Control 65 (2021) 102351

Fig. 8. Results of 𝐹1 𝑠𝑐𝑜𝑟𝑒 obtained for the three datasets using the first three feature sets described in Section 3.6 and the four classifiers described in Section 3.7 using a box
plot. The white squares inside the boxes indicate the mean obtained. The black diamonds represent subjects which result is far away from the main quarters, these elements are
considered as outliers.

Table 2 Table 3
Mean and standard deviation of the best results obtained for each dataset. The value Mean and standard deviation of the best results obtained for each dataset. The value
in parentheses represents the classifier and the feature set that obtained that value. in parentheses represents the classifier and the feature set that obtained that value.
Dataset 𝐹1 𝑠𝑐𝑜𝑟𝑒 𝑃 𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 𝑅𝑒𝑐𝑎𝑙𝑙 Dataset 𝐹1 𝑠𝑐𝑜𝑟𝑒 𝑃 𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 𝑅𝑒𝑐𝑎𝑙𝑙
1 (RF, 1) 0.72 ± 0.07 (RF, 1) 0.65 ± 0.07 (RF, 2) 0.85 ± 0.10 1 (RF, 5) 0.73 ± 0.07 (RF, 4) 0.66 ± 0.08 (RF, 5) 0.87 ± 0.9
2 (RF, 2) 0.78 ± 0.04 (RF, 2) 0.69 ± 0.03 (RF, 2) 0.93 ± 0.06 2 (RF, 5) 0.79 ± 0.04 (RF, 4) 0.70 ± 0.04 (RF, 5) 0.93 ± 0.06
3 (LR, 1) 0.68 ± 0.03 (SVM, 1) 0.62 ± 0.05 (LR, 1) 0.85 ± 0.04 3 (LR, 4) 0.68 ± 0.04 (SVM, 4) 0.65 ± 0.06 (LR, 4) 0.83 ± 0.04

the random forest classifier compared with the first and third dataset. to 0.73. For the second dataset. Similarly to the first dataset, first we
And in this second dataset, the second feature set performs better in obtain a 𝐹1 𝑠𝑐𝑜𝑟𝑒 of 0.78 using the second feature set and the random
all classifiers. However, this dataset also presents outliers. In the third forest classifier. Then this 𝐹1 𝑠𝑐𝑜𝑟𝑒 is increased to 0.79 using the feature
dataset, the 𝐹1 𝑠𝑐𝑜𝑟𝑒𝑠 obtained are lower compared with dataset 1 and 2. set 5. In third dataset, the 𝐹1 𝑠𝑐𝑜𝑟𝑒 obtained using the last two feature
Although there are not many outliers as in datasets 1 and 2. Still, there sets does not change in comparison with using the first three datasets,
are feature sets that perform better with some subjects than others. In both feature sets 1 and 4 achieve an 𝐹1 𝑠𝑐𝑜𝑟𝑒 of 0.68.
Table 2 are summarized the best average and standard deviation results We can see that each dataset performed differently. This behavior
obtained and the classifier that obtained those values. can be explained because each feature set performs differently with
each subject. However, the use of the feature set that contains all
4.2. Feature sets 4 and 5 the features keeps the scores or increases it. Although all the features
analyze the frequency or fractal dimensions of EEG, the exact feature
We must recall that each point in the box plot represents a subject. per subject can be different. This suggests the use of a model with
Thus, each box represents the data for all the subjects in the dataset. different features per subject in order to exploit the best information
We can see that for every subject and every dataset, each feature set given by the EEG signals in the task of detecting segments of imagined
performs different, this is because each dataset has different recording words. This feature selection can be seen as a pre-processing part of the
protocols. To solve this problem and to be able to exploit the infor- workflow in the system that automatizes the segmentation of imagined
mation present in all feature sets, the feature sets 4 and 5 described in words segments. In Fig. 10 is shown a suggestion of this workflow.
Section 3.6 are used. The feature set 4 is used to exploit all the features The difference in metrics obtained in each dataset can also be
and the feature 5 is used to test if it is possible to get similar results explained by analyzing the protocol of recording the EEG signals. In
with less features, thus, speeding up the training times. In Fig. 9, the the first dataset, the periods in between imagined words were not
results for the three datasets using the two last feature sets and the four constant and the onset and end of the imagined words were set by the
classifiers are presented. We can observe that for dataset 1 not only user. In the second dataset, the timing of idle states was controlled by
fewer outliers appear but also the performance has increased. We can the program and the size of ISS were smaller compared to the IWS,
also notice that when using the feature set which has all the features this could explain why the second dataset is the one that achieves
plus a feature selection based on PCA, the performance is better than higher scores. In the third dataset the timing was also controlled by
using all the features. For the second dataset, we can observe that the recording program. However, maybe the picture shown is making
contrary to the first dataset, more outliers have appeared in comparison the subjects to imagine the word before it is needed, and this could
to the first three feature sets. However, the 𝐹1 𝑠𝑐𝑜𝑟𝑒 has increased with produce a higher error. This can be corrected by recording a dataset
the use of all the features, though the change is not as great as in the that has different idle states before the imagined word.
other datasets. The classifier that achieves the higher 𝐹1 𝑠𝑐𝑜𝑟𝑒 is the
random forest. For the third dataset, no outliers appear, and we can see 5. Conclusion
that using all the features raises the performance. Although the use of
the PCA for feature selection reduces the features, this does not increase In this work five feature sets based on DWT, EMD, Energies features,
the performance. In Table 3 is shown the best scores obtained for Fractal dimensions features, and chaos measure features were tested
𝐹1 𝑠𝑐𝑜𝑟𝑒, 𝑃 𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛, and 𝑅𝑒𝑐𝑎𝑙𝑙 with each classifier in all three datasets with three datasets that contain imagined words in continuous signals.
for this last two feature sets. The features were extracted to train four classifiers, and with them to
try to identify IWS in trials that contain an ISS followed by an IWS
4.3. Discussion followed by another ISS. For each subject in each dataset, different
feature sets perform differently, this suggests the use of a model that
In the first dataset, using the first three feature sets, the higher selects the features that contain the most relevant information per
average 𝐹1 𝑠𝑐𝑜𝑟𝑒 obtained was 0.72 with the feature set 1 and the subject. Each dataset has a different recording protocol. The dataset
random forest classifier. Using feature set 5, this 𝐹1 𝑠𝑐𝑜𝑟𝑒 is increased that achieved the higher 𝐹1 𝑠𝑐𝑜𝑟𝑒 was the one that has the imagined

6
T. Hernández-Del-Toro et al. Biomedical Signal Processing and Control 65 (2021) 102351

Fig. 9. Results of 𝐹1 𝑠𝑐𝑜𝑟𝑒 obtained for the three datasets using the last two feature sets described in Section 3.6 and the four classifiers described in Section 3.7 using a box
plot. The white squares inside the boxes indicate the mean obtained. The black diamonds represent subjects which result is far away from the main quarters, these elements are
considered as outliers.

CRediT authorship contribution statement

Tonatiuh Hernández-Del-Toro: Conceptualization, Methodology,

Software, Formal analysis, Investigation, Writing - original draft, Writ-
ing - review & editing, Visualization. Carlos A. Reyes-García: Concep-
tualization, Methodology, Resources, Writing - review & editing, Su-
pervision, Funding acquisition. Luis Villaseñor-Pineda: Methodology,
Resources, Writing - review & editing, Supervision.
Fig. 10. Proposed workflow to detect IWS exploiting the features that achieve best
scores per subject. Declaration of competing interest

The authors declare that they have no known competing finan-

words more controlled. This suggests the recording of another dataset cial interests or personal relationships that could have appeared to
which contains idle states more controlled in the recording protocol. influence the work reported in this paper.
The results shown in this work can be used to build a system that
automatizes the segmentation of IWS followed by the classification of Acknowledgment
the word itself and with this to implement a real time interactive BCI
based on imagined speech that is fully asynchronous. An example of The present work was partially supported by scholarship no. 740971
this is a BCI where the user controls a PC pointer by imagining direction by CONACyT, Mexico.
words. The work presented takes part of a bigger project that aims to
build a BCI based on imagined speech on EEG. The results shown in this References
paper explore the preliminary work into detecting IWS in continuous
[1] A. Akmal Nooh, J. Yunus, S. Mohd Daud, A review of asynchronous
EEG signals. electroencephalogram-based brain computer interface systems, in: International
Conference on Biomedical Engineering and Technology, Vol. 11, 2011, pp.
6. Future work 55–59, URL http://www.ipcbee.com/vol11/11-T025.pdf.
[2] N. Williams, G. McArthur, N. Badcock, 10 years of EPOC: A scoping review
of Emotiv’s portable EEG device, 2020, http://dx.doi.org/10.1101/2020.07.14.
As future work, we still need to implement another metrics to 202085, bioRxiv.
evaluate from the predicted trials, the onset and ending of the word in [3] R. Krepki, B. Blankertz, G. Curio, K.R. Müller, The Berlin Brain-Computer
order to implement an asynchronous BCI that triggers and action when Interface (BBCI) - Towards a new communication channel for online control
in gaming applications, Multimedia Tools Appl. 33 (1) (2007) 73–90, http:
it detects the onset of an imagined word. Although feature sets 4 and
//dx.doi.org/10.1007/s11042-006-0094-3.
5 keep the scores obtained in feature sets 1, 2 and 3, the variation is [4] K. Brigham, B.V.K.V. Kumar, Imagined speech classification with EEG signals for
small, this suggests that there could be other features that could exploit silent communication: A preliminary investigation into synthetic telepathy, in:
more the information contained in EEG signals. The average time of an 2010 4th International Conference on Bioinformatics and Biomedical Engineer-
EEG recording session, since the setup to the end of the recording lasted ing, ICBBE 2010, IEEE, 2010, pp. 1–4, http://dx.doi.org/10.1109/ICBBE.2010.
5515807.
about half an hour, this duration was exhausting for the subjects as at [5] A.A. Torres-García, Análisis y clasificación de electroencefalogramas (EEG)
the end many of them were tired. However, training the models with registrados durante el habla imaginada (Ph.D. thesis), INAOE, 2016, URL http:
just a few samples did not perform as well as using the 75% of trials for //inaoe.repositorioinstitucional.mx/jspui/handle/1009/53.
training. An approach that could solve this problem is the incremental [6] A.A. Torres-García, C.A. Reyes-García, L. Villaseñor-Pineda, G. García-Aguilar,
Implementing a fuzzy inference system in a multi-objective EEG channel selection
learning approach in which the model is learning on the run from the
model for imagined speech classification, Expert Syst. Appl. 59 (2016) 1–12,
signals given by the user while using the BCI, this approach would let http://dx.doi.org/10.1016/j.eswa.2016.04.011.
the users to record only a few signals and start using the BCI while the [7] Y. Song, F. Sepulveda, Classifying speech related vs. idle state towards onset
BCI keeps learning from new samples. Despite the information studied detection in brain-computer interfaces, in: IEEE 2014 Biomedical Circuits and
Systems Conference, BioCAS 2014 - Proceedings, 2014, pp. 568–571, http:
in imagined speech appears in all the EEG channels, there could be
//dx.doi.org/10.1109/BioCAS.2014.6981789.
some combination of channels that could keep the good scores. This [8] Y.J. Song, F. Sepulveda, Classifying siren-sound mental rehearsal and covert
suggests the study of channel selection in order to reduce the dimension production vs. idle state towards onset detection in brain-computer interfaces,
of features. Another approach that can be done is the use of deep in: 3rd International Winter Conference on Brain-Computer Interface, BCI 2015,
learning in the task of identifying from a continuous signal, imagined 2015, pp. 1–4, http://dx.doi.org/10.1109/IWW-BCI.2015.7073020.
[9] Y. Song, F. Sepulveda, A novel onset detection technique for brain-computer
words segments. And the final goal is to perform the classification of interfaces using sound-production related cognitive tasks in simulated-online
imagined words in real time in order to integrate this identification system, J. Neural Eng. 14 (1) (2017) http://dx.doi.org/10.1088/1741-2552/14/
module into a practical working BCI. 1/016019.

7
T. Hernández-Del-Toro et al. Biomedical Signal Processing and Control 65 (2021) 102351

[10] Y. Song, F. Sepulveda, An online self-paced brain-computer interface onset [23] L.A. Moctezuma, M. Molinas, Subject identification from low-density eeg-
detection based on sound-production imagery applied to real-life scenarios, in: recordings of resting-states: EEG study of feature extraction and classification,
5th International Winter Conference on Brain-Computer Interface, BCI 2017, Expert Syst. Appl. 118 (15) (2019) 201–208, http://dx.doi.org/10.1007/978-3-
2017, pp. 46–49, http://dx.doi.org/10.1109/IWW-BCI.2017.7858155. 030-12385-7_57.
[11] T. Hernández-Del-Toro, C.A. Reyes-García, An algorithm for detecting the onset [24] E. Didiot, I. Illina, D. Fohr, O. Mella, A wavelet-based parameterization for
of linguistic segments in continuous electroencephalogram signals, in: Models speech/music discrimination, Comput. Speech Lang. 24 (2) (2010) 341–357,
and Analysis of Vocal Emissions for Biomedical Applications, MAVEBA, Firenze http://dx.doi.org/10.1016/j.csl.2009.05.003.
Univeristy Press, Florence, Italy, 2019, pp. 249–252, http://dx.doi.org/10.36253/ [25] L. Moctezuma, M. Molinas, Classification of low-density EEG-based epileptic
978-88-6453-961-4. seizures by energy and fractal features based on EMD, J. Biomed. Res. 34 (3)
[12] G. Townsend, B. Graimann, G. Pfurtscheller, Continuous EEG classification during (2020) 180–190, http://dx.doi.org/10.7555/JBR.33.20190009.
motor imagery - Simulation of an asynchronous BCI, IEEE Trans. Neural Syst. [26] D. Boutana, M. Benidir, B. Barkat, On the selection of intrinsic mode function
Rehabil. Eng. 12 (2) (2004) 258–265, http://dx.doi.org/10.1109/TNSRE.2004. in EMD method: Application on heart sound signal, in: 2010 3rd International
827220. Symposium on Applied Sciences in Biomedical and Communication Technologies,
[13] J. Pereira, A.I. Sburlea, G.R. Müller-Putz, EEG patterns of self-paced movement ISABEL 2010, 2010, pp. 1–5, http://dx.doi.org/10.1109/ISABEL.2010.5702895.
imaginations towards externally-cued and internally-selected targets, Sci. Rep. 8 [27] T. Higuchi, Approach to an irregular time series on the basis of the fractal
(1) (2018) 1–15, http://dx.doi.org/10.1038/s41598-018-31673-2. theory, Physica D 31 (2) (1988) 277–283, http://dx.doi.org/10.1016/0167-
[14] A. Shakeel, M.S. Navid, M.N. Anwar, S. Mazhar, M. Jochumsen, I.K. Niazi, A 2789(88)90081-4.
review of techniques for detection of movement intention using movement- [28] M.J. Katz, Fractals and the analysis of waveforms, Comput. Biol. Med. 18 (3)
related cortical potentials, Comput. Math. Methods Med. 2015 (2015) http: (1988) 145–156, http://dx.doi.org/10.1016/0010-4825(88)90041-8.
//dx.doi.org/10.1155/2015/346217. [29] T. Di Matteo, T. Aste, M.M. Dacorogna, Scaling behaviors in differently developed
[15] J.S. García-Salinas, L. Villaseñor-Pineda, C.A. Reyes-García, A.A. Torres-García, markets, Physica A 324 (1–2) (2003) 183–188, http://dx.doi.org/10.1016/
Transfer learning in imagined speech EEG-based BCIs, Biomed. Signal Process. S0378-4371(02)01996-9.
Control 50 (2019) 151–157, http://dx.doi.org/10.1016/j.bspc.2019.01.006. [30] T. Di Matteo, T. Aste, M.M. Dacorogna, Long-term memories of developed
[16] L. Moctezuma, Distinción de estados de actividad e inactividad lingüística para and emerging markets: Using the scaling analysis to characterize their stage
interfaces cerebro computadora (Ph.D. thesis), BUAP, Benemérita Universidad of development, J. Bank. Financ. 29 (4) (2005) 827–851, http://dx.doi.org/10.
Autónoma de Puebla, 2017, http://dx.doi.org/10.13140/RG.2.2.10145.10086, 1016/j.jbankfin.2004.08.004.
August. [31] T. Di Matteo, Multi-scaling in finance, Quant. Finance 7 (1) (2007) 21–36,
[17] M.M. Alsaleh, M. Arvaneh, H. Christensen, R.K. Moore, Brain-computer interface http://dx.doi.org/10.1080/14697680600969727.
technology for speech recognition: A review, in: 2016 Asia-Pacific Signal and [32] C.F. Vega, J. Noel, Parameters analyzed of Higuchi’s fractal dimension for EEG
Information Processing Association Annual Summit and Conference, APSIPA brain signals, in: 2015 Signal Processing Symposium, SPSympo 2015, 2015, pp.
2016, Asia Pacific Signal and Information Processing Association, 2017, pp. 1–5, 1–5, http://dx.doi.org/10.1109/SPS.2015.7168285.
http://dx.doi.org/10.1109/APSIPA.2016.7820826. [33] F. Finotello, F. Scarpa, M. Zanon, EEG signal features extraction based on fractal
[18] EmotivLabs, Emotiv, 2020, URL https://www.emotiv.com/. dimension, in: Proceedings of the Annual International Conference of the IEEE
[19] BrainVision, BrainVision, 2020, URL https://brainvision.com/. Engineering in Medicine and Biology Society, Vol. 2015-Novem, EMBS, 2015,
[20] D.J. McFarland, L.M. McCane, S.V. David, J.R. Wolpaw, Spatial filter selection pp. 4154–4157, http://dx.doi.org/10.1109/EMBC.2015.7319309.
for EEG-based communication, Electroencephalogr. Clin. Neurophysiol. 103 (3) [34] S. Banerjee, R. Lamberto, Applications of Chaos and Nonlinear Dynamics in
(1997) 386–394, http://dx.doi.org/10.1016/S0013-4694(97)00022-2. Science and Engineering, Vol. 3, Springer, 2015, p. 304, http://dx.doi.org/10.
[21] L.A. Moctezuma, A.A. Torres-García, L. Villaseñor-Pineda, M. Carrillo, Subjects 1007/978-3-642-34017-8.
identification using EEG-recorded imagined speech, Expert Syst. Appl. 118 (2019) [35] C.L. Martínez-González, A. Balankin, T. López, J. Manjarrez-Marmolejo, E.J.
201–208, http://dx.doi.org/10.1016/j.eswa.2018.10.004. Martínez-Ortiz, Evaluation of dynamic scaling of growing interfaces in EEG
[22] L.A. Moctezuma, M. Molinas, EEG-Based Subjects Identification Based on Bio- fluctuations of seizures in animal model of temporal lobe epilepsy, Comput. Biol.
metrics of Imagined Speech using EMD, in: Lecture Notes in Computer Science Med. 88 (July) (2017) 41–49, http://dx.doi.org/10.1016/j.compbiomed.2017.07.
(including subseries Lecture Notes in Artificial Intelligence and Lecture Notes 003.
in Bioinformatics), vol. 11309 LNAI, no. December, 2018, pp. 458–467, http: [36] A. Tharwat, Classification assessment methods, Appl. Comput. Inform. (2018)
//dx.doi.org/10.1007/978-3-030-05587-5_43, arXiv:1809.06697. http://dx.doi.org/10.1016/j.aci.2018.08.003.