11th International Conference on Intelligent Control and Information Processing

Dali, China, December 3-7, 2021

Emotion Classification Using EEG Data in a

Brain-Inspired Spiking Neural Network
2021 11th International Conference on Intelligent Control and Information Processing (ICICIP) | 978-1-6654-2515-5/21/$31.00 ©2021 IEEE | DOI: 10.1109/ICICIP53388.2021.9642186

Yulin He, Chuandong Li, and Xingxing Ju

College of Electronic and Information Engineering
Southwest University
Chongqing, China
[email protected], [email protected], [email protected]

Abstract—As an important application of emotion artificial In addition, Zheng et al. [3] showed that there is a certain
intelligence, emotion classification provides the basis for the connection between emotional state, brain neural activity area
realization of affective brain-computer interface (aBCI). In this and Electroencephalogram (EEG) signal frequency band, and
study, the NeuCube is used to learn and classify Electroen-
cephalogram (EEG) data from the DEAP dataset. NeuCube is the same individual’s emotional model based on EEG is stable.
a type of spiking neural network (SNN) framework developed The training of model requires a large time cost, because of
based on the real human brain. It is very suitable for analyzing the high feature dimension of the EEG signal. In order to
and processing spatio-temporal data. Based on the 10-fold cross- reduce the feature dimension effectively, Hu et al. [4] used a
validation method, we obtain a mean accuracy of 68.91% in the correlation-based feature selection method in the EEG-based
emotional binary valence classification problem. Meanwhile, the
EEG data recorded from F3 and F4 electrode channels provide attention recognition task. Zheng et al. [5] proposed to use
more information compared with Fp1 and Fp2. The results prove Group Sparse Canonical Correlation Analysis (GSCCA) for
that the spiking neural network can be applied to the task of EEG channel selection in the task of EEG-based emotion
emotion classification effectively. recognition. Ozerdem et al. [6] used artificial neural networks
Index Terms—emotion classification, EEG, spiking neural net- to conduct research on EEG channel selection in emotion
works, NeuCube
recognition tasks. Clarence et al. [7] used spiking neural
networks in emotion classification tasks based on feature
I. I NTRODUCTION
fusion of multimodal data and obtained the accuracy achieved
Affective Computing is an interdisciplinary research field by traditional machine learning methods. A greater number of
involving multiple disciplines such as computer science, psy- electrodes cannot guarantee a higher accuracy [8].
chology, and cognitive science. It aims to research and develop In this study, through analysis and experiments, we demon-
theories and systems that can recognize, interpret, process, strate a method of using a brain-inspired spiking neural
and simulate human emotions. Emotional changes are usually network to perform emotion classification tasks based on EEG
produced under the stimulus of the external environment, and data. Specifically, we first encode the preprocessed EEG data
will be accompanied by changes in individual representations (from the DEAP dataset) into a spike train by using the
and psychological responses. Therefore, they can be measured Step Forward (SF) encoding algorithm, and then the hidden
and simulated by scientific methods. spatio-temporal relationships in the spike train are analysed
Human emotions are affected by factors such as time, by unsupervised and supervised learning in the NeuCube
geography, culture, and individual differences. Currently, re- framework. Finally, test samples undergo supervised learning
searchers mainly focus on two types of emotion models: to achieve binary valence classification. NeuCube is a spiking
discrete models and dimensional models. The discrete model neural network framework developed based on the real human
believes that emotions are composed of a variety of different brain. It can analyze and process spatio-temporal data well.
and discrete basic emotions. The rest of the emotions are
II. M ATERIAL AND M ETHODS
a combination of these basic emotions. The true emotions
of a person include these basic emotions and other complex A. EEG Data Acquisition
emotions. The more famous discrete model is the emotion In order to verify the feasibility of our study, we use the
theory proposed by Ekman [1]. EEG data from DEAP dataset [9] which can be obtained
The dimensional model defines emotions in a coordinate openly. The DEAP dataset collected the EEG signals (32
system composed of multiple dimensions. Among them, the channels) and other physiological signals (8 channels) data
two-dimensional model of emotions proposed by Russell [2] from 32 healthy participants (50% female). Each participant
in 1980 is widely used, which describes all human emotions in watched 40 segments of music videos, each with a duration
terms of valence and arousal by a point in the two-dimensional of 60 seconds. The diagram of DEAP dataset can be seen in
coordinate system. By using the dimensional method, the Fig. 1. The EEG data was first recorded at a sampling rate
emotional response triggered by stimulus can be evaluated and of 512 Hz under the International standard 10-20 electrode
quantified more specifically. placement system [10], and then down-sampled to 128 Hz. In

978-1-6654-2515-5/21/$31.00 ©2021 IEEE 433

Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY CALICUT. Downloaded on September 01,2023 at 09:57:42 UTC from IEEE Xplore. Restrictions apply.
 order to eliminate the noise caused by eye blinking, all the arbitrary continuous functions by using the spike firing time
original physiological signal data had passed a 4-45 Hz band- of the neuron as the input and output of the network directly.
pass filter. Each array consists of the following two parts: SNN is a common model for real-world data classification
• Data: The data recorded the physiological signals data and spatio-temporal pattern recognition. The spatio-temporal
of each participant during the experiment. Each segment data acquired in the real world exhibits spatial characteristics
of data was composed of the baseline of the first 3 at a certain fixed moment, and at the same time generally
seconds and the video time of the following 60 seconds. exhibits temporal characteristics, such as vision and voice
Therefore, each segment of data has a total of 8064 data signals. Therefore, the learning and recognition of spatio-
points. temporal data is a challenging task in artificial intelligence,
• Labels: The labels recorded the scores (from 1 to 9) rated and it is necessary to develop new computational models such
by participants according to valence, arousal, dominance, as spiking neural network.
and like/dislike corresponding to each video. C. The NeuCube Architecture
We know that the human brain is extremely efficient in
processing spatio-temporal information from the real world.
How to analyse the spatio-temporal information efficiently and
accurately has always been the direction of scientists’ efforts.
Traditional machine learning methods are difficult to learn
spatio-temporal patterns efficiently based on the structure of
the human brain. NeuCube is an extension of eSNN [11]. Its
purpose is to convert the spatio-temporal data into spike trains,
and then perform unsupervised and supervised learning to
realize spatio-temporal pattern recognition. NeuCube not only
has been successfully applied to pattern recognition problems
based on EEG and fMRI data, but also has further potential in
analyzing more variety of spatio-temporal information in the
future. A basic NeuCube framework includes the following
parts:
• Input data encoding module: This part encodes the input
data into a spike sequence, which will be learned in the
following part.
Fig. 1. Diagram of DEAP dataset. • 3D SNN reservoir (3D-SNNr): The encoded spike se-
quences are presented input into SNNr. All input spike
sequences are integrated and converted into class infor-
mation containing temporal and spatial attributes.
• Classification module: The input verification samples
are propagated through SNNr again by using the same
supervised learning algorithm as in the learning stage.
D. Encoding Method of Input Data
The filtered EEG data are a train of real-value data. In this
paper, the SF encoding algorithm [12] is used to convert EEG
data sequences into two types of spike trains: a positive spike
sequence (represented as +1) and a negative spike sequence
(represented as -1). More details of the SF algorithm can be
seen in Algorithm 1. Fig. 3 shows an example of encoding a
piece of data (128 data points) in the DEAP database.
Fig. 2. International standard 10-20 electrode placement system [10].
E. 3D-SNNr
Different from other architectures of spiking neural net-
B. Spiking Neural Networks
works, NeuCube introduces a real brain structure, which
As the third-generation artificial neural network model, the allows the spatio-temporal information in the brain to be ana-
Spiking Neural Network (SNN) is more biologically plausible. lyzed and modeled more effectively. 3D-SNNr is composed of
Compared with the traditional artificial neural network, the 1471 leaky integrate-and-fire (LIF) neuron models. According
spiking neural network can realize the efficient processing of to Talairach brain atlas [13], each neuron represents 1cm3
information and the simulation of various neural signals and of human brain neuron. At the same time, each neuron is

434
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY CALICUT. Downloaded on September 01,2023 at 09:57:42 UTC from IEEE Xplore. Restrictions apply.
 Algorithm 1 The Step Forward (SF) encoding algorithm
Input: signal S, baseline B, threshold Thr
Output: spike train Spike
1: L = length(S)
2: Initialize baseline: B(1) = S(1)
3: Initialize spike train: Spike ← 0
4: for t = 2 : L do
5: if S(t) > B(t − 1) + T hr then
6: positive spike : Spike(t) ← 1
7: update baseline: B(t) = B(t − 1) + T hr
8: else if S(t) <= B(t − 1) − T hr then
9: negative spike: Spike(t) ← −1
10: update baseline: B(t) = B(t − 1) − T hr
11: others then
12: no spikes Fig. 4. Talairach brain atlas [13].
13: update baseline: B(t) = B(t − 1)
14: end if will increase, on the contrary, if the pre-synaptic neuron i fires
15: end for
after the post-synaptic neuron j, then the ωij will decrease.
16: return Spike
The STDP learning rule in this study is given as [15],
LR
∆ω=sgn (∆t) (1)
∆t + 1
where ∆ω is the change in the connection weight between
post- and pre-synaptic, LR is the STDP Learning Rate (in
this paper, LR=0.01); sgn(·) is the function sign, sgn(·) = 1
for positive, while sgn(·) = -1 for negative values; ∆t is the
difference between post- synaptic and pre-synaptic times and
∆t= tpost –tpre .
G. Supervised Learning
This is the second stage of learning. After integrating the
training sample labels into SNNr, a classifier module is trained
through supervised learning. This work uses dynamic evolving
spiking neural network (deSNN) [16] as the supervised learn-
Fig. 3. An example of SF algorithm.
ing algorithm at this learning phase. The deSNN algorithm
combines rank-order (RO) [17] learning algorithm and STDP
learning rule. The RO algorithm assumes that the earlier spikes
mapped to SNNr, according to the International standard 10- are more important, the first spike carries the most information.
20 electrode placement system. Before learning, the weights After the neuron receives the spikes, it will adjust the weight
between neurons in SNNr will be initialized according to the according to the spikes arrival sequence and time difference.
small-world (SW) rule [11], where 80% of the weights are The rate at which the weights are adjusted depends on the
initialized as positive, and 20% of the weights are initialized setting of parameters values.
as negative. Then the SNNr will be trained in an unsupervised The method framework of this article is shown in Fig. 5.
learning mode.
III. E XPERIMENTS A ND R ESULTS
F. Unsupervised Learning A. EEG Data Preprocessing
This is the first stage of learning. The purpose is to adjust the Due to the complexity of the acquisition method, EEG
connection synaptic weight in the SNNcube so that the spatio- signals are prone to external interference, often with a lot of
temporal relationship hiding in the input spiking trains can be noise. Therefore, the preprocessing of the original EEG signal
learned in the entire model. As we know from the Hebbian is very important. Effective preprocessing can remove the
plasticity rule, if pre-synaptic neurons continue to stimulate noise in the original EEG signal, improve the signal quality,
post-synaptic neurons, the synaptic weight between them will and help obtain better classification results. However, more
increase. In this paper, we use the STDP learning rule [14] electrodes can not get higher accuracy, only 4 channels (Fp1,
developed from the Hebbian plasticity rule, that is: if the pre- Fp2, F3, F4) can also get high classification accuracy [18].
synaptic neuron i fires before the post-synaptic neuron j, then Therefore, we chose four electrode channels, which included
the connection synaptic weight ωij between the two neurons Fp1, Fp2, F3, F4. According to the dimensional model of

435
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY CALICUT. Downloaded on September 01,2023 at 09:57:42 UTC from IEEE Xplore. Restrictions apply.
 Fig. 5. Diagram of emotion binary valence classification using NeuCube.

emotion classification [2] (as shown in Fig. 6), we marked the grid search method. By setting the minimum, maximum,
a valence higher than 6 as positive, and a valence lower and step length, all target parameters are tested to find the
than 6 as negative, and removed the baseline in the first relatively best combination of parameters. After 320 iterations,
3 seconds. According to [19], the EEG data in the last 30 the relatively better classification accuracy is obtained, and the
seconds improves classification accuracy even more, so we final parameter combination is:
only select the EEG data in the last 30 seconds (3840 data • Small world radius (γ) : 2
points). Finally, we got 1270 samples as our experimental • STDP learning rate (LR) : 0.01
database. • Threshold of firing: 0.5
• Potential leak rate: 0.002
• Refractory time: 6
• Mod: 0.8
• Drift: 0.05
• K:3

D. Results
After optimization of result, the parameter combination is
fixed. There are 32 subjects in all, and each subject has 40
samples. Samples of one subject are presented to NeuCube
with fixed parameters. The data of samples will be learned
in NeuCube according to the aforementioned process. In the
Fig. 6. Dimensional model of emotion classification [4]. end, we get the accuracy of each subject in this binary valence
classification task (see the Fig. 7). The final result is the
average of the accuracy of 32 subjects, which is 68.91%, and
B. Classification Method the maximum classification accuracy is achieved as 82.50%.
The input verification samples are propagated through SNNr
again by using the same supervised learning algorithm deSNN
as in the learning stage. Then the verification samples are
classified by using the K-nearest neighbor algorithm. Specif-
ically, if most of the K (in this paper, K=3) nearest samples
of a sample in the weight space are of a certain class, then
this sample is also judged as that class. In order to get more
accurate results of classification, the model is evaluated by
using the 10-fold cross validation.
C. Optimisation of Results
It is crucial for an emotion classification task to get better
classification results by optimizing. In this study, we used Fig. 7. The classification accuracy of each subject.

436
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY CALICUT. Downloaded on September 01,2023 at 09:57:42 UTC from IEEE Xplore. Restrictions apply.
 It can be seen in Fig. 8 that in this task the input neurons F3 traditional machine learning methods. Despite many efforts,
and F4 have the largest clusters, therefore they are the most computational consumption is still a limitation of this paper.
important features. The Fp1 and Fp2 features are important as In addition, we can consider researching the influence of EEG
well, as their respective clusters are also relatively large. channels and frequency bands to improve accuracy in the
future work.
ACKNOWLEDGMENT
This work is supported by the National Key Research
and Development Project (Grant No. 2018AAA0100101), and
the Graduate Student Innovation Project of Chongqing under
Grant CYB21126.
R EFERENCES
[1] Ekman P. “An argument for basic emotions, ” Cognition and Emotion.
6(3-4): 169-200, 1992.
[2] J. A. Russell, “A circumplex model of affect,” J Personal. Soc. Psychol.,
vol. 39, pp. 1161–78, 1980.
[3] Zheng W L, Zhu J Y, Lu B L. ”Identifying stable patterns over time for
emotion recognition from EEG,” IEEE Trans. on Affective Computing,
2019, 10(3): 417–429.
Fig. 8. Neuron clustering for four electrode channels. [4] Hu B, Li X, Sun S, Ratcliffe M. Attention recognition in EEG-based
affective learning research using CFS+KNN algorithm. IEEE/ACM
Trans. on Computational Biology and Bioinformatics, 2018, 15(1):
To gauge the relative strength of the interaction among 38–45
neurons more effectively, we get the neuron proportion metric, [5] Zheng W. Multi channel EEG-based emotion recognition via group
which is the percentage of neurons in an input neuron cluster sparse canonical correlation analysis. IEEE Trans. on Cognitive and
Developmental Systems, 2017, 9(3): 281–290
out of the total number of neurons in the SNNr. As seen in [6] Ozerdem M S, Polat H. ”Emotion recognition based on EEG features in
Fig. 9, the neuron proportion of F3, F4 electrode channels is movie clips with channel selection,” Brain Informatics, 4(4): 241–252,
31% and 47% respectively compared to 15% and 7% in Fp1 2017.
[7] C. Tan, et al., “FusionSense: Emotion Classification Using Feature
and Fp2 respectively. Fusion of Multimodal Data and Deep Learning in a Brain-Inspired
Spiking Neural Network,” Sensors, vol. 20, no. 18, p. 5328, Sep. 2020.
[8] Chatchinarat, A. , K. W. Wong , and C. C. Fung . ”A comparison study
on the relationship between the selection of EEG electrode channels
and frequency bands used in classification for emotion recognition.”
International Conference on Machine Learning and Cybernetics IEEE,
2016.
[9] S. Koelstra et al., “Deap: A database for emotion analysis; using
physiological signals,” IEEE Trans. Affect. Comput., vol. 3, no. 1, pp.
18–31, 2012.
[10] G.H. Klem, H.O. Lüders, H. Jasper, C. Elger, The ten-twenty electrode
system of the International Federation. Electroencephalogram Clin.
Neurophysiol. 52, 3, 1999.
[11] Kasabov, N.K. ”NeuCube: A spiking neural network architecture for
mapping, learning and understanding of spatio-temporal brain data,”
Neural Netw. 52, 62–76, 2014.
[12] N. Kasabov, N. Scott et al., ”Design methodology and selected ap-
plications of evolving spatio-temporal data machines in the NeuCube
neuromorphic framework,” Neural Netw. 78, 1–14, 2016.
Fig. 9. Neuron proprotion for four electrode channels.
[13] J. Talairach, P. Tournoux, Co-planar Stereotaxic Atlas of The Human
Brain: 3-Dimensional Proportional System: An Approach to Cerebral
Imaging (Thieme Medical Publishers, New York, 1988)
IV. C ONCLUSION [14] S. Song, K. Miller, L. ”Abbott et al., Competitive Hebbian learning
This paper shows a method based on spiking neural net- through spike-timing-dependent synaptic plasticity,” Nat. Neurosci. 3,
919–926, 2000.
works for an emotional binary valence classification task by [15] C. Tan, G. Ceballos, N. Kasabov, and N. Puthanmadam Subramaniyam,
using EEG data. Unlike [14], only EEG data are used in this “FusionSense: Emotion Classification Using Feature Fusion of Mul-
work. At the same time, in order to balance the performance timodal Data and Deep Learning in a Brain-Inspired Spiking Neural
Network,” Sensors, vol. 20, no. 18, p. 5328, Sep. 2020.
and computational consumption, only 4 channels of data [16] N. Kasabov, K. Dhoble, N. Nuntalid, G. Indiveri, Dynamic evolving
(the last 30 seconds) are selected. Finally, we obtained an spiking neural networks for on-line spatio–and spectro-temporal pattern
accuracy of 68.91% under the 10-fold cross-validation method. recognition. Neural Netw. 41, 188–201, 2013.
[17] S. Thorpe, J. ”Gautrais, Rank order coding. Computational Neuro-
Compared with Fp1 and Fp2, the EEG data recorded from F3 science: Trends Res,” 13, 113–119. 1998.
and F4 electrode channels provide more information in this [18] Ali, M., et al. ”EEG-based emotion recognition approach for e-
binary valence classification task. The results prove that the healthcare applications.” 2016 Eighth IEEE International Conference on
Ubiquitous and Future Networks (ICUFN) IEEE, 2016.
spiking neural networks can be applied to the task of emotion [19] N. Kumar, K. Khaund, and S. M. Hazarika, “Bispectral analysis of
classification effectively. Methods based on spiking neural eeg for emotion recognition,” Procedia Computer Science, vol. 84, pp.
networks are more biologically reasonable compared with 31–35, 2016.

437
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY CALICUT. Downloaded on September 01,2023 at 09:57:42 UTC from IEEE Xplore. Restrictions apply.