2018 2nd Int. Conf.

on Innovations in Science, Engineering and Technology (ICISET)

27-28 October 2018, Chittagong, Bangladesh

Emotion Detection from Text

Using Skip-thought Vectors
Maruf Hassan Md. Sakib Bin Alam Tanveer Ahsan
Department of Computer Science and Department of Computer Science and Department of Computer Science and
Engineering Engineering Engineering
International Islamic University International Islamic University International Islamic University
Chittagong Chittagong Chittagong
Chittagong, Bangladesh Chittagong, Bangladesh Chittagong, Bangladesh
[email protected] [email protected] [email protected]

Abstract—Emotion detection from natural language has Deep Learning architectures and algorithms have made
become a popular task because of the primary role of emotions in outstanding advancements in the field of Natural Language
human-machine interaction. It has a wide variety of applications Processing (NLP) in recent years. The idea of the usage of vector
ranging from developing emotional chatbots to better representation of words has been there around for a few times.
understanding people and their lives. The problem of finding But the interest in word embedding has been hovering in the
emotion from text has been handled by using lexical approaches recent years. Word embedding is a technique that produces
and machine learning techniques. In recent years neural network vectors and maps them into corresponding words. Tomas
models have become increasingly popular for text classification. Mikolov's word2vec [2] model which is trained on a large
Especially, the emergence of word embeddings within deep
collection of corpora produce high-dimensional (50 - 300
learning architectures has recently drawn a high level of attention
amongst researchers. In this research, we apply a recently
dimensional) vector representation corresponding to words that
proposed deep learning model named skip-thought, an approach can capture the syntactic and semantic knowledge. This
to learning fixed length representations of sentences, to face the representation seems to capture much linguistic information. In
problem of emotion detection from text. We propose a new word2vec model, they showed fixed length representations of
framework that takes advantage of the pre-trained model and pre- individual words. However, fixed length representation of
trained word vectors. We found that skip-thought vectors are well sentences was an issue there.
suited for emotion detection task. The results of the performance
Skip-thought vectors or simply skip-thought [3] is a name
evaluation are encouraging and comparable to related research.
given to a simple neural networks model for learning fixed
Keywords— emotion detection; machine learning; deep length vector representations of sentences. It was inspired by the
learning; recurrent neural network, word embedding; sentence skip-gram structure used in word2vec model, abstract the skip-
embedding; gram model to the sentence level. Fixed length vector
representation of sentences makes it easy to replace any sentence
with equivalent vectors of numbers. This makes the process of
I. INTRODUCTION understanding and responding to natural language
Emotion is an essential component of human behavior, a mathematically straightforward.
complex experience of consciousness that reflects the personal
criticalness of a thing, an occasion, or a situation. Emotion To get good performance in these techniques, availability of
detection is the process of recognizing a person's emotional large-scale data is needed to train. But, in our placing, we have
state, e.g. as stated in Ekman's basic emotions; specifically, only small datasets. Deep learning model is incapable of giving
anger, disgust, fear, happiness, sadness, and surprise [1]. good result on small datasets. That’s why we make the decision
Accurately detecting emotion from natural language has to take a pre-trained model. We propose a framework that
application of developing emotional chatbots which extent to utilizes the pre-trained model for finding a sentence level feature
better understanding persons and their behavior, developing representation using the word vectors and evaluate how it
intelligent personal assistants to detecting the emotions of social performs well for emotion detection task.
media users to understand their psychological and physical The rest of this paper is structured as follows. Section II
health. discusses related work. Section III provides a brief introduction
Computer system has not yet utilized emotion detection to the skip-thought model that we used for our proposed
properly to facilitate humans in performing their assignments. framework. Section IV introduces our proposed approach to
Linguistic knowledge still becomes the focus on the continued emotion detection. Section V carries the two data sets. We report
efforts in emotion analysis. Language is inherently ambiguous on the experiments and results in Section VI and conclude the
and complex by nature. It is the most important challenges paper in Section VII.
researcher encounter in emotion analysis. Also, presently there
aren't any established classifications of “all human emotions” II. RELATED WORK
because of the complicated characteristics of human minds. Emotion detection is closely connected to sentiment
analysis, they are often used interchangeably. By proposing the

978-1-5386-8524-2/18/$31.00 ©2018 IEEE

role of emotions in human-computer interaction, the concept of are not large enough to train for good vectors. Skip-thought
affective computing was introduced by Picard [4]. model has already generated high-quality word vectors [3].
Inspired by the good results of the skip-thought vectors on
In our literature review, we face two approaches that have different tasks such as semantic relatedness, sentiment analysis
been occurred frequently to face the problem of emotional text etc., we intent to use this technique for our emotion detection
analysis task. Lexicon-based methods use a lexicon of affective task.
words or phrases to detect emotions in text. The primary focus
of lexical-based methods has been on the employment of One of the previous researches showed how to utilize a pre-
lexicons of emotion words. The second one depends on machine trained model to improve the performance of emotion detection
learning tools and technologies. task [20]. Our research is different from their work in a way that
they focused on word embedding as a feature vector. In our
A. Lexicon-based Approach work, we try to apply recurrent neural network (RNN) based
Lexicon-based method is a straightforward method and sentence embedding for emotion detection task as it preserves
considerably easier to implement than other methods. In this much more context.
method, each emotional word carries emotional valences. These
valences are calculated to get the emotional context in the whole III. SKIP-THOUGHT MODEL OVERVIEW
document [5]. The capability of lexicon-based systems relies on In this section, we give a brief introduction to the skip-
the coverage and quality of the chosen emotion lexicon [6] [7]. thought model that we used for our proposed framework. We
Lexical-based method ends up in poor performance and describe how skip-thought model was developed and how the
generalization during unavailability of emotional words in text skip-thought vectors were generated.
[8]. Also, it showed bad performance in handling more complex
linguistic structures of text. Let’s say, we are given three sequential sentences (si-1, si,
si+1). Here si is the i-th sentence from the corpus.
B. Learning-based Approach
Learning based methods are being employed to face the
problem differently. Machine learning algorithms use
computational methods that depend on various classifiers.
Performance comparison of four different classifiers (Bayesian,
Random Forest, Logistic Regression, and SVM) can be found in
[9]. One of the major problems of learning-based approach is
that it results in blurred borders between emotion categories and
an absence of context analysis. They are domain (dataset topic)
dependent and datasets collected at different time periods Fig. 1. Skip-thought Model Overview.
require different features. Also, the classifier needs to be trained
on large feature sets. As a result, long training sessions are In general, the encoder generates the fixed-length
needed and because of the enormous number of features, the representation of the input sentence si and gives the embedding
classifiers take much more time and memory to produce result. zi. The decoders are trained to reduce the reconstruction error of
the preceding sentence si-1 and succeeding sentence si+1 given
C. Deep Learning for NLP the embedding zi. This reconstruction error is back-propagated
to the encoder which maps the relationship of the input sentence
Neural network [10] has come out as an undefeated category
to its surrounding sentence. This mapping is shown in Figure 1.
of techniques that has the ability of spontaneously finding the
representations required for recognition or classification and has The end product of skip-thought model is the encoder
been efficiently carried out a couple of NLP assignments. because it captures both syntactic and semantic properties. The
Named entity recognition (NER), a part of speech (POS) tagging decoders are thrown away after training. The lookup table is
or sentiment analysis are a few the issues wherever neural shared among encoder and decoders allows high-quality word
network models have outperformed ancient approaches. A line embeddings for not only input sentences, but also their context.
of research [11] [12] [13] within the literature target to learn After training, the trained encoder along with the lookup table
sentiment-specific word and phrase embeddings from can then be used as a generic feature extractor for new input
neighboring textual content. On the other hand, some analysis sentences.
concentrates on learning linguistics composition, as well as they
extend to phrases and sentences using recurring neural networks IV. PROPOSED FRAMEWORK FOR EMOTION DETECTION
[14] [15]. There exist two types of Recurrent Neural Networks
(RNN), Gated Recurrent Unit (GRU) [16] [17] and Long-short In this section, we give the detailed description of our
term memory (LSTM) [18] for managing sequential data like proposed framework. Our objective is to entitle appropriate
speech or character recognition. They are also being used emotion label to a text document based on its context. The
effectively for task such as sentiment analysis [13] [19]. overview of our proposed framework for emotion detection task
is depicted in Figure 2.
To generate high-quality word vectors from domain-specific
dataset, we need a large volume of data and heavy computation
resources. All the datasets available for emotion detection task
 In the beginning, our system fetches all the text from the user results in a continuous vector for each input word. Now we will
and queue them for further processing Next stages are described use an RNN with GRU activations to convert the sequence of
in the following. continuous vectors corresponding the words into sentence
vectors. It is shown in the previous research that on language
modeling tasks, GRU and LSTM show analogous performance,
also they are conceptually similar [21]. To encode a sentence,
we iterate the following sequence of equations as suggested in
[3].
(1)

(2)

tanh ⊙ (3)

1 ⊙ ⊙ (4)

Here, = reset gate, = update gate, = embedding for

Fig. 2. Our proposed emotion detection framework
the word, = proposed state update at time t, W, U = parameter
matrices, ⊙ = element-wise product, = sigmoid function,
A. Text Preprocessing
tanh = hyperbolic tangent function.
Since our goal is to evaluate how skip-thought vectors
perform well for emotion detection task, we keep text We use two separate technique for the encoder to extract
preprocessing to a minimum. To encode a new sentence, we features as suggested in [3]. These are uni-skip and bi-skip. For
applied simple word tokenization, stop words removal and uni-skip, the encoder contains a unidirectional forward GRU
conversion of negative short form to long form e.g. "won't" is with 2400 dimension. For bi-skip, the encoder contains a
replaced by "will not". This is carried out to check the forward and a backward GRU with 1200 dimension of each,
effectiveness of the skip-thought vectors for emotion detection forward GRU is offered the sentence in correct order, backward
task. We used nltk1 library, a natural language toolkit, for text GRU is offered the sentence in reverse order. One output is
preprocessing. appended with other to shape a 2400-dimensional vector. We
concatenate uni-skip and bi-skip features which we refer to as
B. Feature Extraction combine-skip. This is 4800-dimensional vectors.
We embed the trained encoder from the skip-thought model Now that we have a summary vector that represents a whole
along with the lookup table layer as depicted in Figure 3. We use input text. This is our feature vector. We feed this feature vector
this layer as feature extractor for our emotion detection task. to the machine learning classifier.

C. Classifier
We use Logistic Regression classifier for classification task.
We select it for two reasons. The first is, it is a simple linear
classifier and fast to train. The second reason is to directly
evaluate the performance of the skip-thought vectors on emotion
detection task. It is possible to use a non-linear classifier such as
Support Vector Machine (SVM) with some additional parameter
tuning but that’s beyond the scope of this study. Our goal is to
analyze the strength of the computed skip-thought vectors on
emotion detection task. We used scikit-learn2 for Logistic
Regression implementation.
After running the system, set of emotion labeled text is
Fig. 3. Feature extraction from the skip-thought model for our framework. returned to the user.

The goal is to generate a vector that summarizes the whole V. DATASETS

input text. To do this, each word in the input sentence is
To analyze the performance of skip-thought vectors on
represented as a one-hot encoded vector. The encoder linearly
projects the one-hot encoded vector wi with a parameter matrix emotion detection task, we have explored two publicly available
E. This matrix is initialized from the lookup table. This research data (summarized in Table 1).
projection si = Ewi (si is the continuous representation of word),

1 2
https://www.nltk.org/ http://scikit-learn.org/
 • ISEAR3 [22] (International Survey on Emotion TABLE 2 SAMPLE INPUT AND OUTPUT FROM ISEAR DATASET
Antecedents and Reactions) contains 7,666 labeled
sentences where 1096 participants from different cultural Input Extracted Feature Output
backgrounds accomplished a set of written questions “I received a letter from a [0.418, 0.04492, 0. 32111,
Joy
with a choice of answers about the experiences and distant friend” ….. ,-0.46765]
reactions to seven emotions. This is a well-balanced “Close relative is dying of
[0.625, -0.8659, -0. 37282,
dataset meaning that the dataset is not biased towards any cancer - found out last Sadness
….. 0.8372]
week”
specific emotion label. “I lied to one of my best [-0.837, 0.37282, -0. 352,
Guilt
friends” ….. ,0.5847]
• Fairy Tales [23] dataset includes annotated sentences
4

from fairy tales. The dataset is not well-balanced. In this B. Classification Report
dataset, there exist related semantics between anger and
Based on the several experimental setups, the summarized
disgust. The author of the dataset merges these two
results are presented in Table 3 in terms of several evaluation
emotions to represent one class.
measures including Recall (R), Precision (P), F1 Score (F1).

TABLE 1 DATASET CHARACTERISTICS

TABLE 3 CLASSIFICATION REPORT FOR BOTH ISEAR AND FAIRY TALE
Dataset # of data # of class Emotion Classes DATASET

Angry, Disgust, Fear, Joy,

ISEAR 7,666 7 ISEAR Fairy Tale
Sad, Shame, Surprise
Angry-Disgusted, Fearful, Emotion P R F1 P R F1
Fairy Tale 1207 5
Happy, Sad, Surprise
Joy/Happy 0.700 0.792 0.742 0.732 0.832 0.778

Fear/Fearful 0.692 0.726 0.710 0.636 0.568 0.598

VI. EXPERIMENTS
Anger 0.474 0.476 0.474 0.656 0.646 0.644
A. Experimental Setup Sadness 0.630 0.596 0.612 0.652 0.650 0.650
We run our framework for both ISEAR and Fairy Tale Disgust 0.610 0.598 0.606
dataset separately. The pre-trained word vector source from - - -
skip-thought model are used as word embedding source in our Shame 0.524 0.490 0.504 - - -
framework. The dimension of word embedding in skip-thought Guilt 0.526 0.496 0.510
vectors is 620. To load the pre-trained model, we use several - - -
python libraries such as cPickle5, numpy6. For the RNN GRU Surprise - - - 0.534 0.332 0.406
implementation, Theano7, a numerical computation library for Average / Total 0.594 0.596 0.594 0.672 0.672 0.666
python, is used. We used 10-fold cross-validation technique to
evaluate the performance of our framework.
C. Performance Analysis
The performance of Logistic Regression depends on the When our proposed framework was employed individually
parameter C. We need to be careful about selecting the value of on ISEAR dataset, it showed an average precision of 59.4%,
C. For each fold, we arbitrarily choose the value [1, 2, 4, 8, 16, average recall of 59.6%, and average F1-measure up to 59.4%.
32, 64, 128, 256] for parameter C, run another nested 10-fold Our framework exhibits high performance for two emotional
cross-validation and check which value gives the best scores. classes, namely, for the joy and fear classes. Joy appears to be
We select that value of C for the outer cross-validation checking. the only category on the positive side of the emotional
During 10-fold cross validation running, we save the scores from categories, contrasted with the other negative emotions fear,
each fold and take the mean of all the scores. anger, sadness, disgust, shame, guilt. Anger category showed
Sample input and output from our framework using ISEAR lower precision of only 47.4%. This is probably because what
dataset are shown below in Table 2. Here, the input is emotion causes someone angry could be the reason of satisfaction to
text and output are the predicted emotion label. The feature another.
extracted from the input text is 4800-dimensional. On Fairy Tales dataset, it showed an average precision of
67.2%, average recall of 67.2%, and average F1-measure up to
66.6%. Fairy Tale dataset was not so well balanced, this dataset
is biased to happy emotion, a number of data in surprise category
are sparse. We can see the effect of such inconsistency in the
classification report. Happy category showed a high recall of
83.2%. On the other side, Surprise category showed a lower
recall of 33.2%.

3 6
https://www.gelbukh.com/emosenticnet/ http://www.numpy.org/
4
http://people.rc.rit.edu/~coagla/affectdata/ 7
http://deeplearning.net/software/theano/
5
https://docs.python.org/2.3/lib/module-cPickle.html
D. Word Embedding Methods Comparison [2] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado and J. Dean,
"Distributed representations of words and phrases and their
We compared the performance of skip-thought vectors compositionality," in Advances in neural information processing
using our proposed framework with two publicly accessible pre- systems, 2013.
trained word vector sources, based on GloVe8 [24] and [3] R. Kiros, Y. Zhu, R. R. Salakhutdinov, R. Zemel, R. Urtasun, A. Torralba
word2vec9, regarding emotion detection performance. The and S. Fidler, "Skip-thought vectors," in Advances in neural information
dimension of word embedding in both sources are 300. processing systems, 2015.
[4] R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W.
Emotion detection datasets have been assigned to multiple Fellenz and J. G. Taylor, "Emotion recognition in human-computer
emotions and are not balanced. That’s why we select macro interaction," IEEE Signal Processing Magazine, vol. 18, pp. 32-80, 2001.
average F1-score for comparative performance analysis of the [5] L. Polanyi and A. Zaenen, "Contextual valence shifters," in Computing
proposed framework with the related work. We take Continuous attitude and affect in text: Theory and applications, Springer, 2006, pp.
bag of words (CBOW) classifier as our baseline from [20]. 1-10.
[6] D. Ghazi, D. Inkpen and S. Szpakowicz, "Prior versus Contextual
Emotion of a Word in a Sentence," in Proceedings of the 3rd Workshop
TABLE 4 COMPARISON OF EACH PRE-TRAINED WORD VECTOR SOURCE
in Computational Approaches to Subjectivity and Sentiment Analysis,
WORD2VEC, GLOVE, AND SKIP-THOUGHT IN TERMS OF MACRO F1-SCORES
2012.
Word2vec Glove [7] M. Taboada, J. Brooke, M. Tofiloski, K. Voll and M. Stede, "Lexicon-
Dataset Skip-thought
CBOW CBOW based methods for sentiment analysis," Computational linguistics, vol.
ISEAR 0.566 0.577 0.594 37, pp. 267-307, 2011.
Fairy Tales 0.617 0.661 0.666 [8] A. Koumpouri, I. Mporas and V. Megalooikonomou, "Evaluation of
Four Approaches for Sentiment Analysis on Movie Reviews: The Kaggle
Competition," in Proceedings of the 16th International Conference on
Table 4 shows the macro F1-scores for both datasets and Engineering Applications of Neural Networks (INNS), 2015.
each text representation technique that is based on word vectors [9] R. Meo and E. Sulis, "Processing affect in social media: a comparison of
and sentence vectors. The highest scores are printed in bold font. methods to distinguish emotions in tweets," ACM Transactions on
From the result, we can take the decision that for both dataset Internet Technology (TOIT), vol. 17, p. 7, 2017.
skip-thought vectors performed better than using word2vec and [10] I. Goodfellow, Y. Bengio, A. Courville and Y. Bengio, Deep learning,
Glove based vectors. For instance, on an average, skip-thought vol. 1, MIT press Cambridge, 2016.
vectors sources showed a 2 - 5% improvement in F1-score [11] I. Labutov and H. Lipson, "Re-embedding words," in Proceedings of the
relative to word2vec. 51st Annual Meeting of the Association for Computational Linguistics
(Volume 2: Short Papers), 2013.
[12] A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng and C. Potts,
VII. CONCLUSION AND FUTURE WORK "Learning word vectors for sentiment analysis," in Proceedings of the
49th annual meeting of the association for computational linguistics:
In this research, we presented a recently proposed deep
Human language technologies-volume 1, 2011.
learning model named skip-thought for representing sentences
[13] D. Tang, B. Qin and T. Liu, "Document modeling with gated recurrent
for emotion detection task. We demonstrated the use of a pre- neural network for sentiment classification," in Proceedings of the 2015
trained sentence embedding model for classifying emotions in conference on empirical methods in natural language processing, 2015.
user's text. We proposed a framework for emotion detection task [14] J. Bradbury, S. Merity, C. Xiong and R. Socher, "Quasi-recurrent neural
which embeds the trained encoder from the skip-thought model networks," arXiv preprint arXiv:1611.01576, 2016.
along with the lookup table layer. To validate the framework, we [15] O. Irsoy and C. Cardie, "Deep recursive neural networks for
executed a set of experiments on two datasets ISEAR and Fairy compositionality in language," in Advances in neural information
Tales to explore how effective this framework is at detecting processing systems, 2014.
emotions. The RNN sentence embedding technique with skip- [16] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H.
thought vectors has exhibited satisfactory results across all Schwenk and Y. Bengio, "Learning phrase representations using RNN
performance metrics. We have shown how the problem of encoder-decoder for statistical machine translation," arXiv preprint
emotion detection and classification at the sentence level can be arXiv:1406.1078, 2014.
tackled as one of supervised classification, even with relatively [17] J. Chung, C. Gulcehre, K. Cho and Y. Bengio, "Gated feedback recurrent
small labeled datasets. neural networks," in International Conference on Machine Learning,
2015.
In the future, we want to investigate how other word vector [18] S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural
sources such as word2vec and Glove perform using RNN computation, vol. 9, pp. 1735-1780, 1997.
sentence embedding technique in terms of emotion detection [19] M. Zhang, Y. Zhang and D.-T. Vo, "Gated Neural Networks for Targeted
task. We also plan to train the skip-thought model on domain Sentiment Analysis.," in AAAI, 2016.
adapted pseudo-labeled large-scale datasets. [20] J. Herzig, M. Shmueli-Scheuer and D. Konopnicki, "Emotion Detection
from Text via Ensemble Classification Using Word Embeddings," in
Proceedings of the ACM SIGIR International Conference on Theory of
REFERENCES Information Retrieval, 2017.
[21] J. Chung, C. Gulcehre, K. Cho and Y. Bengio, "Empirical evaluation of
[1] P. Ekman, "An argument for basic emotions," Cognition & emotion, vol. gated recurrent neural networks on sequence modeling," arXiv preprint
6, pp. 169-200, 1992. arXiv:1412.3555, 2014.

8 9
https://nlp.stanford.edu/data/glove.840B.300d.zip https://code.google.com/archive/p/word2vec/
[22] K. R. Scherer and H. G. Wallbott, "Evidence for universality and cultural [24] J. Pennington, R. Socher and C. Manning, "Glove: Global vectors for
variation of differential emotion response patterning.," Journal of word representation," in Proceedings of the 2014 conference on
personality and social psychology, vol. 66, p. 310, 1994. empirical methods in natural language processing (EMNLP), 2014.
[23] E. C. O. Alm, Affect in* text and speech, University of Illinois at Urbana-
Champaign, 2008.