Università di Pisa

Sentiment Classification

Human Language Technologies

Giuseppe Attardi
Unsupervised Sentiment Classification
 Unsupervised methods do not require labeled examples.
 Knowledge about the task is usually added by using lexical resources and
 hard-coded heuristics, e.g.:
 Lexicons + patterns: VADER
 Patterns + Simple language model: SO-PMI
 Neural language models have been found that they learn to recognize
 sentiment with no explicit knowledge about the task.
Supervised/unsupervised
 Supervised learning methods are the most commonly used one, yet also
 some unsupervised methods have been successfully.
 Unsupervised methods rely on the shared and recurrent characteristics
of the
 sentiment dimension across topics to perform classification by means of
 hand-made heuristics and simple language models.
 Supervised methods rely on a training set of labeled examples that
describe
 the correct classification label to be assigned to a number of documents.
 A learning algorithm then exploits the examples to model a general
 classification function.
VADER
 VADER (Valence Aware Dictionary for
sEntiment Reasoning) uses a curated lexicon
derived from well known sentiment lexicons
that assigns a positivity/negativity score to
7k+ words/emoticons.
 It also uses a number of hand-written
pattern matching rules (e.g., negation,
intensifiers) to modify the contribution of
the original word scores to the overall
sentiment of text.
 Hutto and Gilbert. VADER: A Parsimonious
Rule-based Model for Sentiment Analysis of
Social Media Text. ICWSM 2014.
 VADER is integrated into NLTK
The classification pipeline
The elements of a classification pipeline are:
1. Tokenization
2. Feature extraction
3. Feature selection
4. Weighting
5. Learning
 Steps from 1 to 4 define the feature space and how text is converted into
vectors.
 Step 5 creates the classification model.
Scikit-learn
 The scikit-learn library defines a rich number of data processing and
machine learning algorithms.
 Most modules in scikit implement a 'fit-transform' interface:
 fit method learns the parameter of the module from input data
 transform method apply the method implemented by the module to the data
 fit_transform does both actions in sequence and is useful to connect modules
in a pipeline.
Sentiment Analysis on
Tweets
Evolution
 SemEval Shared Task Competition
 2013, Task 2
 2014, Task 9
 2015, Task 10
 2016
 2017
 Evolution of technology:
 Top system in 2013: SVM with sentiment lexicons and many lexical features
 Top system in 2016: CNN with word embeddings
 In 2017: most systems used CNN or variants
SemEval 2013, Task 2
Best Submission:
TNRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets,
Saif M. Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu, In Proceedings
of the seventh international workshop on Semantic Evaluation Exercises
(SemEval-2013), June 2013, Atlanta, USA.
SemEval 2015 – Task 10
Best Submission:
A. Severyn, A. Moschitti. 2015. UNITN: Training Deep Convolutional Neural
Network for Twitter Sentiment Classification. Proceedings of the 9th
International Workshop on Semantic Evaluation (SemEval 2015), pages 464–
469, Denver, Colorado, June 4-5, 2015.
https://www.aclweb.org/anthology/S15-2079
Deep Learning for Sentiment
Analysis
Convolutional Neural Network
 A convolutional layer in a NN is A filter using a window length of 5 is
composed by a set of filters. applied to all the sequences of 5 words in a
 A filter combines a "local" selection of input text.
values into an output value. 3 filters using a window of 5 applied to a
 All filters are "sweeped" across all input. text of 10 words produce 18 output values.
Why?
 During training each filter specializes
Filters have additional parameters that
into recognizing some kind of relevant
define their behavior at the start/end of
combination of features. documents (padding), the size of the sweep
 CNNs work well on stationary features, step (stride), the possible presence of holes
in the filter window (dilation).
i.e., those independent from position.
CNN for Sentiment Classification
S
Not
-
going
-
to
+
the
beach
tomorrow
:-(

embeddings convolutional layer with Multilayer perceptron

max over time pooling with dropout
for each word multiple filters

1. Embeddings Layer, Rd (d = 300)

2. Convolutional Layer with Relu activation
Multiple filters of sliding windows of various sizes h
ci = f(F  Si:i+h−1 + b)
3. max-pooling layer Frobenius elementwise
4. dropout layer matrix product
5. linear layer with tanh activation
6. softmax layer
Distant Supervision
 A. Severyn and A. Moschitti, UNITN at SemEval 2015 Task 10.
 Word embeddings from plain text are completely clueless about their
sentiment behavior
 Distant supervision approach using our convolutional neural network to
further refine the embeddings
 Collected 10M tweets treating tweets containing positive emoticons,
used as distantly supervised labels to train sentiment-aware
embeddings
Results of UNITN on SemEval 2015

Phrase-level subtask A
Dataset Score Rank
Twitter 15 84.79 1

Message-level subtask B
Dataset Score Rank
Twitter 15 64.59 2
Sentiment Specific Word Embeddings
 Sentiment Specific Word Embeddings
LM likelihood + Polarity
 

       

       

the cat sits on

 Uses an annotated corpus with polarities (e.g. tweets)
 SS Word Embeddings achieve SotA accuracy on tweet sentiment
classification
 G. Attardi, D. Saertiano. UniPi at SemEval 2016 Task 4: Convolutional
Neural Networks for Sentiment Classification.
https://www.aclweb.org/anthology/S/S16/S16-1033.pdf
 Learning SS Embeddings
Generic loss function x is a sentence and xc is a
LCW(x, xc) = max(0, 1  f(x) + f(xc)) corrupted sentence,
obtained by replacing the
SS loss function center word with a random
LSS(x, xc) = max(0, 1  ds(x) f(x)1 + ds(x) f(xc)1) word

Gradients f(x)  {0, 1}2

is the function computed
by the network

represents the polarity of x

Semeval 2015 Sentiment on Tweets
Team Phrase Level Tweet
Polarity
Attardi (unofficial) 67.28
UNITN 84.79 64.59
KLUEless 84.51 61.20
IOA 82.76 62.62
WarwickDCS 82.46 57.62
Webis 64.84
SwissCheese at SemEval 2016

Three-phase procedure:
1. creation of word embeddings for initialization of the first layer.
Word2vec on an unlabeled corpus of 200M tweets.
2. distant supervised phase, where the network weights and word
embeddings are trained to capture aspects related to sentiment.
Emoticons used to infer the polarity of a balanced set of 90M tweets.
3. supervised phase, where the network is trained on the provided
supervised training data.
Ensemble of Classifiers
 Ensemble of classifiers
 combining the outputs of two 2-layer CNNs having similar architectures but
differing in the choice of certain parameters (such as the number of
convolutional filters).
 networks were also initialized using different word embeddings and used
slightly different training data for the distant supervised phase.
 A total of 7 outputs were combined
Results
  2013 2014 2015 2016 Tweet
Live-
Sarcas Avg
  Tweet SMS Tweet Journa Tweet Acc
m F1
l
SwissCheese 70.0 63.72 71.62 56.61 69.57 67.11 63.31 64.61
Combination 5

SwissCheese
67.00 69.12 62.00 71.32 61.01 57.19
single

UniPI 59.218 58.511 62.718 38.125 65.412 58.619 57.118 63.93

UniPI SWE 64.2 60.6 68.4 48.1 66.8 63.5 59.2 65.2
Breakdown over all test sets
SwissCheese Prec. Rec. F1 UniPI 3 Prec. Rec. F1
positive 67.48 74.14 70.66 positive 70.88 65.35 68.00
negative 53.26 67.86 59.68 negative 50.29 58.93 54.27
neutral 71.47 59.51 64.94 neutral 68.02 68.12 68.07
Avg F1     65.17 Avg F1     61.14
Accuracy     64.62 Accuracy     65.64
Sentiment Classification from a single neuron
 A char-level LSTM with 4096 units has been
trained on 82 millions reviews from Amazon.
 The model is trained only to predict the next
character in the text
 After training one of the units had a very
high correlation with sentiment, resulting in
state-of-the-art accuracy when used as a
classifier.
 The model can be used to generate text.
 By setting the value of the sentiment unit,
one can control the sentiment of the Blog post - Radford et al. Learning to
Generate Reviews and Discovering
resulting text. Sentiment. Arxiv 1704.01444