IJCAT - International Journal of Computing and Technology, Volume 7, Issue 6, June 2020

ISSN (Online) : 2348-6090

www.IJCAT.org
Impact Factor: 0.835

Machine Learning Based Sentiment Analysis for Text

Messages
1
Abhishek Bhagat; 2Akash Sharma; 3 Sarat Kr. Chettri
1, 2
Department of Computer Science & Engineering
Assam Don Bosco University, Assam, India
3
Department of Computer Applications
Assam Don Bosco University, Assam, India

Abstract - People use online platforms such as Facebook, Twitter, etc. for social networking and share their opinions, feelings or beliefs
with others. Sharing is done by posts on these platforms. Sentiment analysis or opinion mining of these posts using machine learning
techniques is of great significance. The analysis is generally carried out with sentiment, subjectivity analysis or polarity calculations. In
this study, we performed a sentiment analysis of text messages using supervised machine learning techniques. They are mainly online
product reviews, general tweets in Tweeter and movie reviews. Messages are pre-processed and then used three different machine
learning techniques, namely Naïve Bayes, Decision Tree and Support Vector Machine (SVM) for sentiment analysis.

Keywords - Machine Learning, Natural Language Processing, Sentiment Analysis, Twitter

1. Introduction
follows; in Section 2, we discuss about related work in this
area. Section 3 is about the proposed system of our

I n an online micro-blogging platform like Twitter,

millions of tweets on various topics are created every
day. These tweets are very important for understanding
the trending themes. Classification of Twitter messages is
one of the most important fields of research related to
approach towards sentiment analysis in opinions and
tweets. Section 4 discusses about machine learning
techniques followed by implementation in Section 5 and
finally we conclude with future prospects.
tweets. It is important and appropriate to identify these
topics in different categories with high accuracy for better 2. Related Work
information retrieval. Sentiment analysis is a method to
determine the polarity of a piece of writing as positive, Several studies have been conducted in the field of
negative or neutral. A text sentiment analysis program sentiment analysis and it can be concluded that sentiment
incorporates natural language processing (NLP) and analysis of text messages can be carried out in a variety of
machine learning techniques to assign weighted sentiment ways. The work [1] shows the types and techniques of
scores to persons, subjects, themes and categories within a sentiment analysis used to extract sentiments from tweets
sentence or expression. and have taken a comparative study of the different
techniques and approaches of sentiment analysis using
Machine learning techniques which are generally used for twitter as data. Various supervised learning methods on
sentiment analysis, one is unsupervised and the other is pre-processing and information extraction of tweets from
supervised. Unsupervised learning does not consist of a twitter have also been explored [2]. The author concludes
category and they do not provide with the correct targets at that Support Vector Machine (SVM) for text
all and therefore conduct clustering. Supervised learning is categorization can be used to detect the polarity of a text
based on labeled dataset and thus the labels are provided to tweet. It was inferred that SVM recognizes some text
the model during the process. properties such as High Dimensional Feature Space, few
irrelevant features, and a sparse instance vector. The
In this study, we focused on supervised machine learning performance of SVM can be evaluated using precision and
algorithms where the labeled datasets are used to train and recall. Different results show that SVM delivers strong
check to generate appropriate outputs when encountered text categorization efficiency compared to artificial neural
during decision-making. The rest of the paper proceeds as network (ANN).

103
IJCAT - International Journal of Computing and Technology, Volume 7, Issue 6, June 2020
ISSN (Online) : 2348-6090
www.IJCAT.org
Impact Factor: 0.835

Table 1. Summary of some of the attempts made by authors in sentiment analysis of text data
References
Data Techniques Approaches
(Year)
The effectiveness of applying Naïve Bayes and Support
[8] Naïve Bayes, Vector Machine are explored to classify sentiments based
IMDB dataset
(2002) SVM on the reviews made in movies.
Proposed a system based on SVM and Fuzzy domain
[9] Online hotel Fuzzy ontology ontology for opinion mining based on the collection of
(2016) reviews with SVM online reviews about hotels. The system computes the
polarity term of each feature.

The main approach followed here is to summarize and

Semi-supervised classify online user reviews of products in order to guide
[10] Online product
learning consumers in making decisions while making online
(2013) reviews
technique purchases.

Proposed a three-stage cascade model to address the

Multi-domain polarity shift problem in the context of document-level
Rule based and sentiment classification Three classification algorithms
[11] sentiment data
statistical were also evaluated; SVM, logistic regression and Naïve
(2016) extracted from
method Bayes.
Amazon.com

Proposed a model based on the original ontology-based

techniques for efficient analysis of Twitter posts
[12] Domain sentiments. The Twitter posts are not simply characterized
Twitter dataset by a sentiment score but sentiment grade is assigned to
(2013) Ontology
each distinct notion available in the post.

Sentiment classification techniques were applied to movie

reviews and comparison of three supervised machine
[13] SVM, Naive learning techniques has been made namely, SVM, Naïve
Movie reviews Bayes, and kNN for classifying the sentiments.
(2013) Bayes and KNN

Opinion mining approach has been used to summarize user

reviews, which are mainly unstructured and non-
[14] Online product grammatical. The author attempts to perform the
SVM classification of features and polarity for each class of
(2012) reviews
features.

Proposed a framework that automatically prepares training

and testing datasets for sentiment mining in order to
[15] Naïve Bayes analyze the online reviews made by hotel customers of
Hotel reviews their services.
(2018) multinomial

The authors attempt to do a sentiment analysis of

comments from people on Oman tourism on social media.
Tourism The sentiment analysis is performed using a domain
[16] Domain specific
reviews in specific ontology and a POS tagger.
(2019) ontology
social media

104
IJCAT - International Journal of Computing and Technology, Volume 7, Issue 6, June 2020
ISSN (Online) : 2348-6090
www.IJCAT.org
Impact Factor: 0.835

In the works [3, 6, 7], authors find keywords in tweet and

predict whether they have positive or negative weights by
applying machine-leaning algorithms. Multi-classification
algorithms such as SVM, Naïve Bayes, Logistic
Regression for classification, KNN and Decision Tree
have been used for trend analysis prediction. In another
work [4], the authors discuss methods for the automatic
extraction of sentiments or opinions of the text unit. The
proposed model deals with mining emotions of tweets
about Indian politicians using Support Vector Machine.

They have used unigram and TF-IDF as feature vectors

measured the performance of the proposed system in terms
of classification accuracy, and F-measure. Y. Luo et. al [5]
has made a comparison of positive and negative sentences.
Their proposed model extracts information from the web
and manually labels a word set that requires a lot of effort.
The main focus of the sentiment analysis is to categorize
people's opinions by analyzing their views on social media.
Table 1 provides a summary of the works done by
different authors in the area of sentiment analysis or
opinion mining available in the literature.

3. Proposed System

We present a framework where there are multiple layers

for sentiment analysis (see Fig.1). The first layer is the
initialization layer where the data collection and pre-
processing of messages is performed. The second layer is
the learning layer where the pre-processed data will be
split into training (70% data) and test dataset (30% data).
The training dataset will be used to develop three different
models using supervised machine learning techniques
namely Naïve Bayes, Decision Tree and Support Vector
Machine.

Again, the run-time behavior of three trained models using

model-based testing techniques will be used to check the
model's predictions. Finally, in the third layer, comparison
of the performance of the models will be made on the
basis of the evaluative metrics , i.e. precision, recall,
Fig. 1 Proposed framework for sentiment analysis
macro average F1-score, polarity classification accuracy to
determine how the effects of the statistical analysis can be
generalized to an independent dataset. 3.1. Data Collection

The task of sentiment classification can usually be seen as For sentiment analysis on Twitter, varieties of benchmark
a two-class classification problem. This type of work datasets have been released over the last few years and are
mainly involves sentiment analysis as a text classification available online. We choose to select five different
problem, where feature selection has significant effect on datasets that have been widely used in Twitter sentiment
the performance of developed classifier models. analysis studies in literature. Table 2 provides a brief
overview of the datasets used in this paper.

105
IJCAT - International Journal of Computing and Technology, Volume 7, Issue 6, June 2020
ISSN (Online) : 2348-6090
www.IJCAT.org
Impact Factor: 0.835

• All URLs and special characters are removed

Table 2. Brief description of the datasets from the datasets.
No. of No. of Total no of Average • Uppercase letters are converted into lowercase.
Dataset Positive Negative Messages no. of
Message Message words
• Stop words like articles, prepositions,
conjunctions, and pronouns are filtered. Stop
IMDB 25,000 25000 50,000 255
words provide little or no information.
Sentiment 140 800,000 800,000 1,600,000 15 • Stemming is done where the inflected words are
SemEval-2013 2315 861 3176 23 reduced to their own word stem that affixes to
SemEval-2014 2509 932 3441 22 suffixes and prefixes.
• Many of the texts are broken down into tokens. It
STS-Gold 632 1402 2034 16
is a method called tokenization. It is mainly used
to eliminate the delimiters from the messages. For
The datasets used in this paper consist of example, Tweets datasets are rich data sources is
messages/reviews made in English on products, movies broken into individual tokens such as Tweets,
and general tweets made in Twitter. Tweets are extracted datasets, are, rich, sources, of, data. A token is
using Tweeter API and manually annotated by polarity detected on encountering a space.
(Positive, Negative, and Neutral).
3.3. Feature Extraction
The IMDB dataset is a large movie review dataset used for
binary sentiment collection. It consists of 50,000 polarized Feature extraction is the most important tasks in
movie review posts. classification functions. It includes removing meaningless
words or terms that do not convey any emotions. Unigram
The Sentiment 140 dataset consists of 1.6 million Twitter (n=1) and term frequency and inverse document frequency
messages collected using a distant supervision approach (TF-IDF) are the features extracted from the preprocessed
[17]. The general tweets in the dataset have been annotated datasets. The unigram features represents individual,
(0 = negative, 4 = positive) and the large collection of data distinct words. TF-IDF assigns a score for each term. The
in this dataset is very useful for the development of term-frequency is determined by counting the number of
machine learning models to be used to detect sentiment. times the word or phrase appears in the document and the
The SemEval-2013 dataset consists of comments taken inverse frequency of the document is measured by
from a variety of topics discussed on Twitter; a number of dividing the total number of documents by the number of
entities, items and events. documents containing the phrase.
The SemEval-2014 dataset [18] deals with product
reviews made by consumers on two entities or domains. 4. Machine Learning Algorithm
They’re laptops and restaurants. Both SemEval datasets
are not publicly accessible but can be downloaded from 4.1. Naïve Bayes Algorithm
the source first.
The Bayesian Classification is a supervised learning
The Stanford Twitter Sentiment Gold dataset (STS-Gold) system as well as a statistical classification method. It
[19] is a subset of tweets from the Stanford Twitter assumes the underlying probabilistic model and allows us
Sentiment Corpus. In the dataset, tweets and targets to capture uncertainty about the model in a principled
(entities) are annotated separately and therefore different manner by determining the probabilities of outcomes.
sentiment labels may be used to help the evaluation of Bayesian classification offers functional learning
sentiment classification models at both the entity and the algorithms and prior knowledge and data can be mixed.
tweet level. Bayesian Classification provides a valuable viewpoint for
recognizing and testing a variety of learning algorithms. It
3.2. Pre-processing of Data calculates explicit probabilities for hypotheses and is
resilient to noise in input data. Naive Bayes conditional
Pre-processing is needed when building machine learning independence assumption can be shown in equation 1.
systems based on tweet data. All tweets are normalized
during pre-processing. This included the following steps: length(doc)
• All targets (@username) are removed from the 𝑃(𝑑𝑜𝑐|𝑉𝑗 ) = ∏ 𝑃(𝑎𝑖 = 𝑊𝑘 |𝑉𝑗 ) (1)
data. 𝑖=1

106
IJCAT - International Journal of Computing and Technology, Volume 7, Issue 6, June 2020
ISSN (Online) : 2348-6090
www.IJCAT.org
Impact Factor: 0.835

(𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 ×𝑅𝑒𝑐𝑎𝑙𝑙)
Where, P(ai = wk | vj) is probability that word in position i 𝐹1 − 𝑆𝑐𝑜𝑟𝑒 = 2 × (𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛+𝑅𝑒𝑐𝑎𝑙𝑙) (6)
is wk, given vj .. One more assumption: P(ai = wk | vj) =
P(am =wk | vj) ∀ i, m.
Where, TP, FN, FP and TN refer respectively to the
number of true positive instances, the number of false
4.2. Support Vector Machine negative instances, the number of false positive instances
and the number of true negative instances. For sentiment
A Support Vector Machine (SVM) is a classifier that analysis on a tweet, high accuracy ensures that the
discriminates the data in numerous planes. For a given emotions of most messages in a dataset are evaluated and
supervised learning instance the SVM gives an optimal correctly predicted. The F1-score is a common metric for
hyper plane as output to categorize new data records. The both precision and recall. A receiver operating
SVM algorithms are implemented using a kernel. The characteristic (ROC) curve is a graphical plot that shows
kernel is responsible for translating input data to the the diagnostic capability of a binary classifier model as its
correct type. The SVM kernel takes input data of small discrimination threshold is varied. The ROC curve is
dimensions and transforms it into data space of higher developed by plotting the true positive rate (TPR) against
dimensions, thereby increasing data separability. This the false positive rate (FPR) at different threshold settings
strategy is called kernel tricks. We used the linear kernel Built classifier models are evaluated on the basis of
of the SVM, as it is mainly used to manage text data accuracy, macro-averaged F1-Scores of each model in five
because it includes many features and is usually linearly separate datasets (split as 70 percent training dataset and
separable. 30 percent testing dataset), and the ROC curves. The
results obtained are shown in Tables 3 and 4.
4.3. Decision Tree
Table 3. Classification accuracy (in %) of the models using the datasets
Decision Tree is a classification techniques in which the Dataset Naïve Bayes SVM ID3
divide-n-conquer process operates. The essential aspect of
IMDB 75.43 72.5 82.7
the decision tree is that it breaks down the complicated
decision-making process into a series of simpler decisions. Sentiment 140 73.18 75.5 81.94
In a tree where the root and the inner node are labeled with SemEval-2013 83.56 84.93 82.23
a query, and a leaf node is a prediction of an answer. We SemEval-2014 75.12 78.25 69.56
used an ID3 algorithm. It is a precursor to the C4.5
algorithm and is generally used in the area of machine STS-Gold 87.78 84.34 73.6
learning and natural language processing. For
classification of messages, information gain (see Equation Table 4. Macro-Averaged F1-Score of the classifier models using the
datasets
2) has been used as a split criterion.
Dataset Naïve Bayes SVM ID3
∆𝑦𝑖 IMDB 0.75 0.75 0.73
𝐺(𝑥, 𝑦) = 𝐻(𝑥) − ∑𝑖∈𝑣𝑎𝑙𝑢𝑒(𝑦) ∆𝑦 𝐻(𝑦𝑖 ) (2)
Sentiment 140 0.73 0.76 0.82
Where, H(x) is the entropy of the training set x with its SemEval-2013 0.85 0.862 0.72
attribute y. SemEval-2014 0.76 0.78 0.63
STS-Gold 0.862 0.827 0.682
5. Experimental Results
The plotted ROC curves of the three different classifier
The performance of sentiment classification can be models developed using the Sentiment 140 dataset are
evaluated by using the following metrics. shown (see Figures 2, 3, and 4). The reason for showing
(𝑇𝑃+𝑇𝑁)
the ROC curve only for Sentiment 140 dataset is because
𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 = (𝑇𝑃+𝑇𝑁+𝐹𝑃+𝐹𝑁) (3) of its large size which really helps in developing better
classifier models. Figures 2, 3, 4 reflect ROC curves of the
𝑇𝑃 Naïve Bayes classifier model, SVM and decision tree
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = (𝑇𝑃+𝐹𝑃) (4)
model using ID3 algorithm respectively.
𝑇𝑃
𝑅𝑒𝑐𝑎𝑙𝑙 = (5) The results shows a close competion between SVM and
(𝑇𝑃+𝐹𝑁)
ID3, as they are found to be better than Naïve Bayes

107
IJCAT - International Journal of Computing and Technology, Volume 7, Issue 6, June 2020
ISSN (Online) : 2348-6090
www.IJCAT.org
Impact Factor: 0.835

classifaction algorithm. However, Naïve Bayes performs Bayes and decision tree. We evaluated our models on the
better with the STS-Gold dataset. In case of SemEval- basis of metrics; classification accuracy, precision, recall,
2014, the best accuracy achieved is 78.25% which is quite F1-score, and ROC curve. After evaluating the developed
low. classifiers, we find that the results obtained from the
Decision Tree and SVM have a lower mean square error or
a higher accuracy with most of the datasets and are
considered to be good classifiers. We find our work to be
unique, as we have attempted in our study to provide an
overview of the various methods used in the sentiment
analysis of text data. We have also built and compared
three different classifiers using machine learning
techniques to five different datasets of varying sizes
and domains.

Future prospects in this area include the development of

techniques for aspect-based sentiment analysis, which will
Fig. 2 ROC curve of Naïve Bayes classifier model take into account the various aspects of the text. Social
networking data can be accessible in various languages,
making it an obstacle to sentiment analysis. The
development of a multilingual approach to sentiment
analysis is therefore a major challenge. Also, the main
drawback of using a machine learning approach to opinion
mining is that each opinion is treated as a single uniform
statement and assigns a sentiment score to the post as a
whole. Efficient methods must therefore be developed to
identify the subjects discussed in the message and to
convert each message into subject-level aspects.

References

[1] A. Mittal and S. Patidar, Sentiment Analysis on Twitter

Fig. 3 ROC curve of SVM classifier model Data: A Survey, In: Proceedings of the 7th International
Conference on Computer and Communications
Management, pp. 91--95, 2019.
[2] Megha Rathi, Aditya Malik, Daksh Varshney, Rachita
Sharma, Sarthak Mendiratta, Sentiment Analysis of
Tweets using Machine Learning Approach, In
proceeding of Eleventh International Conference on
Contemporary Computing (IC3), IEEE, 2018.
[3] Tejal Rathod , Mehul Barot, Trend Analysis on Twitter
for Predicting Public Opinion on Ongoing Events,
International Journal of Computer Applications, Vol.
180 (26), 2018.
[4] Suman Rani, Jaswinder Singh Sentiment Analysis Of
Tweets Using Support Vector Machine, International
Journal of Computer Science and Mobile Applications,
Fig. 4 ROC curve of decision tree classifier model Vol.5 (10), 2017.
[5] Y. Luo, W.Huang, Product Review Information
Extraction Based on Adjective Opinion Words, In:
6. Conclusion and Future Works Fourth International Joint Conference on Computational
Sciences and Optimization (CSO), pp.1309 – 1313,
In this paper, text data from product reviews, general 2011.
tweets and movie reviews are taken into account to assess [6] Aditya Kulkarni, Shubham Mhaske, Tweet Sentiment
the polarity (positive or negative) of messages or tweets. Analysis and Study and Comparison of Various
Approaches and Classification Algorithms Used,
We used the classification algorithms namely SVM, Naïve

108
IJCAT - International Journal of Computing and Technology, Volume 7, Issue 6, June 2020
ISSN (Online) : 2348-6090
www.IJCAT.org
Impact Factor: 0.835

International Research Journal of Engineering and [15] K. Zvarevashe, O.O. Olugbara, A framework for
Technology, Vol 7(4), 2020. sentiment analysis with opinion mining of hotel reviews.
[7] Apurva Dixit, Alok Kumar Pal, Shraddha Temghare and In: conference on Information Communications
Vikas Mapari, Emotion Detection Using Decision Tree, Technology and Society (ICTAS), pp. 1–4, 2018.
International Journal of Advance Engineering and [16] V. Ramanathan, T. Meyyappan, Twitter text mining for
Research Development. 2017. sentiment analysis on people's feedback about Oman
[8] B. Pang, L. Lee, S. Vaithyanathan, Thumbs up?: tourism, In: 4th MEC International Conference on Big
sentiment classification using machine learning Data and Smart City (ICBDSC) IEEE. pp. 1--5, 2019.
techniques. In: Proceedings of the ACL-02 conference [17] Alec Go, Richa Bhayani and Lei Huang, Twitter
on empirical methods in natural language processing, Sentiment Classification using Distant Supervision
vol 10, pp. 79-86. 2002. Stanford University Stanford, CS224N project report,
[9] Farman Ali, Kyung-Sup Kwak and Yong-Gi Kim, Stanford, 2009
Opinion mining based on fuzzy domain ontology and [18] S. Rosenthal, Semeval 2014 task 9 description
Support Vector Machine: A proposal to automate online http://alt.qcri.org/semeval2014/ task9/
review classification, Applied Soft Computing, 2016. [19] H. Saif, M. Fernandez, Y. He, and H. Alani, Evaluation
[10] M.K. Dalal, M.A. Zaveri, Semi supervised learning datasets for twitter sentiment analysis: a survey and a
based opinion summarization and classification for new dataset, the sts-gold, In: Proceedings, 1st Workshop
online product reviews, Applied Computational on Emotion and Sentiment in Social and Expressive
Intelligence and Soft Computing, pp. 1–8, 2013. Media (ESSEM), Turin, Italy, 2013.
[11] Rui Xia, Feng Xu, ,Jianfei Yu et.al., Polarity shift
detection, elimination and ensemble: A three stage Authors -
model for document-level sentiment analysis,
Information Processing and Management, vol 52(1), pp. Abhishek Bhagat is currently pursuing his BTech (CSE) from the
36– 45, 2016. Department of CSE, School of Technology, Assam Don Bosco
[12] Efstratios Kontopoulo, Christos Berberidis, Theologos University. Currently, he is in his final year. His areas of interest
Dergiades , Nick Bassiliades, Ontology- based sentiment are Machine Learning, Big data analytics and Natural Language
analysis of twitter posts, Expert Systems with Processing.
Akash Sharma is currently pursuing his BTech (CSE) from the
Applications
Department of CSE, School of Technology, Assam Don Bosco
vol. 40, pp.4065-4074, 2013. University. He is a final year student at present. His areas of
[13] P. Kalaivani, K. L. Shunmuganathan, Sentiment interest are Machine Learning, Internet of Things (IoT) and Natural
classification of movie reviews by supervised machine Language Processing (NLP).
learning approaches, Indian Journal of Computer Sci.
Eng. Vol 4, pp.285–292, 2013. Dr. Sarat Kr. Chettri is an Assistant Professor in the Department
[14] T. Hassan, A. Soliman, M.A. Elmasry, A.R. Hedar, of Computer Applications, School of Technology, Assam Don
M.M. Doss, Utilizing Support Vector Machines in Bosco University. He has made several publications in
mining online customer reviews, In: Proceedings of international journals and conferences. His research area includes
22nd International Conference on Computer Theory and data science, machine learning and Internet of Things (IoT).
Applications (ICCTA), pp.192—196, 2012.

109