International Journal of Scientific and Research Publications, Volume 3, Issue 6, June 2013 1

ISSN 2250-3153

Machine Learning Algorithms for Opinion Mining and

Sentiment Classification
Jayashri Khairnar*, Mayura Kinikar**
*
Department of Computer Engineering, Pune University, MIT Academy of Engineering, Pune
**
Department of Computer Engineering, Pune University, MIT Academy of Engineering, Pune

Abstract- With the evolution of web technology, there is a huge extraction of knowledge from the opinion of others about some
amount of data present in the web for the internet users. These particular topic or problem. This paper will try to focus on the
users not only use the available resources in the web, but also basic definitions of Opinion Mining, analysis of linguistic
give their feedback, thus generating additional useful resources required for Opinion Mining, few machine learning
information. Due to overwhelming amount of users opinions, techniques on the basis of their usage and importance for the
views, feedback and suggestions available through the web analysis, evaluation of Sentiment classifications.
resources, its very much essential to explore, analyse and Current-day Opinion Mining and Sentiment Analysis is a
organize their views for better decision making. Opinion Mining field of study at the crossroad of Information Retrieval (IR) and
or Sentiment Analysis is a Natural Language Processing and Natural Language Processing (NLP) and share some
Information Extraction task that identifies the users views or characteristics with other disciplines such as text mining and
opinions explained in the form of positive, negative or neutral Information Extraction. Opinion mining is a technique to detect
comments and quotes underlying the text. Various supervised or and extract subjective information in text documents. In general,
data-driven techniques to Sentiment analysis like Nave Byes, sentiment analysis tries to determine the sentiment of a writer
Maximum Entropy and SVM.For classification use support about some aspect or the overall contextual polarity of a
vector machine (SVM), it performs the sentiment classification document. The sentiment may be his or her judgment, mood or
task also consider sentiment classification accuracy. evaluation. A key problem in this area is sentiment classification,
where a document is labeled as a positive or negative evaluation
Index Terms- Text mining, support vector machine (SVM), of a target object (film, book, product etc.).In recent years, the
Sentiment Classification, Feature extraction, opinion mining. problem of opinion mining has seen increasing attention.
Sentiment classication is a recent subdiscipline of text
classication which is concerned not with the topic a document is
I. INTRODUCTION about, but with the opinion it expresses. Sentiment classication
also goes under dierent names, among which opinion mining,
T ext mining offers a way for individuals and corporations to
exploit the vast amount of information available on the
Internet. In current search engine people to search for other
sentiment analysis, sentiment extraction, or aective rating.

peoples opinions from the Internet before purchasing a product

or seeing a movie because practically, when we are not familiar II. SENTIMENT ANALYSIS
with a specific product, we ask our trusted sources to recommend Sentiment analysis of natural language texts is a large and
one [6]. Many website provide user rating and commenting growing field. Sentiment analysis or Opinion Mining is the
services, and these reviews could reflect users opinions about a computational treatment of opinions, sentiments and subjectivity
product. With the propagation of reviews, ratings, of text. Sentiment analysis is a Natural Language Processing and
recommendations, and other forms of online expression, online Information Extraction task that aims to obtain writers feelings
opinion could present essential information for businesses to expressed in positive or negative comments, questions and
market their products and manage their reputations. Current requests, by analyzing a large numbers of documents. Converting
search engines can efficiently help users obtain a result set, a piece of text to a feature vector is the basic step in any data
which is relevant to users query. However, the semantic driven approach to Sentiment analysis. Term frequency has
orientation of the content, which is very important information in always been considered essential in traditional Information
the reviews or opinions, is not provided in the current search Retrieval and Text Classification tasks. But Pang-Lee [1] found
engine. For example, Google will return around 7 380 000 hits that term presence is more important to Sentiment analysis than
for the query Angels and Demons review. If search engines term frequency. That is, binary-valued feature vectors in which
can provide statistical summaries from the semantic orientations, the entries merely indicate whether a term occurs (value 1) or not
it will be more useful to the user who polls the opinions from the (value 0).It also reported that unigrams outperform bigrams when
Internet. A scenario for the aforementioned movie query may classifying movie reviews by sentiment polarity. As a result, the
yield such report as There are 10 000 hits, of which 80% are sentiment analysis research from the determination of the
thumbs up and 20% are thumbs down. This type of service semantic orientation of the terms. Determining semantic
requires the capability of discovering the positive reviews and orientation of words Hatzivassiloglou and McKeown [8]
negative reviews.Opinion Mining is a process of automatic hypothesize that adjectives separated by and" have the same

www.ijsrp.org
International Journal of Scientific and Research Publications, Volume 3, Issue 6, June 2013 2
ISSN 2250-3153

polarity, while those separated by but have opposite polarity III. MACHINE LEARNING APPROACHES
.Starting with small seed lists, this information is used to group The aim of Machine Learning is to develop an algorithm so
adjectives into two clusters such that maximum constraints are as to optimize the performance of the system using example data
satisfied. Sentiment classication is a recent sub discipline of text or past experience. The Machine Learning provides a solution to
classication which is concerned not with the topic a document is the classification problem that involves two steps:
about, but with the opinion it expresses. Functional to the 1) Learning the model from a corpus of training data
extraction of opinions from text is the determination of the 2) Classifying the unseen data based on the trained model.
orientation of subjective terms contained in text, i.e. the In general, classification tasks are often divided into several sub-
determination of whether a term that carries opinionated content tasks:
has a positive or a negative connotation [2]. Esuli and Sebastiani 1) Data preprocessing
proposed new method for determining the orientation of 2) Feature selection and/or feature reduction
subjective terms. The method is based on the quantitative 3) Representation
analysis of the glosses of such terms, i.e. the denitions that these 4) Classification
terms are given in online dictionaries, and on the use of the 5) Post processing
resulting term representations for semi-supervised term
classication. Sentiment classication can be divided into several Feature selection and feature reduction attempt to reduce the
specic subtasks: determining subjectivity, determining dimensionality (i.e. the number of features) for the remaining
orientation, determining the strength of orientation [2]. Esuli and steps of the task. The classification phase of the process finds the
Sebastiani [4] described SENTIWORDNET, which is a lexical actual mapping between patterns and labels (or targets). Active
resource in which each WordNet synset is associated with three learning, a kind of machine learning is a promising way for
numerical scores, i.e., Obj(s), Pos(s), and Neg(s), thus describing sentiment classification to reduce the annotation cost. The
how objective, positive, and negative the terms contained in the following are some of the Machine Learning approaches
synset. commonly used for Sentiment Classification [10].
Traditionally, sentiment classication can be regarded as a
binary-classication task [1], [5].Dave,Lawrence,Pennock [5] 4.1 Naive Bayes Classification
use structured reviews for testing and training, identifying It is an approach to text classification that assigns the class
appropriate features and scoring methods from information c* = argmaxc P(c | d), to a given document d. A naive Bayes
retrieval for determining whether reviews are positive or classifier is a simple probabilistic classifier based on Bayes'
negative. These results perform as well as traditional machine theorem and is particularly suited when the dimensionality of the
learning method then use the classier to identify and classify inputs are high. Its underlying probability model can be
review sentences from the web, where classication is more described as an "independent feature model". The Naive Bayes
difcult. Various supervised or data-driven techniques to (NB) classifier uses the Bayes rule Eq. (1),
Sentiment analysis like Nave Byes, Maximum Entropy and
SVM. Pang Lee [1] compared the performance of Nave Bayes,
Maximum Entropy and Support Vector Machines in Sentiment
analysis on different features like considering only unigrams,
Where, P (d) plays no role in selecting c*. To estimate the
bigrams, combination of both, incorporating parts of speech and
term P(d|c), Naive Bayes decomposes it by assuming the fis are
position information, taking only adjectives etc. It is observed
conditionally independent given ds class as in Eq.(2),
from the results that:
a. Feature presence is more important than feature
frequency.
b. Using Bigrams the accuracy actually falls.
Where, m is the no of features and fi is the feature vector.
c. Accuracy improves if all the frequently occurring words
Consider a training method consisting of a relative-frequency
from all parts of speech are taken, not only Adjectives.
estimation P(c) and P (fi | c). Despite its simplicity and the fact
d. Incorporating position information increases accuracy.
that its conditional independence assumption clearly does not
e. When the feature space is small, Nave Bayes performs
hold in real-world situations, Naive Bayes-based text
better than SVM. But SVMs perform better when feature space
categorization still tends to perform surprisingly well; indeed,
is increased.
Naive Bayes is optimal for certain problem classes with highly
According to their experiment, SVMs tended to do the best,
dependent features.
and unigram with presence information turns out to be the most
effective feature. In recent years, some researchers have
4.2 Maximum Entropy
extended sentiment analysis to the ranking problem, where the
Maximum Entropy (ME) classification is yet another
goal is to assess review polarity on a multipoint scale. Goldberg
technique, which has proven effective in a number of natural
and Zhu [7] proposed a graph-based semi supervised learning
language processing applications. Sometimes, it outperforms
algorithm to address the sentiment-analysis task of rating
Naive Bayes at standard text classification. Its estimate Of P(c |
inference and their experiments showed that considering
d) takes the exponential form as in Eq. (3) [10],
unlabeled reviews in the learning process can improve rating
inference performance.

www.ijsrp.org
International Journal of Scientific and Research Publications, Volume 3, Issue 6, June 2013 3
ISSN 2250-3153

Where, Z (d) is a normalization function. Fi, c is a

feature/class function for feature fi and class c, as in Eq. (4),

For instance, a particular feature/class function might fire if

and only if the bigram still hate appears and the documents
sentiment is hypothesized to be negative. Importantly, unlike
Naive Bayes, Maximum Entropy makes no assumptions about
the relationships between features and so might potentially
perform better when conditional independence assumptions are
not met.

4.3 Support Vector Machines

Support vector machines (SVMs) have been shown to be Figure 1: Different boundary decisions are possible to
highly effective at traditional text categorization, generally separate two classes in two dimensions. Each boundary has
outperforming Naive Bayes. They are large-margin, rather than an associated margin.
probabilistic, classifiers, in contrast to Naive Bayes and
Maximum Entropy. In the two-category case, the basic idea What SVM is used for?
behind the training procedure is to find a maximum margin SVM is primarily used for categorization. Some examples
hyperplane, represented by vector w, that not only separates the of SVM usage include bioinformatics, signature/hand writing
document vectors in one class from those in the other, but for recognition, image and text classification, pattern recognition,
which the separation, or margin, is as large as possible. This and e-mail spam categorization. Many research documents such
corresponds to a constrained optimization problem; letting cj as the ones mentioned above have shown that SVM can classify
{1, 1} (corresponding to positive and negative) be the correct reasonably well. In this project, SVM is used for text
class of document dj, the solution can be written as in Eq. (5) classification. Text classification is a method used to put text into
[10], meaningful groups. Besides SVM, there are many other methods
for text classification such as Bayes and k-Nearest Neighbor.
Based on many research papers (Joachims, T., 1998), SVM
Where, the js are obtained by solving a dual optimization outperforms many, if not all, popular methods for text
problem. That dj such that j is greater than zero are called classification. The studies also show that SVM is effective,
support vectors, since they are the only document vectors accurate, and can work well with small amount of training data
contributing to w. Classification of test instances consists simply [12].
of determining which side of ws hyperplane they fall on.
Support vector machines were introduced in [3] (Vapnik) How SVM Works
and basically attempt to nd the best possible surface to separate The idea for SVM is to find a boundary (known as a
positive and negative training samples. Support Vector Machines hyperplane) or boundaries that separate clusters of data. SVM
(SVMs) are supervised learning methods used for classification. does this by taking a set of points and separating those points
In this project, SVM is used for sentiment classification. First using mathematical formulas. The following figure illustrates
module is sentiment analysis and Support vector machines the data flow of SVM.
perform sentiment classification task on review data. The goal of
a Support Vector Machine (SVM) classifier is to find a linear
hyperplane (decision boundary) that separates the data in such a
way that the margin is maximized. Look at a two class separation
problem in two dimensions like the one illustrated in figure 1,
observe that there are many possible boundary lines to separate
the two classes. Each boundary has an associated margin. The
rationale behind SVMs is that if we choose the one that
maximizes the margin we are less likely to misclassify unknown
items in the future.

Figure 2: SVM Process Flow

In Figure 2, data are input in an input space that cannot be

separated with a linear hyperplane. To separate the data linearly,
points are map to a feature space using a kernel method. Once
the data in the feature space are separate, the linear hyperplane
gets map back to the input space and it is shown as a curvy non-

www.ijsrp.org
International Journal of Scientific and Research Publications, Volume 3, Issue 6, June 2013 4
ISSN 2250-3153

linear hyperplane. This process is what makes SVM amazing.

The SVMs algorithm first starts learning from data that has
already been classified, which is represented in numerical labels
(e.g. 1, 2, 3, etc.) with each number representing a category.
SVM then groups the data with the same label in each convex
hull. From there, it determines where the hyperplane is by
calculating the closest points between the convex hulls (Bennett,
K. P., & Campbell, C., 2000). Once SVM determines the points
that are closest to each other, it calculates the hyperplane, which
is a plane that separates the labels.

Simple SVM Example

Let us use a few simple points to illustrate the concept of
Figure 4: Simple Data in a Feature
SVM. The following example is similar to Dr. Guestrins lecture
(Guestrin, C., 2006). Given the following points with The next step is finding a hyperplane
corresponding classes (labels) in Figure 3, find a hyperplane that
w x + b = +1 (positive labels) (1)
separated the points [12].
w x + b = -1 (negative labels) (2)
Table 1: Simple Data in 1-Dimension
w x + b = 0 (hyperplane) (3)

From these equations, find the unknowns, w and b.

Expanding the equations for the SVM problem will get:
w1x1 + w2x2 + b = +1
w1x1 + w2x2 + b = -1
w1x1 + w2x2 + b = 0
Solve w and b for the positive labels using equation, w1x1 +
w2x2 + b = +1.
w1x1 + w2x2 + b = +1
w10 + w20 + b = +1
w13 + w29 + b = +1
Solve w and b for the negative labels using equation, w1x1
+ w2x2 + b = -1.
w1x1 + w2x2 + b = -1
w11 + w21 + b = -1
Figure 3: Simple Data in an Input Space w12 + w24 + b = -1

As Figure 3 shows, these points lay on a 1-dimensional By using linear algebra, we find that the solution is w1 = -3,
plane and cannot be separated by a linear hyperplane. The first w2 = 1, b = 1, which satisfies the above equations. Many times,
step is to find a kernel that maps the points into the feature space, there is more than one solution or there may be no solution, but
then within the feature space, find a hyperplane that separates the SVM can find the optimal solution that returns a hyperplane with
points. A simple kernel that would do the trick is (X1) = (X1, the largest margin. With the solutions: w1 = -3, w2 = 1, b = 1,
X12). This kernel is actually a polynomial type. As the reader positive plane, negative plane, and hyperplane can be calculated.
sees, this kernel will map the points to a 2-dimensional feature
space by multiplying the points to the power of 2. From Table 3: Calculation Results of Positive, Negative, and
calculating the kernels, we get (0, 0, +1), (1, 1, -1), (2, 4, -1), (3, Hyperplane
9, +1) [12].

Table 2: Simple Data in 2-Dimension

www.ijsrp.org
International Journal of Scientific and Research Publications, Volume 3, Issue 6, June 2013 5
ISSN 2250-3153

Figure 5: Simple Data in a Feature Space Separated by a

Hyperplane
Accuracy is the portion of all true predicted instances
Thus, we have the model that contains the solution for w against all predicted instances. An accuracy of 100% means that
and b and with margin 2/ (w w). The margin is calculated as the predicted instances are exactly the same as the actual
follow. instances. Precision is the portion of true positive predicted
2/ (w w) (4) instances against all positive predicted instances. Recall is the
2/ (-32 + 12) margin = 0.632456 portion of true positive predicted instances against all actual
positive instances. F1 is a harmonic average of precision and
In SVM, this model is used to classify new data. With the recall.
solutions, new data can be classified into category. For example,
if the result is less than or equal -1, the new data belongs to the -1
class and if the result is greater than or equal to +1, the new data V. CONCLUSION
belongs to the +1 class. Some of the machine learning techniques like Nave Bayes,
LIBSVM is a well-known library for SVM that is developed Maximum Entropy and Support Vector Machines has been
by Chih-Chung Chang and Chih-Jen Lin. LIBSVM is a library discussed. Many of the applications of Opinion Mining are based
for Support Vector Machines (SVMs).LIBSVM is an integrated on bag-of-words, which do not capture context which is essential
software for support vector classification, (C-SVC, nu-SVC), for Sentiment Analysis. The recent developments in Sentiment
regression (epsilon-SVR, nu-SVR) and distribution estimation Analysis and its related sub- tasks are also presented. The state of
(one-class SVM) [9]. It supports multi-class classification. the art of existing approaches has been described with the focus
LIBSVM involves two steps: rst, training a data set to obtain a on the Sentiment Classification using various Machine learning
model and second, using the model to predict information of a techniques. This paper introduced and surveyed the field of
testing data set. SVM procedure includes Transform data to the sentiment analysis and opinion mining. It has been a very active
format of an SVM package, Conduct simple scaling on the data, research area in recent years. In fact, it has spread from computer
Select model here use linear formula, Use cross-validation to nd science to management science. Finally, this paper concludes
the best parameter, Use the best parameter to train the whole saying that all the sentiment analysis tasks are very challenging.
training set and Test. The concept of SVM is explained through a small set of data in a
2-dimenional feature space. With the use of kernel methods,
SVM can classify data in high dimensional space. SVM is an
IV. EVALUATION OF SENTIMENT CLASSIFICATION excellent method for data classification. Finally, the future
In general, the performance of sentiment classification is challenges and directions so as to further enhance the research in
evaluated by using four indexes. They are Accuracy, Precision, the field of Opinion Mining and Sentiment Classification are
Recall and F1-score [11]. The common way for computing these discussed.
indexes is based on the confusion matrix as shown below:

Table 4: Confusion Matrix REFERENCES

[1] B. Pang, L. Lee, and S. Vaithyanathan,Thumbs up?:Sentiment
classification using machine learning techniques,inProc.ACL-
02Conf.Empirical Methods Natural Lang. Process., 2002, pp. 7986.
[2] A. Esuli and F. Sebastiani, Determining the semantic orientation of terms
through gloss classification, in Proc. 14th ACM Int. Conf. Inf.
Knowl.Manage., 2005, pp. 617624.
[3] V. N. Vapnik, The Nature of Statistical Learning Theory.New
York:Springer-Verlag, 1995.
[4] A. Esuli and F. Sebastiani, SENTIWORDNET: A publicly available
lexical resource for opinion mining, in Proc. 5th Conf. Lang. Res. Eval.,
These indexes can be defined by the following equations: 2006, pp. 417422.

www.ijsrp.org
International Journal of Scientific and Research Publications, Volume 3, Issue 6, June 2013 6
ISSN 2250-3153

[5] K. Dave, S. Lawrence, and D. M. Pennock, Mining the peanut

gallery:opinion extraction and semantic classification of product reviews,
in Proc. 12th Int. Conf. World Wide Web, New York: ACM, 2003, pp.
519528. AUTHORS
[6] Chien-Liang Liu, Wen-Hoar Hsaio, Chia-Hoang Lee, Gen-Chi Lu, and First Author Jayashri Khairnar received her
Emery Jou, Movie Rating and Review Summarization in Mobile
Environment,IEEE VOL. 42, NO. 3, MAY 2012
Bachelors degree in Information Technology
[7] A. B. Goldberg and X. Zhu, Seeing stars when there arent many
.Now; she is pursuing her M.E degree in
stars:Graph-based semi-supervised learning for sentiment categorization, Computer Engineering from MIT Academy of
in Proc. TextGraphs: First Workshop Graph Based Methods Nat. Lang. Engineering, Pune University, Pune, India. Her
Process, Morristown, NJ: Assoc. Comput. Linguist. 2006, pp. 4552. research areas are Data mining and Text mining.
[8] V. Hatzivassiloglou and K. R. McKeown, Predicting the semantic Email- [email protected]
orientation of adjectives, in Proc. 8th Conf. Eur. Chap. Assoc.
Comput.Linguist., Morristown, NJ: Assoc. Comput. Linguist, 1997, pp. Second Author Prof. Mayura Kinikar, B.E.,
174181. M.E. Computer was educated at Doctor
[9] (2001). LIBSVM: A library for support vector machines [Online]. Babasaheb Ambedkar Marathwada University.
Available: http://www.csie.ntu.edu.tw/ cjlin/libsvm.
Now, she is pursuing her PhD. She has worked
[10] S. ChandraKala and C. Sindhu ISSN: 2229-6956(ONLINE) ICTACT
JOURNAL ON SOFT COMPUTING, OCTOBER 2012, VOLUME: 03,
in various capacities in academic institutions.
ISSUE: 01 OPINION MINING AND SENTIMENT CLASSIFICATION: Now she is Assistant Prof in MIT Academy of
A SURVEY. \ Engineering, Alandi, Pune. Her areas of interest
[11] International Journal of Ad hoc, Sensor & Ubiquitous Computing (IJASUC) include Data mining, text mining, web mining
Vol.4, No.1, February 2013, Opinion Mining and Sentiment Analysis An and warehousing.
Assessment of Peoples Belief: A SurveyS Padmaja and Prof. S Sameen
Fatima. [email protected].
[12] Clustering High Dimensional Data Using SVMTam P. Ngo,December
2006.

www.ijsrp.org