2018 4th IEEE International Conference on Information Management

Fake Reviews Detection Based on LDA

Shaohua Jia Xianguo Zhang, Xinyue Wang, Yang Liu

College of Computer Science College of Computer Science
Inner Mongolia University Inner Mongolia University
Hohhot,China Hohhot,China
e-mail: [email protected] e-mail: [email protected]
[email protected]
[email protected]

Abstract—It is necessary for potential consume to make demonstrate the validity of our method through experiments.
decision based on online reviews. However, its usefulness The rest of the paper is organized as follows: in Section 2
brings forth a curse ‒ deceptive opinion spam. The deceptive we summarize related work; in Section 3 we discuss our
opinion spam mislead potential customers and organizations dataset, features and classifiers; we show the results and
reshaping their businesses and prevent opinion-mining discussion in Section 4; finally, conclusion and future work are
techniques from reaching accurate conclusions. Thus, the given in Section 5.
detection of fake reviews has become more and more fervent.
In this work, we attempt to find out how to distinguish II. RELATED WORKS
between fake reviews and non-fake reviews by using
linguistic features in terms of Yelp Filter Dataset. To our There are many significant study on how to classify
surprise, the linguistic features performed well. Further, we authentic and fictitious reviews. Ott Collected 800 deceptive
proposed a method to extract features based on Latent opinions via Mechanical Turk and 800 truthful opinions from
Dirichlet Allocation. The result of experiment proved that TripAdvisor, then Integrating work from psychology and
the method is effective. computational linguistics, they develop and compare three
approaches to detecting deceptive opinion spam, and
Keywords-Review detection; Linguistic features; Latent ultimately develop an admirable classifier that is nearly 90%
Dirichlet Allocation accurate on their gold-standard opinion spam dataset [2, 3]. It
is admirable for Ott to open their gold-standard opinion spam
I. INTRODUCTION dataset, which have make a great impact on the field of fake
With the dramatically increasing of online reviews, reviews detection. Nitin Jindal and Bing Liu deals with a
the review spam come along due to the fact that there is restricted problem, identifying unusual review patterns which
no control, anyone can write anything on the web [1]. The can represent suspicious behaviors of reviewers [4]. Snehasish
review that describe authentic post-purchase experience Banerjee and Alton YK Chua extract linguistic features to
can help potential consume get a satisfactory commodity, distinguish fake reviews by word n-gram, psycholinguistic
business have its own accurate positioning. Instead, deception words, part-of-speech distributions, readability of
review spam misleads consume and business. Thus, reviews and review writing style [5]. Heydari focuses on
detection of review spam has become increasingly urgent systematically analyzing and categorizing models that detect
and important. review spam [6]. Michael Crawford mainly provide a strong
There are generally three types of spam reviews: and comprehensive comparative study of current research on
Type 1: untruthful opinions (also known as fake reviews). detecting review spam using various machine learning
Type 2: reviews on brands onlyˈType 3: Non-reviews [1]. techniques [7]. Interestingly, Lim, P and Liu, B proposed
In this paper, we aim to detect deceptive fake reviews by ranking and supervised methods to discover spammers and
looking into deep-level semantics of reviews. Our goal is outperform other baseline method based on helpfulness votes
then to cast the deceptive fake review detection problem along [8].
into binary classification task and build classification In terms of Yelp Filter Dataset, Mengqi Yu found sentiment
model. Using term frequency, LDA, word2vec to extract features are very useful for rating prediction [9]. Dao Runa
features, then we fed those kinds of features extracted treated the fake review detection problem as binary
from each review of our dataset into several Machine classification task and built classification models by extracting
Learning models for classification and finally compare semantic based features and relational based features with
the performances of features in those Machine Learning several data mining techniques [10]. Boya Yu use a Support
models. Vector Machine model to decipher the sentiment tendency of
We perform our experiment in Yelp dataset. As each review from word frequency. Word scores generated
Yelp.com is a well-known large-scale online review site from the SVM models are further processed into a polarity
that filters fake or suspicious reviews which can be used index indicating the significance of each word for special
as fake reviews in our experiment. In the end, we types of restaurant [11].

978-1-5386-6147-5/18/$31.00 ©2018 IEEE 280

 Arjun Mukherjee used linguistic features and and Non-fake reviews is twice as much as the fake reviews.
behavioral features to train classification model, the result New Dataset statistics are shown in Table III.
of linguistic features are shown in Table I, which consists
of hotel and restaurant. Because the low accuracy of B. Feature Extraction
linguistic features, the author paid attention to behavioral In order to classify the reviews either fake or non-fake, we
features, and yield a respectable accuracy in [12]. In this needed a set of features that can distinguish them. In our work,
paper, we will further research linguistic features on Yelp we mainly use linguistic feature, which respectively aims to
Filter Dataset based on the experiment conducted by term frequency, Latent Dirichlet Allocation and word2vec,
Arjun Mukherjee. then merged into one model to conduct experiment, then
We will use linguistic features to train classification selecting the best method.
model based on Yelp Filter Dataset by conducting three
Followings are the details of extracted features:
classification model, using term frequency, LDA,
word2vec to extract features. Then making contrastive 1) Term frequency
study with experiments conducted by Arjun Mukherjee in Term frequency can be used to measure the importance of
linguistic features [12]. a word in a paper. In this paper, we extract term frequency by
using Scikit-learn, and adopt 5000 words of the highest term
III. DESIGN frequency.
2) Word2vec
A. Data
Mikolov proposed word2vec, a model generates word
Obtain gold standard dataset for detecting fake review embedding for semantic modeling [13].We train a skip-gram
is always a challenging problem. With deceptive and model on our corpus by gensim.models.word2vec.Word2Vec
disguised characteristics, the fake reviews are hard to be and represent every review as a vector by calculating the
identified just by looking at individual review text. In average of the embedding vectors of each word in the review.
addition to large scale number of reviews online, manual The vector size is set to 300, the window size is 5, the
labeling is hard for ground truth reviews. min-count is 2, and the iteration is 5.
We use the subset of Yelp dataset in [12]. Yelp.com is 3) Latent Topic Distribution [14]
a well-known large-scale online review site that filters Topic modeling is a technique in natural language
fake or suspicious reviews which can be used as fake processing and tries to extract hidden topics from a collection
reviews in our work without manual labeling. Our dataset of documents. In our work, we treated fake and non-fake
contains 64195 reviews across 85 hotels and 130 reviews as two document and used LDA model to extract
restaurants in the Chicago area. For each review there are topic-words. We decided the number of topics by minimizing
information about <date, reviewID, reviewerID, the model’s perplexity on held-out data, the result showed that
reviewContent, rating, usefulCount, coolCount, the best number of topics chosen from 100, 150, 200, 250 and
funnyCount, flagged, restaurantID> (in which flagged are 300 for fake reviews was 150, non-fake reviews was 200, each
shown as yes or no that represent the reviews are fake or topic contain 30 words, there will show respectively 5 topics
non-fake). Dataset statistics are shown in Table II. and each topic contains 8 words in Table IV and Table V. Each
But the class distribution is extremely imbalanced. So topic-words is viewed as a new review, the LDA was
the review processed basically will be abandoned that implemented by Liu Yang [15], then integrating new reviews
review’s length below 25. Then selected all fake reviews with old reviews. Finally we have a distorted data shown in
Table VI.
TABLE I. THE RESULTS OF EXPERIMENT [12]

P R F1 A P R F1 A
Features
hotel restaurant

Word unigrams (WU) 62.9 76.6 68.9 65.6 64.3 76.3 69.7 66.9

WU + IG (top 1%) 61.7 76.4 68.4 64.4 64 75.9 69.4 66.2

WU + IG (top 2%) 62.4 76.7 68.8 64.9 64.1 76.1 69.5 66.5

Word-Bigrams (WB) 61.1 79.9 69.2 64.4 64.5 79.3 71.1 67.8

WB + LIWC 61.6 69.1 69.1 64.4 64.6 79.4 71 67.8

POS Unigrams 56 69.8 62.1 57.2 59.5 70.3 64.5 55.6

WB + POS Bigrams 63.2 73.4 67.9 64.6 65.1 72.4 68.6 68.1

WB + Deep Syntax 62.3 74.1 67.7 64.1 65.8 73.8 69.6 67.6

WB + POS Seq. Pat 63.4 74.5 68.5 64.5 66.2 74.2 69.9 67.7

281
 B. Discussion
TABLE II. DATASET STATISTIC
LDA can extract topic-words from one document, and
to some extent, topic-words can represent whole document.
Domain fake Non-fake %fake total
Thus, we use LDA to respectively extract topic-words from
fake reviews and non-fake reviews, it is more reflected the
Hotel and
802 4876 14.1% 5678 features of fake or non-fake reviews. Then when we counts
Restaurant the term frequency of each word, the import words to
8368 50114 14.3% 58517 reflect the features of fake or non-fake reviews will have a
higher term frequency, and then increase the accuracy of
TABLE III. NEW DATASET STATISTIC classification models. But due to the quantity of data is
enormous, the quantity of topic-words is far less than that.
Domain fake Non-fake %fake total Therefore, the accuracy with LDA slightly higher than the
Hotel and accuracy without LDA.
4017 8034 1/3 12051
restaurant
V. CONCLUSION AND FUTURE WORK
C. Classification and Evaluation This paper performed a linguistic investigation of the
Features from the two approaches just introduced are nature fake reviews in the commercial setting of Yelp.com.
used to train Support Vector Machine and Logistic Our study shows that linguistic features yielded a
Regression and Multi-layer Perceptron classifiersDŽ respectable 81.3% accuracy, which obviously higher than
The classification results of above mentioned techniques the 68.1% accuracy reported by Arjun Mukherjee on the
are evaluated by accuracy, precision, recall and F-measure. Yelp Filter Dataset as far as linguistic features linguistic
features [12]. Meanwhile, the study proved the
IV. RESULTS AND DISCUSSION effectiveness of features extracted based on LDA.
A. Results Possible directions for future work is to explore why
Logistic Regression and Multi-layer Perceptron have a high
In our experiment, we train SVM, Logistic Regression, accuracy, SVM in not. There is a hypothesis that sigmoid
and Multi-layer Perceptron models in Python 3.6. make a decisive influence, which will be testified in future
We choose 80% of the dataset as training data and 20% work.
as testing data. As we can see from the experimental results
in Table VII. TABLE IV. TOPIC-WORDS OF FAKE REVIEWS
Then compared the experimental results with the result Topic1 Topic2 Topic3 Topic4 Topic5
in [12]. The result of compare will be showed in Fig. 1. promise park stones writing cube
The difference between with LDA processing data and
quality comments discarded reserve parings
without LDA will be showed in Fig. 2.
pushy ramps split injure shined
Table VII shows that the results by using SVM yielded
accuracy of 65.7%, LDA+ SVM yielded a maximum rationalize edge eavesdrop damn pomp
accuracy of 67.9%, which slightly lower than the 68.1% podium cliff strict autographed bamboo
accuracy reported by Arjun Mukherjee on the Yelp Filter decorated spray breadth hate heroin
Dataset [12]. But LDA+ Logistic Regression yielded a peeled shots settle zealand absurd
maximum accuracy of 81.3%, which obviously higher than gulped care swirling olfactory unsalted
the 68.1% accuracy, and LDA+ Multi-layer Perceptron also
yielded a maximum accuracy of 81.3%. The accuracy of TABLE V. TOPIC-WORDS OF NON-FAKE REVIEWS
LDA+ Logistic Regression keep up to LDA + Multi-layer
Perceptron’s, but the F1-score of LDA+ Logistic Topic1 Topic2 Topic3 Topic4 Topic5
Regression slightly higher than the 71.1% F1-score of extremely decadent confirmed prospect collective
LDA+ Multi-layer Perceptron in fake reviews. burnt entertain duke eaten smiled
In terms of Fig. 1, the green line represent LDA+ vaguely hiccup warm previous cultural
Logistic Regression results in this paper, red and gray line arrives successor pour night mystery
respective represent the hotel and restaurant’s best results in content troubles laugh dish smothering
[12]. We can obviously notice the green line far higher than unstuck mustards transmogrify completely observing
red and gray line, which indicates that method of this paper
twists brighter care recognizable kindle
has a good effectiveness in binary classification task.
redefining responds school notable tire
In terms of Fig. 2, red line represent the accuracy by
using LDA, green line represent the accuracy without LDA.
TABLE VI. EXPERIMENT DATA
We can notice that the accuracy with LDA slightly higher
than the accuracy without LDA, which indicate the Domain fake Non-fake %fake total
effectiveness of LDA in this experiment. Hotel and
4167 8234 33.6% 12401
restaurant

282
 TABLE VII. EXPERIMENTAL RESULTS

Accuracy Non-fake Fake

Classifier Approach
(%) P(%) R(%) F(%) P(%) R(%) F(%)
SVM 65.7 65.7 100 79.3 1 2.6 5.1
LDA+ SVM 67.9 67.6 99.9 81.6 96.7 3.5 6.8
SVM
Word2Vec+SVM 61.3 65.5 100 79.2 \ 0 \
LDA+Word2Vec+SVM 61.3 65.7 100 79.3 \ 0 \
Logistic Regression 80.5 83.9 86.8 85.3 73.6 70 71.8

Logistic LDA+ Logistic Regression 81.3 85.2 87.2 86.2 73 69.5 71.2
Regression Word2Vec+Logistic Regression 65.1 65.1 1 78.9 \ 0 \
LDA+Word2Vec+Logistic Regression 65.8 66.8 1 80.1 \ 0 \
Multi-layer Perceptron 80.3 84 86.2 85.1 72.9 69.4 71.1

Multi-layer LDA+ Multi-layer Perceptron 81.3 85 87.4 86.2 73.2 69.1 71.1
Perceptron Word2Vec+Multi-layer Perceptron 65.5 65.5 1 79.2 \ 0 \
LDA+Word2Vec+Multi-layer Perceptron 65.7 65.7 1 79.3 \ 0 \

patterns using unexpected rules”, Proceedings of the 19th ACM

international conference on Information and knowledge management,
October 26-30, 2010, pp. 1549-1552.
[5] Banerjee, Snehasish, and Alton YK Chua. “Applauses in hotel
reviews: Genuine or deceptive?.” Science and Information
Conference (SAI), 2014. IEEE, 2014, pp. 938-942.
[6] Heydari, Atefeh & Tavakoli, Mohammadali & Salim, Naomie &
Heydari, Zahra. “Detection of review spam: A survey.” Expert
Systems with Applications, 2015, pp. 3634-3642.
[7] Crawford, M.; Khoshgoftaar, T. M.; Prusa, J. D.; Richter, A. N. & Al
Najada, H. Survey of review spam detection using machine learning
Figure 1. Results of this paper and A. Mukherjee’s [12] techniques, J. Journal of Big Data, February 23, 2015.
[8] Lim, P., Nguyen, V., Jindal, N., Liu, B., & Lauw, H, “Detecting
product review spammers using rating behaviors.” In Proceedings of
the 19th ACM international conference on Information and
knowledge management. ACM. 2010, pp. 939-948
[9] Yu, Mengqi, Meng Xue and Wenjia Ouyang, “Restaurants Review
Star Prediction for Yelp Dataset,” 2015.
[10] DaoRuna; XianguoZhang; YongxinZhai.Try to Find Fake Reviews
with Semantic and Relational Discovery, 2017 13th International
Conference on Semantics, Knowledge and Grids (SKG), 2017, pp.
234 – 239.
[11] Yu, Boya; Zhou, Jiaxu; Zhang, Yi; Cao, Yunong, “Identifying
Restaurant Features via Sentiment Analysis on Yelp Reviews,”
Figure 2. Results of with LDA and without LDA ARXIV, 2017.
[12] A. Mukherjee, V. Venkataraman, B. Liu, and N. S. Glance, “What
yelp fake review filter might be doing?” Proceedings of the 7th
REFERENCES International Conference on Weblogs and Social Media,
[1] Jindal, N., & Liu, B, “Opinion spam and analysis,” In Proceedings of ICWSM 2013, pp. 409-418.
the international conference on web search and web data mining, [13] Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean,
ACM, 2008, pp.219-230. “Distributed representations of words and phrases and their
[2] M. Ott, Y. Choi, C. Cardie, and J.T. Hancock, “Finding deceptive compositionality,” J. In Advances in Neural Information Processing
opinion spam by any stretch of the imagination.” In Proceedings of Systems, 2013, pp. 3111-3119.
the 49th Annual Meeting of the Association for Computational [14] David M. Blei, Andrew Y. Ng, Michael I. Jordan “Latent Dirichlet
Linguistics: Human Language Technologies-Volume 1, Association Allocation,” [J].Journal of machine learning research, 2003, pp.
for Computational Linguistics, 2011, pp. 309– 319. 993-1022.
[3] Myle Ott, Claire Cardie, Jeffrey T. Hancock “Negative Deceptive [15] Liu Yang, Minghui Qiu, Swapna Gottipati, Feida Zhu, Jing Jiang,
Opinion Spam,” North American Chapter of the Association for Huiping Sun and Zhong Chen. “CQARank: Jointly Model Topics
Computational Linguistics: human language technologies (NAACL and Expertise in Community Question Answering.” In Proceedings
HLT 2013), June 9-14, 2013, pp. 497-501. of the 22nd ACM International Conference on Information and
[4] Nitin Jindal , Bing Liu , Ee-Peng Lim, “Finding unusual review Knowledge Management, 2013. pp. 99-108.

283