Stock Market Trend Prediction Using Sentiment Analysis

Senior Project
∗
Nirdesh Bhandari
Earlham College
801 National Rd W
Richmond Indiana
[email protected]

ABSTRACT accuracy. Google search queries have been shown to predict

For decades people have tried to predict the stock mar- consumer spending and online chat activity affecting book
kets. Some have used historical price trends to predict fu- sales.
ture changes, while others rely on their gut feeling to make
predictions. The prices of stocks reflect the overall confi- Moreover, one prominent study done by [4] was able to show
dence the market has on the stocks. The level of this confi- that the levels of public anxiety correlated with the SP500
dence, according to the behavioral economics, is a collective values. Then, one might ask -If public mood and emotions
of society’s emotions towards a particular stock, which to are so indicative of changes in stock prices, why doesn’t ev-
some extent influences their decision-making. However, is eryone just use it to get to that pot of gold? While it is
there a way to know the collective mood of society towards true that collective public sentiment predicts stock values,
a stock? Can this mood be extracted from newspaper ar- the larger problem is extracting and quantifying the indica-
ticles and magazines? To address this question, I turn to tors for these sentiments accurately. One cannot enter into
the field of Natural Language Processing. With the help of people’s minds to know how they feel when they read an
various sentiment dictionaries, I ran various types of senti- article in the newspaper or see a TV segment. Furthermore,
ment analysis over 1.7million newspaper articles published different people might react differently to the same news.
in The Guardian between 2000 and 2016. I then chart the
changing sentiments over a time period against the various The measure of public opinion can be obtained from many
stock market indices to see whether or not news sentiment sources. The more prominent sources would be Television,
is predictive of economic indicators such as stock prices. social media, magazines and newspaper while other sources
include blogs, reviews, and forums. To review these sources
Keywords different sentiment tracking techniques can be applied. While
it might be easy for a human to read a black of text and un-
Sentiment Analysis, Stock Market Prediction, Natural Lan-
derstand the meaning behind it, the same task can be very
guage Processing
difficult for a machine. The degree of accuracy of a senti-
ment analysis ultimately depends on how well the classifier
1. INTRODUCTION can deduce the polarity/sentiment of a sentence. The polar-
Predicting the stock market has been a century-old quest ity of a sentence measures whether the meaning conveyed by
promising a pot of gold to those who succeed in it. The a sentence is positive or negative and is usually measured on
difficulty of making these predictions lies in the fact that a scale of -1 to 1, where -1 represents a negative sentiment
the stock markets respond to the news. The market confi- while 1 represents a positive sentiment. Using Textblob 1 ,
dence a particular stock changes as new developments are a natural language processing package we can evaluate the
made and public opinions shift signaling actions from the polarities of sentences. For example, ”This fat rat is bigger
investors. Indeed, many previous pieces of research done on than a cat” carries a polarity score of 0 while the sentence-”
the subject have found a correlation between public senti- The rat loves cheese more” has a polarity score of 0.5 com-
ment and economic indicators.Research done by [1] found pared to a -0.7 for ”Rat thinks a cat is ugly.” While most
that public sentiments related to movies as expressed on sentiment analysis tools can easily correctly deduce the po-
Twitter could predict box office receipt to a great degree of larity of sentences, there are instances where they can fail.
∗Student at Earlham College Sarcasm, satire, poetry and other complex forms of expres-
sions where the meaning isn’t straightforward are some of
such instances where even the most complex sentiment ana-
lyzers fail. TextBlob returns a 0.83 polarity for the sentence
-”You crashed the car? Very good, well done, nice!” How-
ever, it is important to note that when dealing with news
articles within large corpuses, satire and sarcasm constitute
a small share of sentences and their effects are often not sig-
nificant enough to throw off the polarity aggregates of an

1
http://textblob.readthedocs.io/en/dev/
entire news section. Before I dive into the details of my pro- ment. Using various sentiment dictionaries to look at overall
gram, I will be going over some previous research in the next sentiment of a news article, the program graphs the senti-
section. ments over a certain timescale and then compares it to the
different stock market indexes. Figure 1 gives an overall soft-
2. PREVIOUS RESEARCH ware architecture of my project.The first step involves data
Traditionally sentiments and opinions have been collected collection from using the Guardian API2 .The second step
using surveys and polls. While these are fairly reliable, is to collect data for the stock market indices from Yahoo
they are highly dependent on the sample size. There has Finance3 for the same time range. The next involves pro-
always been tremendous potential for research, competitive cessing the data and creating a workable data frame. Then,
analysis, and market data collection in being able to recog- the data frame is fed to a sentiment analyzer, which iterates
nize human opinion from online documents, chat rooms and over the articles and returns their sentiment value. The final
news articles. While humans can easily recognize the text in step involves the visualization of data along with tests for
these documents, similar understanding of textual context cross-correlation and a simple Random Forrest prediction.
and common linguistic occurrences proves to be difficult for As of writing this paper, the source code for my program is
a machine. hosted on GitHub 4 .

While most sentiment analysis in the past focused had fo-

cused on using statistical approaches to look at adjectives
and verbs as well as parts of speech, later work focused on
parsing sentences to better understand it. [7] recognized
that the essential issue to a sentiment analysis is not only
the identification polarity and the strength of that polar-
ity, but it is also important to look at semantic relation-
ships between subjects. Based on the hypothesis that local
sentiments affect overall perception more than global senti-
ments, they used this subject-based approach to achieve a
higher accuracy in predicting the overall polarity of a doc-
ument. Other work done in the field looks at concept-level
approaches to sentiment analysis. These methods use a ran-
dom walk and iterative regression to build a concept-level
dictionary. Meanwhile [8] employed an emotion labeling
mechanism to sentiment analysis on a concept-level.[10]

On the other hand, machine learning based sentiment analy- Figure 1: Software Architecture
sis techniques such as Artificial Neural Net (ANN), Random
Forest, Support Vector Machines (SVM), NaÃŕve Bayes,
3.1. Data Collection
Multi-Kernel Gaussian Process (MKGP), XGBoost Regres-
sors(XGB), etc are also used to classify sentiments. How-
The first task of this project was collecting and compiling
ever, given the complex nature of linguistic identification,
a newspaper corpus to run sentiment analysis. While there
machine-learning approaches rarely attain more than 70%
were corpuses of historical texts and emails available on the
accuracy [10].
Internet, not many online archives provided the articles in a
chronological structure with well-organized Metadata. Fur-
Similar work has been done by [5] where they collected Ap-
thermore, digital forms of newspapers would require text
ple Inc. stock information and news related to Apple Inc.
extraction and often had copyright issues.
over a time span of three years. They gathered news from
Google as well as yahoo finance. Using a sentiment detection
I decided to compile my own corpus of news articles. The
algorithm, they collected the sentiments over a preprocessed
two helpful APIs I found for this purpose was the NY Times
corpus of news articles. Their sentiment classifier employed
API and The Guardian API. I used the Guardian API to
a bag of words approach where they counted the number
compile a corpus of 1.75 million articles. While the Guardian
of positive and negative words in each piece. Then using
API was free for developers and offered structure data, it
several models of SVM, Random Forest, and Naive Bayes,
would only return the URL for the articles. So I used python
they obtained an accuracy score range between 75% to 90%
to return daily search queries, open each article and created
on the test dataset. Likewise, [6] also employed a similar
a compilation of articles of each day. My data was organized
method as done by [5] on Indian stock companies over a
on JSON files for each day from the 1st of January of 2000 to
ten-year span and got an accuracy score range between 47%
2016. I picked the Guardian API because the data returned
to 75%.on the test dataset for SVM, KNN and Naive Bayes.
contained the Title, date, section ID, web publication date,
They also employed a bag of words approach.
text publication date and other helpful structured metadata.
3. PROGRAM DESIGN As for the stock data, I decided to use Yahoo Finance and
For this project, I decided to build sentiment analysis tool
2
that retrieves, processes, evaluates, visualizes and tests news- http://open-platform.theguardian.com/
3
paper articles. The goal of my project was to find the cor- https://finance.yahoo.com/
4
relation between stock prices and the overall market senti- https://github.com/nirdesh1995/CS488_Project
download their historical data for the end of day closing time series since stock markets tend to close on weekends
prices for key stock indices to create a pickle file. The Ya- and some public holidays. Given the fact that I was deal-
hoo Finance API also connects to the Pandas5 package at ing with time series data where response lags in variables
python, which makes it convenient for gathering information were expected, I decided to interpolate values for the missing
of various stock prices. For the stock indices, I picked the days instead of removing them from my analysis. For this,
Standard Poor’s 500, Dow Jones Industrial Average, NAS- I used the inbuilt interpolate function of Pandas to evenly
DAQ Composite, Wilshire 5000 Total Market Index and the space out the values between the gaps. While interpolating
Russell 2000 Index. I believe that these five indices would with only closing values isn’t always the ideal solution, the
be sufficient enough to evaluate various sectors of the econ- evenly spaced averaged prices approach is better than not
omy in respect to what appears on newspaper articles. The looking at those dates altogether. Since stocks are respon-
program I designed simply takes in the start and the end sive to news articles and news is published throughout the
dates to collect the news articles and stock values. weekend, my intuition was that not factoring in the week-
ends altogether would throw off predictions. Furthermore,
when data frames for the stocks and polarity values were
combined, I noticed that some sections had missing values.
The missing values create problems in the visualization and
analysis portion later on. Since these missing polarities were
occasional, I adopted the simple solution of backfilling the
data frame whereby any empty entry would take the value
from the next non-empty entry.

3.3. Sentiment Classification

The two major ways to do a sentiment analysis include either

the use of a Lexicon based approach or a Machine-Learning
approach. The machine learning approach employs the use
of either brute force unsupervised learning or the creation of
a classifier using supervised machine learning. The unsuper-
Figure 2: Data Collection vised approached seemed impractical partly because of the
amount of computation it would require and the absence
3.2. Pre-Processing of a proper dataset to train the classifier on. As for the
supervised approach, the most common ones included lin-
After creating a corpus of newspaper articles, the next step ear neural network classifiers and Naive Bayes and Bayesian
involved processing the data before it could be fed into the Network Probabilistic classifiers. Previous work is done on
sentiment analyzer. Using the pandas inbuilt JSON pack- supervised classifiers often employed a movie review corpus
age I was able to read and parse articles for a given time or the use of Amazon reviews to train a classifier. While
period. The JSON files included many unnecessary fields these approaches were commonly used in previous research,
such as weburl, id, pillar-id, etc. Only necessary fields such because of the computational complexities and my lack of
as publication date, section ID, word count and body-text knowledge in the field, I decided to revisit these later once
were used to compile the final data frame. I felt that it I had a working model. Furthermore, my corpus occupied
would help to save the section ID for the articles as further more than 17 gigs of space and training a machine learning
analysis on different sections (health, money, UK-news, tech, classifier for such a large dataset would take an enormous
etc.) might improve tests in the future. The process of data amount of time.
collection and processing is shown in Figure 2
Therefore, I decided to go with a lexicon-based approach
My sentiment analysis was based on evaluating the body for the initial phase. The two popular python packages
text for the articles to receive the sentiment scores. My ini- that I found for this task were Valence Aware Dictionary
tial design removed stop words from the body text before and sEntiment Reasoner6 (VADER) and TextBlob. Because
compiling the data frame. However, upon a closer inspec- VADER is designed explicitly for sentiment analysis on so-
tion of the TextBlob stopwords dictionary, I saw that some of cial media, I decided to use the TextBlob library. Textblob is
the stop words carried non-neutral polarity scores. Indeed, built on top of the NLTK package7 in python and seemed to
as [9] found, removing of stop words to reduce noise from a be a reliable tool for classification of news articles. TextBlob
pre-set dictionary often tends to lower the accuracy of senti- employed a lexicon based dictionary approach where a pre-
ment classification since stop words tend to add contextual existing dictionary of words, classified based on their polar-
information. Therefore, I did not remove stop words from ity values, is used to calculate the overall polarity score of
the articles for my analysis. Also, the TextBlob analyzer a block of text. TextBlob was particularly useful because of
was not sensitive to upper/lower case and while sentence the ease in pre-processing and tokenization as well as sen-
and word tokenization were initially useful in understanding timent classification. Using text blob, my python tool ex-
how the tool worked, it was not relevant later on as I was tracted the sentences out of an article and then use a bag of
evaluating entire articles at the same time. words approach to return the sentiment polarity scores. The

As for the stock data, I noticed that there were gaps in the 6
https://github.com/cjhutto/vaderSentiment
5 7
http://pandas.pydata.org/ http://www.nltk.org/
sentiment polarity scores were in the range of -1 to 1, where results would give me a quicker view of the data rather than
-1 represented a highly negative article while 1 represented going over the numbers in a large table. I, therefore, opted
a very positive article. These scores were then aggregated for a heat map using the Seaborn9 visualization packet to
and stored in a Pandas data frame along with the headline display the cross-correlation results. Because the correla-
of the article, the word count and the sections these articles tions I was interested in were the ones between the polarity
belonged to. Because of the large size of the corpus, I com- values and the stock indices, I set a max value of 0.3 for the
piled these polarity value data frames for yearly chunks of heat map. This would be helpful to accurately see changes
news articles. Each year took roughly 20 min to run through in the smaller correlation values between stocks and polari-
the analysis tool. ties since the larger values of stock-stock correlations would
simply occupy extreme end of the spectrum. Figure 4 shows
3.4. Visualization one of the heat maps I obtained.

Once the polarity scores had complied, the next phase of

the project was to take the compilation of these scores and
graph them along with the stock prices. Because of how my
metadata was organized, I decided to track the aggregate
sentiment scores by different sections (for example, health,
tech, business, etc.) within the newspaper articles. A quick
grouping evaluation showed me that the most of the articles
belonged to the following sections: world, business, poli-
tics technology, money, and media. While there were other
sections such as ’commentisfree’, ’film’, ’lifeandstyle’, etc.,
these accounted for minimal volume in the corpus. Further-
more, I felt that the film section would not be predictive of
changes in the stock market. Because the data spanned 16
years, I decided to look first at chunks of 6-12 months to get
a more detailed view of the correlations before scaling it up.

Figure 4: Correlation Heatmap For Absolute Values

3.5. Analysis

When looking at the different heat maps, I observed very

little correlation between stocks and polarity values. Upon
closer examination of the numbers I saw that the stock
indices usually consisted of very high values (often in the
thousands) while polarity values ranged between -1 and 1.
Also, when I looked at polarity numbers of different sec-
tions, I realized some sections had a higher average value.
This was because news sections consist of a large number of
articles and sections like ’technology’ would always be pre-
dominantly positive. While negative articles would create
Figure 3: Data Visualization fluctuations within daily aggregates, the trend line would
be higher for some and lower for others. I decided to tackle
The first level of my analysis involved a simple visualiza- these problems by evaluating percent changes in daily po-
tion to see how the polarities within these select sections larity aggregates and stock-indices. I felt that looking at
corresponded to the stock prices. Using Matplotlib8 , I was changes as opposed to absolute values would better improve
able to graph stock indices along with the polarities. Fig- my analysis. Indeed, looking at percentage changes did show
ure 3 shows an example of the graphs I obtained. While the significantly stronger results for cross-correlations and can
graphs might visually show correlation or the lack of one, it be seen in Figure 5.
is always more accurate to use a scientific method compare
predictability. Therefore, for the second level of analysis, I Another factor that I felt would be significant in my analysis
decided to look at cross-correlations within the stock val- was time lag. It often takes time for news to reach people,
ues and polarities of articles from the different sections. I and response from stock markets could be delayed. Further-
used the Pandas cross-correlation function for this task. Be- more, I wanted to check if these lags in correlation were dif-
cause I had a total of 11 variables (5 stock indices and 6 ferent based on the section of the newspaper. Figure 6 shows
section polarities), the cross-correlation table was a 11X11 the plots for correlations against some sections based on dif-
table with very high correlation values within the stock in- ferent lag values for SP500. I found similar graphs when
dices themselves and lower values for correlation between I ran the same tests against NASDAQ prices and Wilshire
polarities and stocks. Before moving on to other levels of prices. Figure 6 shows that the correlation of the ’world’
analysis, I decided that visualizing these cross-correlation section is slightly higher when the lagged by one day while
8 9
https://matplotlib.org/ https://seaborn.pydata.org/
Figure 5: Correlation Heatmap for Percentage Figure 7: Rolling Correlation for Money Section
Change

those for ’business’ and ’politics’ sections peak at the fourth

day. However, given that my data was a time series, the
constant peaks and falls over the days led me to believe that
this could just be noise within the data. Furthermore, I
was also unsure whether the Guardian’s publication hours
for articles would affect my results. Articles published at
1 am in the morning, as well as articles published at 11:59
pm, are being compiled under the same date while the stock
market opens at 9:30 am and closes at 4 pm. However, given
the time constraint, I decided to leave that analysis for the
future.

Figure 8: Rolling Correlation For Technology Sec-

tion

decided to use the rolling correlation between polarity val-

ues for money section for the year 2008 and SP500 prices
with a rolling window of 50 days. As Figure 7 shows, there
were significant shifts in correlation occurring in July and
August 2008- right around the time of the Financial Cri-
sis. Meanwhile, when I ran the same test for the ’media’
section, the results were different than what was seen with
the ’money’ section and can be observed in Figure 8. While
finding these significant shifts in themselves might not help
predict future price changes in response to the news, the
Figure 6: Correlation for different lag days tests confirm that certain sections could be more predictive
of stock price movement than others.
Likewise, another question I had was whether or not the
correlations between sentiments in news articles and stock 3.6. Predictive Analytic
values changed over time. For example, it would be reason-
able to think that during times of financial crises, the news While correlations and visual analysis of polarities within
would have more of an impact on stock prices. To test this sections of the newspaper and stock indices were useful in
hypothesis, I decided to use rolling correlation or the dy- getting a closer look at the data, I wanted to predict fu-
namic correlation test. In simple terms, a rolling correlation ture prices using these polarities. For this purpose, I tried
looks at the correlation between two time-series variables to implement a simple Random Forest Regression using the
as a rolling window calculation. Rolling correlation visu- Scikit-learn10 package available for python. For a prelimi-
alization helps check for changes in correlation over time nary experiment, I decided to implement the simplest ver-
and can be helpful to detect significant events that shift the 10
http://scikit-learn.org/stable/modules/generated/
relationship of one variable with respect to another [3]. I sklearn.ensemble.RandomForestRegressor.html
sion of this machine-learning model to produce a predictive offer as much as 15 categories for sentiments. These dic-
graph. Using a training dataset comprised of the first 8 tionaries offer a wider range of sentiment classes and could
months of 2010, I trained model the to predict SP500 prices be helpful to check if other sentiments (besides positive or
using the polarities of the business, money and world sec- negative) were more predictive of changes in stock values.
tions. My previous analysis showed that these sections had Another test could be to use VADER sentiment analysis
significantly higher correlation values to SP500 compared to tool as mentioned above. Furthermore, this pipeline could
other sections. Figure 9 shows the results I obtained. While be implemented in a different set of new articles from the
my projections were far off from the testing data, I believe New York Times to check if the sentiments expressed in this
that the accuracy of the predictions can be improved using century-old newspaper would be more predictive of stock
a more extensive training data set and by testing with other indices.
models.
4.2. Other Analytic Methods and Prediction Models

Further improvements in statistical analysis could be made

on this project. One way to see how much a variable affects
another is to run a Bivariate Granger Causality Analysis [2].
Simply put, this is a linear regression based econometric test
that checks whether changes in values of sentiment scores led
to the changes in the value of stocks. Meaning, if the chang-
ing sentiment scores in a certain section led to a change in
the value of stock, the lagged values on one side show a pos-
itive correlation to the effects that took place on the other
side. Another method to improve this project would be by
using the SOFNN or Self Organizing Fuzzy Neural Network
model as described in [2]. Since sentiment trends are not
always linearly occurring, this would be a more accurate
method to conduct the analysis. However, since SOFNN
predicts stock values based on the past n days as an input
to self-organize its neurons, and I was dealing with a large
Figure 9: Prediction based on Random Forest Re- dataset. I felt that this would be computationally challeng-
gression ing for this project but would be a great model to check out
in the future.
4. DISCUSSION AND FUTURE WORK
The field of Natural Language Processing and sentiment 5. ACKNOWLEDGEMENTS
analysis for stock predictions turned out to be much larger The process of compiling this project has helped me a great
than I had initially anticipated. Over the course of this deal in polishing my problem solving and analytical skills.
project, I realized that there are many different choices of The tools that I have acquired over the course of this project
tools at each step and many various methods to analyze and will be vital to my possible future career in Data Science.
interpret the results. Indeed, if predicting the stocks were A particular word of gratitude is due to Charlie Peck for
that simple, Wall Street would not be investing millions into guiding my research for this project and for helping me over
predictive forecasting programs. Overall I am happy with the course of this semester. I would also like to express
the work I did for this project. For me, this project was my most profound appreciation to David Barbella being my
about diving into an unknown field of interest and explor- Capstone advisor.
ing it best I could within a short timespan. I am sure I
could improve my program’s prediction capabilities if I keep
working on this project in the future. My initial goal for
6. REFERENCES
[1] S. Asur and B. A. Huberman. Predicting the Future
this project was to explore if there was a correlation be-
with Social Media. In 2010 IEEE/WIC/ACM
tween sentiments of news articles and stock indices and I
International Conference on Web Intelligence and
was able to meet that goal through this project. There is
Intelligent Agent Technology, volume 1, pages
much I could improve on my statistical tools as well the clas-
492–499, Aug. 2010.
sifier. During this project I went with the choices that made
intuitive sense to me; however, there were many different [2] J. Bollen, H. Mao, and X. Zeng. Twitter mood
places where other tools/options could have been employed. predicts the stock market. Journal of Computational
Science, 2(1):1–8, Mar. 2011.
4.1. Different Classification Tools [3] business science.io. Tidy Time Series Analysis, Part 3:
The Rolling Correlation, July 2017.
My project is built on top of a lexicon dictionary-based [4] E. Gilbert and K. Karahalios. Widespread worry and
approach of sentiment classification. An improvement to the stock market. 2010.
this project could be to test the polarities using the various [5] J. Kalyani, P. H. N. Bharathi, and P. R. Jyothi. Stock
other dictionaries that were initially put aside because of trend prediction using news sentiment analysis.
the complexities. This would still be a bag of words method arXiv:1607.01958 [cs], July 2016. arXiv: 1607.01958.
to aggregate sentiment scores. However, certain dictionar- [6] M. D. K. Kirange and D. R. R. Deshmukh. Open
ies such as the Harvard IV Sentiment Dictionary (HVD) Journal Systems.
 [7] T. Nasukawa and J. Yi. Sentiment analysis: capturing
favorability using natural language processing.
page 70. ACM Press, 2003.
[8] S. Poria, A. Gelbukh, A. Hussain, N. Howard, D. Das,
and S. Bandyopadhyay. Enhanced SenticNet with
Affective Labels for Concept-Based Opinion Mining.
IEEE Intelligent Systems, 28(2):31–38, Mar. 2013.
[9] H. Saif, M. Fernandez, Y. He, and H. Alani. On
Stopwords, Filtering and Data Sparsity for Sentiment
Analysis of Twitter. May 2014.
[10] P. Takala, P. Malo, A. Sinha, and O. Ahlgren.
Gold-standard for topic-specific sentiment analysis of
economic texts. 2014.