See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/366020486

E-commerce Data Analysis and Visualization using Random Forest Regression

and Prophet model

Preprint · December 2021

DOI: 10.13140/RG.2.2.18694.45126

CITATIONS READS

0 326

4 authors, including:

Tasnim Rafia Tanveer Mahmood Hasan

International Islamic University Malaysia International Islamic University Malaysia
3 PUBLICATIONS 0 CITATIONS 2 PUBLICATIONS 0 CITATIONS

SEE PROFILE SEE PROFILE

K. M. Zubair
International Islamic University Malaysia
5 PUBLICATIONS 1 CITATION

SEE PROFILE

All content following this page was uploaded by K. M. Zubair on 05 December 2022.

The user has requested enhancement of the downloaded file.

 E-commerce data analysis and visualization
Using Random Forest Regression and Prophet model
E-commerce sales prediction.
Tasnim Rafia, Hasan Tanveer Mahmood, Billah Syed Mashkur, KM Zubair
Department of Computer Science, Kulliyyah of Information and Communication Technology,
International Islamic University Malaysia, Kuala Lumpur, Malaysia.
[email protected], [email protected], [email protected] , [email protected]

Abstract— The motivation behind this study is to better understand the ecommerce sector by
examining public datasets. In our case, we’ll work on Pakistan’s Largest E-commerce dataset that
we found from kaggle. We’ll work on problems like finding the most sold products in the store to
develop a better business model, most popular categories in terms of sale, demand for products
that are in high rising phase, products that get cancelled the most and predicting upcoming year’s
sales condition to analyze the growth rate of sale in different timespan. We'll use Apache Spark, an
open-source cluster-computing technology, as well as Tableau, which is also crucial for the tasks at
hand. This research could be applied to commodity assessments of purchased things on
e-commerce sites to do sentiment analysis.

Keyword: E-commerce, Sales Analysis, Upcoming sales prediction, fbProphet model, RandomForestRegressor

benefits of word vectors, sentiment lexicons, GRU, CNN and

I. INTRODUCTION test it on a book review dataset from a genuine e-commerce
This project is primarily trying to analyse one website. To begin, the sentiment lexicon is employed to
E-commerce dataset (Pakistan's Largest E-Commerce improve the sentiment elements in the reviews. The CNN
Dataset), where the product values will be analysed along and the Gated Recurrent Unit (GRU) networks are then
with the popularity by comparing the sales history among utilised to extract the major sentiment and context
themselves. According to the analysis, some suggestions to characteristics in the reviews, which are then weighted using
undertake are going to be provided in order to increase the the attention method. Finally, categorise the weighted
popularity of the company by forecasting the sales condition sentiment characteristics. Zhang et al. [2] proposed a
of next year with the help of time series analysis in terms of Sentiment analysis-based Fuzzy decision support model for
the dataset history such as product status, the most sold e-commerce item comparison. To describe online reviews,
product history etc. that is already accorded in the dataset. the model uses probability multivalued neutrosophic
Much research has been already conducted using public linguistic numbers (PMVNLNs). By taking into account
datasets, still many business questions yet to be answered neutral information and reluctance in text reviews, it
from realtime dataset. The purpose of this research is to overcomes the limitations of previous models. The
understand the ecommerce domain better by analysing the suggested characterisation technique, unlike existing item
public datasets. An overall depth analysis will be performed rating algorithms, incorporates positive, neutral, and
by comparing the product status history so far. The main aim negative information in text evaluations and reflects review
of the project is to implement a solution to assist the reticence. Because of the characteristics of item comparison
company to enhance their popularity. Therefore, this difficulties in e-commerce, the model proposes QUALIFLEX.
research will propose a strategy on how they can improve Using the integrated regret theory-QUALIFLEX, it addresses
their sales. customers' constrained rational behaviour, unlike previous
models.
II. RELATED WORKS
Yang et al. [1] proposed a paper for Sentiment III. DATA DESCRIPTION
analysis for e-commerce product reviews. Sentiment analysis
The data was obtained via Kaggle's website, which
for product evaluations is a way of automatically estimating
offers complementary datasets that may be viewed and
the customer's emotional inclination by analysing subjective
used for additional research. We choose Data analysis of
remark language with the customer's emotional colour.
Pakistan's largest e-commerce dataset that has 1048574
They’ve used rule-based methods and lexicon-based
rows and 26 columns. From July 2016 to August 2018, the
methods to construct a sentiment lexicon using degree
dataset contains sales information such as product quantity,
adverbs, sentimental intensity, words and polarity. They
payment method, order status, and so on.
develop a novel sentiment analysis model based on the
 IV. EXPERIMENTAL SETUP is a combination of learning models to increase overall
This section of the paper covers the stages in detail, as results.
selecting the right things to sell is not always simple. There
Prophet algorithm is used to conduct tasks on time
are numerous elements that influence whether a product
series data and large volumes of data. From commonly
succeeds or fails, and one split choice could mean the
purchased item sets, the prophet algorithm can build
difference between massive success and comparable
association rules. Following the analysis of all of these
disaster. Thus, e-commerce data analysis works by
factors, a visual depiction of our study findings will be
forecasting changes in a customer's buying behaviour as a
presented in order to make the research findings clear to
result of qualitative products, seasonal sales and delivery
people from all walks of life.
performance, and a variety of other elements. The study will
give effective monitoring analyses that will assist merchants
C. Data Cleaning
in anticipating potential purchasing impulses and capitalising
on trends while sustaining sales growth rates. The datasets to be utilised are first reviewed and, if
required, cleansed. Several data cleaning procedures were
A. Algorithm - Random Forest Regression / Prophet
used to guarantee that the datasets were suitable for
In this model Random Forest Regression machine testing. The following graphic depicts the data cleaning
algorithms will be applied. It is a supervised learning procedures that were performed using Google collab:
algorithm which uses the approach of ensemble learning for
classification and regression. There is no inter relation
between these trees while constructing the trees. It works
by building a multitude of decision trees at training time and
gives output which is mode of classification and regression.
It can combine multiple predictions.

Prophet is a time series data forecasting process based

on an additive model that fits non-linear trends with yearly,
weekly, and daily seasonality, as well as holiday impacts. It
can be implemented best with time series with substantial
seasonal influences and historical data from multiple
seasons. Prophet is forgiving of missing data and trend
shifts, and it usually handles outliers as well.

We have used both methods and tried to figure out

which methods provide the most accuracy and fits with
datasets best. We have split data in a ratio of 7.5:2.5.

Figure 1: Flowchart for data Cleaning and Preprocessing.

To begin, the dataset's directory is specified. After that,

the dataset is loaded into Google collab. For our project.
Raw data will have missing values, columns containing
information that we do not want, and some of them may be
duplicated.To delete any columns that we do not need for
our project, we have used the instruction datasets.drop
(['column_name'], axis='columns', inplace=True) where
datasets is the name of our data and [‘column_name’] is the
column that we want to remove, in this case columns
‘sales_commission_code', 'Unnamed: 21', 'Unnamed: 22' and
image source: Random forest regression so on we’ve removed.
B. Performance Evaluation
In addition, we tried to eliminate the rows with missing
We'll train our model using random forest regression, values, So, we used the python command .isnull().sum() to
which will look for a correlation between prior sales rates see the null values or missing values and .notna() to remove
and months. Predictive analysis of sales growth will be missing values from the dataset. We only want the
conducted using the trained model. It basically trained with completed version of the dataset with no missing values
the bagging method. The basic idea of this bagging method since we want to achieve an accurate result. By using
sns.heatmap we’ve visualized the null values column.

Figure 5: Flow on Modeling Steps

The first step, as shown in Fig. 3, is to import the

dataset into Google Collab. Data preprocessing is required
before modelling can begin in order to learn more about the
dataset. The goal of exploratory data analysis is to find
Figure 2: Heatmap before removing null values. analytical graphs and the distribution of the data. After EDA,
the dataset is randomly divided into a training set and a test
set. The training set should account for 75% of the total
dataset, whereas the test set should account for 25%. The
Random Forest Regression is then applied to the training
data. The number of clusters chosen is determined by the
results of prior internal evaluation.

A. Exploratory Data Analysis (EDA)

a) Correlation Test

Following clustering on the training set, a correlation

test is performed to determine whether there is a strong,
weak, or no association between two variables. The
direction of the association between the variables is also
Figure 3: Heatmap after removing the null values and revealed by the correlation test heatmap. The correlation
unnecessary column coefficient is usually between +1 and -1, with the positive
side indicating a stronger degree of relationship. When the
V. MODELLING coefficient hits one, the correlation is said to be perfect.
Pearson and Spearman Rank correlations are examined in
We will move on to the modelling section after cleaning the this dataset.
datasets and selecting the sample to utilise. The Algorithm
chosen for the dataset is “Random Forest And Prophet”.

Figure 4: Data Description

The modelling process is divided into several steps. We

follow this flowchart below.
Figure 6: Correlation between rows and columns.
 b) Filtering Columns for Visualization and Analysis:

Figure 10 : The categories receiving the most cancellation.

B. Train Dataset

After EDA has been used to analyse the dataset, it is divided

into two sets: one for the training phase and one for the
testing phase. The data is divided into 7.5 : 2.5 ratios, with 75
percent of the data serving as the training set and the
remainder serving as the test set. Random Forest Regression
is used to implement the training set.

VI. ANALYSIS OF RESULTS

Figure 7: Filtered column (Name of the categories by summing
up the item number for each category) A. Hypothesis Testing
We assumed previously from the analysis that, “One
Particular Period will Provide a higher Sales” The line graph
shown in the prediction model agrees on the point that at a
specific time, in the last quarter of 2018, the sale will
increase.

B. Model Testing
We have used two models for the prediction in our
project which are Random Forest Regression model and
Prophet model. Both models have been considered in terms
of their r2, mean squared error, mean absolute score. As the
Figure 8: Customers with total cancelled order
result of justification, we have preferred the Random Forest
method to forecast the future sales condition.

C. Results and Analysis:

a) Descriptive Analysis:
The research question for descriptive analysis was,
“Which category has the most popularity in terms of
sale? ”

Figure 9: Customers with total confirmed order

c) Creating plots for visualization:

Figure 11 :Result of descriptive analysis.

 From the analysis, we found that the most sold The result found that cancellation of items in the
category is ‘mobiles and tablets’. To simply say, that last quarter of year is really significantly high. So,
is the most popular item according to the analysis. ‘time’ might be a reason for item negation.

b) Diagnostic Analysis:
Regarding the diagnostic analysis, the
research question we selected is, “How was the
sales condition throughout the time from July, 2016
to July, 2018 ? ”

Figure 15: Customers who order and cancel the most

Again, while scrutinizing the customer history of

item negation, we found that there are some
customers who order the most; whether they
cancel the orders the most as well.
Figure 12: Result of diagnostic analysis
d) Exploratory Analysis:
As from the investigation, the figure below shows “Which product demand is in a high-rising phase? ”
the sales state of the past 3 years (2016 to 2018).

c) Causality Analysis:
For causality analysis, the research question
was, “Does the item cancellation get affected by time
or customer ?”

Figure 16: Product demanding phase by month

In this analysis, we found the popularity of products

differ from month to month in terms of some
products such as, mobile and tablets remaining at
the peak of demand in the last quarter, when others
Figure 13: Item cancellation based on time are less noticeable. On the other hand, in the first
quarter, other items are being sold out the most.
Again, there are some variations as well like, school
stationery, books are being bought at a continuous
same rate over the year.

e) Predictive Analysis:
We have made some predictions with the next
year's sales state condition which is based on this
research question: “How is the sales state going to
Figure 14: Items that not cancelled based on time
be next year ?” We have used two models for the
prediction which are ‘RandomForest Regression
We were determined to discover the answer whether the Model’ as well as ‘Prophet Model’
item cancellation depends on some particular time of a year,
or it is some other reason such as customers etc.
 View publication stats

users cancelling more than the average or 100 times might

have to accept some penalization. This technique, so far, will
be able to reduce the amount of cancellation that is
supposed to bring more profit for the E-commerce company.

Furthermore, the sales condition, at some particular

period of year, is being defined by some certain products'
popularity such as, demand for books and school stationery
products remain almost same over the whole year. But in
the last quarter, while mobile and tablets, men’s fashion and
appliances are altering the acceptance, it can be advised to
the company to concentrate more on the items other than
Figure 17: Predicted sales state of next year (2019) mobile and tablets in the last moments of years.

VIII. ACKNOWLEDGMENT
This project was conducted as part of the Big Data
Analytics (CSC 3303) course offered by the Department of
Computer Science at the International Islamic University
Malaysia in Kuala Lumpur. The authors of the study would
Figure 18: r2, MSE, MAE scores using like to express their gratitude to Dr. Sharyar Wani for his
RandomForestRegressor assistance in making this work successful.

Using the RandomForest regression model, we got the r2 IX.REFERENCES

score 0.4277885705891722, the mean squared error is
1. L. Yang, Y. Li, J. Wang and R. S. Sherratt, "Sentiment
4.700391261310883e+16, and the mean absolute score is
Analysis for E-Commerce Product Reviews in
138643103.93385413
Chinese Based on Sentiment Lexicon and Deep
Learning," in IEEE Access, vol. 8, pp. 23522-23530,
2020, doi: 10.1109/ACCESS.2020.2969854.

2. P. Ji, H. -Y. Zhang and J. -Q. Wang, "A Fuzzy Decision

Support Model With Sentiment Analysis for Items
Figure 19: r2, MSE, MAE scores using Prophet Model
Comparison in e-Commerce: The Case Study of
http://PConline.com," in IEEE Transactions on
Using the fbProphet model, we achieved the r2 score of Systems, Man, and Cybernetics: Systems, vol. 49,
-0.6234392611467166, the mean squared error of no. 10, pp. 1993-2004, Oct. 2019, doi:
32187147347009.316, and the mean absolute score of 10.1109/TSMC.2018.2875163.
4901326.587481598.

From the score (r2, MSE, MAE) stated above, after proper
justification, it may be declared that, RandomForest
regression model worked better in this case in terms of
accuracy and nicer prediction.

VII. CONCLUSIONS

Our aim was to suggest a solution for the

E-commerce sales site in order to increase their profit. We
have focused on some certain analysis and results to
discover the solution. At the end of the research, we have
acknowledged a couple of solutions.

An important tip would be to start a reward and penalizing

system. Because, the customers ordering most, are
cancelling their orders at large. So, the customers who are
ordering might get promotional offers as reward while the