Received May 26, 2020, accepted June 8, 2020, date of publication June 15, 2020, date of current version

June 25, 2020.

Digital Object Identifier 10.1109/ACCESS.2020.3002164

Performance Evaluation of IoT Data Management

Using MongoDB Versus MySQL Databases in
Different Cloud Environments
MAHMOUD EYADA 1, WALAA SABER 2, MOHAMMED M. EL GENIDY 3,

AND FATHY AMER 4

1 Mathematicaland Computer Science Department, Faculty of Science, Port Said University, Port Said 42511, Egypt
2 Electrical
Engineering Department, Faculty of Engineering, Port Said University, Port Said 42511, Egypt
3 Mathematicaland Computer Science Department, Faculty of Science, Port Said University, Port Said 42511, Egypt
4 Computer and Information Sciences Department, Faculty of Engineering, Cairo University, Cairo 12613, Egypt

Corresponding author: Walaa Saber ([email protected])

ABSTRACT The Internet of Things (IoT) introduces a new challenge for Database Management Systems
(DBMS). In IoT, large numbers of sensors are used in daily lives. These sensors generate a huge amount
of heterogeneous data that needs to be handled by the appropriate DBMS. The IoT has a challenge for the
DBMS in evaluating how to store and manipulate a huge amount of heterogeneous data. DBMS can be
categorized into two main types: The Relational DBMSs and the Non-relational DBMSs. This paper aims
to provide a thorough comparative evaluation of two popular open-source DBMSs: MySQL as a Relational
DBMS and MongoDB as a Non-relational DBMS. This comparison is based on evaluating the performance
of inserting and retrieving a huge amount of IoT data and evaluating the performance of the two types of
databases to work on resources with different specifications in cloud computing. This paper also proposes
two prediction models and differentiates between them to estimate the response time in terms of the size
of the database and the specifications of the cloud instance. These models help to select the appropriate
DBMS to manage and store a certain size of data on an instance with particular specifications based on the
estimated response time. The results indicate that MongoDB outperforms MySQL in terms of latency and
the database size through increasing the amount of tested data. Moreover, MongoDB can save resources
better than MySQL that needs resources with high capabilities to work with less performance.

INDEX TERMS IoT, DBMS, SQL, NoSQL, MySQL, MongoDB, AWS, cloud, multiple non-linear regres-
sions.

I. INTRODUCTION IoT data efficiently is one of the major challenges. Flexible

Nowadays Internet of Things (IoT) technology become the Database Management System (DBMS) is an important tar-
backbone of many industries like smart home systems, indus- get to implement this challenge [1].
trial control systems, monitoring of pharmacies and hospitals, The DBMS is a software that is responsible for storing
open-source web data and weather stations. IoT is a system and managing databases. Relational DBMSs (RDBM) [2]
that is based on sensing, collecting and sharing data. This are widely used systems. They are based on the Relational
system results in exchanging operations for a large amount model and they use Structured Query Language (SQL) as
of IoT data. Using IoT technology generates a large amount their application programming interface language. RDBMSs
of heterogeneous data like texts, numbers, audio, videos, and work well for storing structured data and managing their
pictures. These types of data need to be transferred, processed relationships. However, using IoT means dealing with a large
and stored in a cloud server. Also, they need to be queried and amount of heterogeneous data which has a harmful effect on
updated on-demand. Storing and managing a large amount of the performance of the traditional RDBMSs. Non-relational
DBMSs [4] are proposed to solve the limitations of the tradi-
The associate editor coordinating the review of this manuscript and tional RDBMSs. NoSQL databases are schema-free; they are
approving it for publication was Tariq Umer . designed to work in harmony with the unstructured data. They

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
110656 VOLUME 8, 2020
M. Eyada et al.: Performance Evaluation of IoT Data Management using MongoDB versus MySQL Databases

are supposed to be easily distributed with high scalability

and availability. These properties are needed to realize the
vision behind the IoT from the data perspective. Managing
and storing a large amount of heterogeneous data through an
appropriate DBMS is one of the contemporary challenges.
Performance evaluation of the Relational and Non-relational
DBMS is one of the solutions that help to meet this challenge
[3]–[6].
Cloud computing [7] technology enables access to a strong
pool of configurable computing resources such as servers,
networks, storage, services and applications; which can be
rapidly provisioned and released with minimal management
effort or service provider interaction. Cloud storage service
is an important part of the cloud infrastructure. It can easily
deal with a large amount of data. Such databases use the cloud
computing paradigm to optimize consistency, availability and
FIGURE 1. Managing and storing the IoT data.
partition tolerance.
The main objective of this paper is to find an effective way
to manage and store a large amount of heterogeneous data
in appropriate DBMS. This objective is implemented by per- industrial automation IoT and broadband IoT. Herein the
forming comprehensive experiments to compare and evaluate performance requirements are evaluated in terms of data size,
the performance of the two types of databases: MySQL as a latency, data rate, reliability and availability [8]. The essential
Relational DBMS and MongoDB as a Non-relational DBMS. aim of IoT technology is to manipulate the data efficiently.
The performance metrics include latency and database size. Servers and data centers are responsible for manipulating
This evaluation is implemented as follows: and storing all the received data smoothly. Then the data is
- Evaluate the effect of manipulating heterogeneous IoT processed by appropriate programming language to handle
data on performance. the manipulation operations. Finally, this data is stored in
- Evaluate the increase of workloads that result from the databases [9]. Figure 1 shows the main operations of man-
expansion of the IoT network and increasing the number aging and storing the IoT data.
of connected sensors on the way the data is stored within Cloud Service Providers such as Microsoft Azure [10],
the database and also on the size of stored data. Google cloud [11] and Amazon Web Services (AWS) [12]
- Investigate the effect of improving the cloud instances are needed for providing business applications, network ser-
capabilities on improving the performance. vices and infrastructure to the cloud users. Microsoft Azure
- Provide a new way for assessing and comparing the is used for general business applications and development
DBMSs and choosing the most appropriate one based on environments for Microsoft-centric organizations. Moreover,
formulating a prediction model to estimate the latency of it is used in cloud-native applications and batch comput-
the response time; in terms of realizing the database size ing. Google cloud is used for cloud-native applications and
and the instance performance. batch computing as well as projects that leverage the Google
cloud platform as a whole. AWS is the most popular in the
The rest of this paper is organized as follows. Infrastructure as a Service (IaaS) market due to the highest
Section 2 describes both background and related work. share and compute capacity which it uses in comparison with
Section 3 introduces tools of hardware and software which are other IaaS providers. AWS has many services, one of them
used in the experimental evaluations. Section 4 presents per- is Elastic Compute Cloud (EC2) that provides the ability to
formance evaluation experiments and results. Section 5 intro- create Virtual Private Server (VPS) such as your PC but with
duces a statistical analysis of predicting the appropriate dedicated Internet Protocol Address (IP), number of GB of
DBMS that meets user’s needs. RAM and Processors cores besides bandwidth and storage
space. It is actually one of AWS services, Amazon Simple
II. BACKGROUND AND RELATED WORK Storage Service (Amazon S3), that is used to save the dataset
A. BACKGROUND in the database server.
IoT technology is based on collecting a huge amount of Server-side programming languages are responsible for
data from different sources such as sensors, tracking devices, many tasks such as processing the collected data, inter-
smart-phones, social media and vehicles. These types of acting with servers, interacting with databases and manip-
data are emitted through the network to the appropriate ulation operations over databases. PHP [13], Python [14]
cloud platform to be managed by the appropriate services. and Node.JS [15] are the most common used server-
IoT applications are categorized; based on the performance side programming languages. Now Node.JS is the most
requirements, into four segments: critical IoT, massive IoT, commonly used one because it is JavaScript operating

VOLUME 8, 2020 110657

 M. Eyada et al.: Performance Evaluation of IoT Data Management using MongoDB versus MySQL Databases

environment, event-driven, asynchronous programming and B. RELATED WORK

specially designed for network services. Many researches were introduced to compare between SQL
Good database management systems are needed to store and NoSQL DBMSs based on IoT data structure. The main
the collected data in an appropriate way. DBMSs are cat- metrics in these researches were storage, syntax, and latency
egorized into two main categories: Relational DBMSs and of queries, database connection time and schema design.
Non-Relational DBMSs. Relational DBMS such as MySQL In [27], it discussed the possibility of a hybrid database
[16], Oracle [17] and PostgreSQL [18] are commonly known between MongoDB and MySQL. It shows that using Mon-
as SQL databases. They are the most common and widely- goDB in conjugate with MySQL improves the performance
used databases. SQL requires some predefined schemas to set of using the MySQL database. And in [28] authors showed a
the structure before you start to deal with the database. Here disparity in performance between the data models reference,
all your data must follow the same structure which means embedded and hybrid model. The hybrid model was based on
that a change in the structure would be both difficult and merging collections to improve the performance and reduce
disruptive to any system. Unfortunately, using IoT networks the storage size.
caused generating a large amount of heterogeneous data. In [29], [30] the performance of Relational and Non-
Traditional SQL DBMSs were not designed to solve these relational DBMSs was evaluated using two models: one was
problems. Non-Relational DBMSs such as MongoDB [19], implemented on a small scale of data and the other was
Cassandra [20] and Hbase [21] are commonly known as implemented on a large scale of data. The performance was
Not-only SQL (NoSQL) databases. NoSQL databases have a evaluated based on the CRUD operations to get the advan-
dynamic schema for heterogeneous data. All types of data in tages and disadvantages of both kinds of DBMS. The perfor-
NoSQL databases are stored and managed in many ways such mance was evaluated in terms of the time of queries and the
as column-oriented, KeyValue and graph-based; or organized size of data. However, the comparison was implemented on
as a document-oriented store. simple structured data and one type of MongoDB schema was
MySQL is a popular open-source RDBMS that is devel- examined, which didn’t show the benefits of MongoDB.
oped, distributed and supported by Oracle Corporation. Like In [31], [32] Relational and Non-relational databases were
other Relational systems, MySQL stores data in tables and examined on IoT data. The experiments were implemented
uses SQL language for database access. In MySQL you on MongoDB, PostgreSQL and MySQL. The results showed
pre-define your database schema based on your require- that NoSQL was better than SQL in different scenarios. How-
ments and set up rules to govern the relationships between ever, fixed numbers of sensors were used in the experiments.
fields in your tables. Any change in schema necessi- Also, a single type of MongoDB schema was examined.
tates a migration procedure that can take the database This paper is proposed to solve a lot of the previous lim-
offline or significantly reduce application performance. itations. It compares between MySQL database as a Rela-
MySQL supports various types of replication services. Also, tional database and MongoDB database as a Non-relational
its distributed database engine is more robust than the database in two different schema types; reference and hybrid,
PostgreSQL [22]. as a Non-relational database. More enhancements are pro-
MongoDB is a document-oriented Non-relational database duced on both databases to increase the flexibility and reduce
that can be used to distribute and store large binary files the redundancy. It uses the IoT benchmark to evaluate the
like videos and images. It stores data as documents in a performance of the two types of databases on the IoT data.
binary representation called BSON (Binary JSON) objects, Moreover, different scenarios with different numbers of sen-
which are binary encoded JSON like objects. Related infor- sors are examined. The performance is evaluated in terms of
mation is stored together for quick query access through the size and latency. In addition, the experiments were imple-
MongoDB query language. Documents in MongoDB can be mented on different cloud servers with different hardware
organized in ‘‘collections’’. Fields can vary from document specifications. Furthermore, statistical analysis is introduced
to document; there is no need to declare the structure of to predict the best DBMSs you should use depending on data
documents to the system – documents are self-describing. size and server performance.
If a new field needs to be added to a document, then the
field can be created without affecting all other documents
in the collection, without updating a central system catalog III. THE PROPOSED PERFORMANCE EVALUATION
and without taking the system offline. Therefore, it has a A. EXPERIMENT OVERVIEW
better performance than other databases in terms of resource The IoT data structure is considered as one of the main hetero-
usage and long-term storage to work with large amounts of geneous data structured types that need appropriate DBMS
IoT sensor data [23], [24]. MongoDB also supports indexing to deal with. The evaluation of DBMSs varies according to
over embedded objects and arrays. It interacts efficiently with the used criteria. This paper attempts to propose comprehen-
memory storage, complex data and dynamic queries com- sive criteria for performance evaluation and try to show the
pared to the other NoSQL databases. MongoDB is a scale- behavior of each type of database through this evaluation.
out based scheme that provides flexibility to work in case of The aim of the proposed experiments is to evaluate and
hardware expansion [25], [26]. compare the performance of the two types of DBMSs: SQL

110658 VOLUME 8, 2020

M. Eyada et al.: Performance Evaluation of IoT Data Management using MongoDB versus MySQL Databases

TABLE 1. Instance specification. elements of air pollution indoors and outdoors. The indoor
air frequently and severely polluted more than the outdoor
air, which can be refined naturally. The IoT sensor data that
depends on the temporal status and variables related to the
spatial characteristics of the space being measured should
be considered differently from other homogeneous inputs,
such as image and audio, because it considers heterogeneous
data [35]. Some changes are added to increase the flexibility
and decrease the redundancy for working with the intended
database. The database can be partitioned into three parts:
database and NoSQL database on IoT data. The main metrics the sensors part, the location longitude and latitude part and
in these experiments are the response time and the size of the the timestamp part. The first part is concerned with adding
database. This evaluation is performed in three main parts. a new sensor to a specific station. This part is a multi-value
First, it examines the impact of increasing workloads on the part so it can be changed from adding a new field for each new
two databases. This part helps to compare the performance of sensor to add a new record for each new sensor by converting
the two databases handling a large scale of IoT data. Second, the sensors data fields (i.e. the number of fields is equal to
it tests the effect of improving the capabilities of the cloud the number of all sensors) to only two fields: sensor_id field
instance on increasing the performance of the two databases. and sensor_data field. This change increases flexibility as
It helps in deciding which database can save resources better. it allows increasing any number of sensors to the database
Third, it proposes a prediction model that is concluded from without affecting its structure.
statistical analysis on the measured data of the introduced The second part is concerned with storing the longitude
experiments. This part applies two approaches of prediction and latitude of the location. This part can be normalized into
and compares between them to estimate the latency of the an independent table or collection that contains three fields:
data; in terms of the data size and instance performance. station_id, longitude and latitude for each station location in
It also evaluates these two estimation approaches and defines addition to the original table or collection but with replacing
which one is more accurate. The proposed prediction model longitude and latitude fields with station_id field. This change
provides the flexibility to evaluate and compare the two types reduces the redundancy in the original data as for every
of databases on any size of data and any instance. It selects sensor data insertion to a table or collection there is no need
the DBMS with low estimated latency. to add the station location. The third part, the timestamp
part, is concerned with instant receiving of sensor data. This
B. SOFTWARE AND HARDWARE part of data is important and must be added to the original
This section discusses the utilized software and hardware in table or collection.
the proposed performance evaluation. Node.JS LTS version With these changes, the database becomes more flexible
10.16.3 and NPM version 6.10.2 are used to process the col- to deal with MySQL and MongoDB DBMSs. After editing
lected data. Ubuntu 16.04 LTS version is used as an operating the database, it becomes able to receive and store data from
system to setup MongoDB, MySQL and Node.JS. any number of towns, stations and sensors; besides saving the
Elastic Compute Cloud (EC2) is used during the imple- locations of every station the longitude and the latitude with
mentation of this comparison. EC2 remains a core service Date and Time of every record.
of the AWS cloud platform. It provides a customer with
an opportunity to build and host a software system on the 2) MYSQL DATABASE SETUP
Amazon virtual servers (EC2 Instances). EC2 is a virtual MySQL server 8.0.11 is the version that is used in this evalua-
private server (VPS) within a cloud, where storage can be tion. Table 2 shows the SQL statements that are used to create
resizable and almost unlimited. With Ec2 service, three types the tested MySQL database schema specifying the structure
of instances were used t3.large (referred to as VM1), t3.xlarge of a database for managing a series of town’s base stations
(referred to as VM2) and t3.2xlarge (referred to as VM3). The and the related sensors. Two tables are created for MySQL
T3 instances feature is the Intel Xeon Platinum 8000 series schema: station_location and town_name.
(Skylake-SP) processor with a sustained all core Turbo CPU The station_location table is to save the location of every
clock speed of up to 3.1 GHz. Additionally, there is a support station. Referential integrity constraint is implemented with
for the new Intel Advanced Vector Extensions 512 (AVX- FOREIGN KEY to stop any station to be inserted in the town
512) instruction set [33]. Table 1 shows the specifications of table without being mentioned in the stations_locations. The
the instances. main target of dividing the dataset in two tables is to reduce
the data redundancy.
C. DATABASE SETUP
1) IoT BENCHMARK 3) MONGODB DATABASE SETUP
The pollution database [34] is used as a base for this MongoDB version 4.2 is the current stable release ver-
paper. This database is based on collecting information about sion that is applied in this comparison. As in MySQL, two

VOLUME 8, 2020 110659

 M. Eyada et al.: Performance Evaluation of IoT Data Management using MongoDB versus MySQL Databases

TABLE 2. MySQL creation queries. TABLE 4. Database description.

TABLE 3. MongoDB creation queries.

FIGURE 2. One sensor scenario.

and located within one town. The structure of the proposed

database is described in Table 4. Each station collects and
buffers data every second from its related sensor nodes and
sends it to the cloud with a rate of 1000 records per trans-
mission. Several scenarios have been introduced to illus-
trate the most important database operations. The proposed
experiments scenarios were carried out based on increasing
the number of sensors connected to each station from 1 to
12 sensors. This leads to increasing number of sensors for the
collections are created for MongoDB. The first one saves experiments from 4 to 48 by increasing one sensor per station
the location of every station, so no record can be inserted in in each experiment.
town_name collection from any station that is not inserted in
stations_locations collection. B. IMPACT OF INCREASING THE NUMBER OF SENSOR
The second collection is the sensors table of all sensors NODES ON THE INSERTION OPERATION
in every station in the town. As stations_locations collection, Increasing the workloads in the proposed experiments is rep-
sensors table controls the insertion query in the town_name resented by increasing the number of sensors connected to
collection. Thus, there is no record from the sensor can be each station. In each experiment, each station collects the data
inserted when it is not available in sensors collection first. But from all its related connected sensors in data records. The data
in MongoDB, the insertion is controlled with insertion query record contains readings of all the related sensors at a specific
and is not automatically inserted as in the MySQL from the time. These collected data records are inserted as records in
creation query. MySQL and as record objects in MongoDB. To shed light
MongoDB as NoSQL DBMS has three models to deal with on the effect of increasing the number of connected sensors
data set: reference, embedded and hybrid. The hybrid model on the number of inserted records in both databases, two
occupies less size of memory than the other reference and scenarios are introduced in detail.
embedded models [27]. Table 3 shows the creation collection In the first scenario, one sensor node is connected to each
statements for the tested MongoDB database based on refer- station as shown in Figure 2. After the stations collect data
ence and hybrid models. from the sensor nodes, each station sends its data with a rate
of 1000 data records per transmission to the cloud for process-
IV. EXPERIMENT AND RESULT ing and insertion operations. Four stations are transmitting
A. EXPERIMENT ANALYSIS 4000 data records at a time. Each record contains data from
The proposed experiments manage the data collected through one sensor node. Each record is inserted in one record in case
a fixed number of stations (i. e. in this case four stations) of MySQL and one record object in case of MongoDB. The

110660 VOLUME 8, 2020

M. Eyada et al.: Performance Evaluation of IoT Data Management using MongoDB versus MySQL Databases

TABLE 5. The latency of the insertion operation.

FIGURE 3. Two sensors scenario.

transmitted data is inserted in 4000 records in MySQL and

4000 record objects in MongoDB.
In the second scenario, two sensor nodes are connected to
each station as shown in Figure 3. The four stations transmit
4000 data records at a time for the cloud. Each record contains
data from two sensor nodes. In MySQL, the data for each
sensor needs to be inserted in a separate record. It needs
8000 records to insert the transmitted data. Unlike MongoDB,
all sensors data can be inserted into one record object. So,
it needs 4000 record objects to insert the transmitted data.
The remaining scenarios are introduced in Table 5 to show
the number of inserted records in each case. Due to the
structured schema of MySQL, the addition of a new sensor
is represented by adding an independent record for this sen-
sor measurement. Therefore, the data record received from
the station is separated into independent records one per
each sensor data. While MongoDB is a document-oriented
database and the addition of new sensor results in a docu- FIGURE 4. Insert query Instance T3.large.
ment with a different structure for the newly inserted record
object. This means the data record received from the station
is inserted in one record object. Increasing the number of
Figure 4 shows the impact of increasing workloads on the
connected sensors means increasing the number of records
latency using VM1 with a performance of 6.2 GHz. It shows
(i.e. vertical increasing) in MySQL and increasing the width
that in the first scenario MongoDB decreases the latency
of the record object (i.e. horizontal increasing) in MongoDB.
compared with MySQL by 82.3%. By increasing the number
of connected sensors and in the last scenario, MongoDB
C. IMPACT OF INCREASING THE WORKLOADS ON THE decreases the latency compared with MySQL by 97.7%.
LATENCY Figure 5 shows the impact of increasing the same
In the proposed scenarios, increasing the workloads is based workloads on the latency using VM2 with a performance
on increasing the number of connected sensors per station of 12.4 GHz. Using VM2 improves the performance of
from 1 to 12 sensors and therefore increasing the number MySQL with 29% over using VM1while it improves the
of insertion operations. These insertion operations are imple- performance of MongoDB with 23% over using VM1. It also
mented on three instances with different specifications VM1, shows that in the first scenario, MongoDB decreases the
VM2 and VM3; which were described in Table 1. latency compared with MySQL by 83.3%. Also, in the last
MySQL accepts from 4000 records; in the first insertion scenario MongoDB decreases the latency compared with
operation, to 48000 records; in the last insertion operations. MySQL by 98%.
On the other side, MongoDB accepts 4000 records in all cases Figure 6 shows that MongoDB decreases the latency com-
from the first to the last operations. Table 5 shows the latency pared with MySQL by 83.7% in the first scenario and by
of these insertion operations in all cases. 97.5% in the last scenario using VM3 with a performance

VOLUME 8, 2020 110661

 M. Eyada et al.: Performance Evaluation of IoT Data Management using MongoDB versus MySQL Databases

FIGURE 5. Insert query Instance T3.xlarge. FIGURE 7. The storage size of MongoDB based hybrid model, MongoDB
based reference model, and MySQL databases.

TABLE 6. Database size comparison between MongoDB based Hybrid

Model, MongoDB based reference model, and MySQL.

FIGURE 6. Insert query Instance T3.2xlarge.

of 24.8 GHz. Using VM3 improves the performance of

MySQL with 43% over using VM1 and with 26% over using
VM2. Besides, it improves the performance of MongoDB
with 35% over using VM1 and with 8% over using VM2.
The results show that through increasing the number of
connected sensors per station and accordingly increasing
workloads, the difference in latency between MongoDB and
MySQL is also increased in favor of MongoDB. Moreover,
the results show that using MongoDB on VM1with low capa-
bilities has average improvement in latency with 43% over in MongoDB based reference model, the database size is
MySQL on VM3 with high capabilities. This ensures that increased linearly but the rate of increase is higher than
MySQL needs instances with high capabilities to work well the database size of MongoDB based hybrid model. On the
on the contrary to MongoDB which works very well with all other side in MySQL, the database size has periods of lin-
instances capabilities. ear increase and periods of relative stability. According to
scenarios from S1 to S3 and from S7 to S8, database size
D. IMPACT OF INCREASING THE WORKLOADS ON THE is increased linearly with increasing the workloads. Whereas
DATABASE SIZE the scenarios from S3 to S7 and from S8 to S12, database
In this section, two types of schemas for the MongoDB size is relatively stable with increasing the workloads. This
database are examined and compared with MySQL database behavior of MySQL is due to its storage nature [4].
in terms of the database size: hybrid and reference models. The results show that MySQL outperforms the other two
Table 6 shows the effect of the same experiments on the databases in case of inserting small number of records and
database size (i.e. size of the stored data). also at the end of each stability period as shown in the high-
Figure 7 shows the impact of increasing the workloads on lighted records of table 6 scenarios S1, S2, S6, S7, S11 and
the three databases: MongoDB based hybrid model, Mon- S12. For inserting small number of records up to scenario S2,
goDB based reference model and MySQL databases. It shows MySQL is much more efficient than the MongoDB based
that in MongoDB based hybrid model, the database size hybrid model by 15% and the MongoDB based reference
is increased linearly with increasing the workloads. Also, model by 33%. Also, by increasing the number of inserted

110662 VOLUME 8, 2020

M. Eyada et al.: Performance Evaluation of IoT Data Management using MongoDB versus MySQL Databases

FIGURE 8. MySQL records in the case of one sensor scenario.

FIGURE 10. MySQL records in the case of two sensor scenario.

FIGURE 9. MongoDB record objects in the case of one sensor scenario.

records and at the end of each stability period; scenarios

from S6 to S7 and from S11 to S12, MySQL outperforms
the other two databases. Whereas, at the beginning of each
stability period; scenarios from S3 to S5 and from S8 to S10,
FIGURE 11. MongoDB record objects in the case of two sensor scenario.
MongoDB based hybrid model outperforms the other two
databases. In addition, the results also show that MongoDB
based hybrid model has average improvement in the database TABLE 7. The latency of the selection operation.

size by 6.6% over MySQL and 51.5% over MongoDB based

reference model.

E. IMPACT OF INCREASING THE NUMBER OF SENSOR

NODES ON THE SELECTION OPERATION
In this section, the selection operation of both databases is
evaluated in terms of latency. It is based on retrieving the
data of all the connected sensors at a specific time. The
selection operations are started by selecting the data that is
collected based on one sensor per each station (i.e. the first
scenario) and end with selecting the data that is collected
based on 12 sensors per each station (i.e. the last scenario),
twelve scenarios and selection operations are implemented.
The data retrieved in the case of one connected sensor per
station is shown in Figures 8 and 9. Also, the data retrieved
in the case of two connected sensors per station is shown
in Figures 10 and 11. Table 7 shows the impact of the selection
operations on the latency in different instances VM1, VM2,
and VM3.
The selection operation considers a connection time that The results show that VM3 has an average improvement of
connects the DBMS with the database server before retrieving the selection latency with 63.2% over VM1 and with 22.5%
the data. In the proposed experiments, this time is 112 ms in over VM2 in case of MySQL. Also, VM3 has an average
case of MongoDB, and 5 ms in case of MySQL. Figures 12, improvement of the selection latency with 38% over VM1 and
13, and 14 show that MongoDB is faster than MySQL in all with 12.3% over VM2 in case of MongoDB. These results
scenarios when implemented on instances VM1, VM2, and ensure that MySQL is affected; more than MongoDB, by the
VM3. As a result, to increase the performance of the instance, instance performance.
the latency decreases, the same operation is processed on the The comprehensive results ensure that MongoDB database
three instances but with better performance. performance can significantly outperform the MySQL

VOLUME 8, 2020 110663

 M. Eyada et al.: Performance Evaluation of IoT Data Management using MongoDB versus MySQL Databases

is to evaluate the two estimation approaches and select the

appropriate one.

A. DATASETS AND METHODS

1) DATASET
The analysis starts by establishing the equations models from
a previously measured dataset which contains three param-
eters; namely latency in milliseconds, data size in KB and
instance performance in GHz.
The instance performance was calculated as the product of
the virtual central processing unit (vCPU) and the CPU clock
speed as shown in Table 1. The data size was calculated as the
FIGURE 12. Select query Instance T3.large.
product of the calculated average size of one record and the
number of records from Table 6. Table 8 shows the dataset
information that is needed in this analysis for both MySQL
and MongoDB databases.

2) MULTIPLE REGRESSION ANALYSIS

This section is proposed to compare two approaches for esti-
mating the latency of the database: Multiple Linear Regres-
sion and Multiple Non-linear Regression. This estimation
is used to evaluate the performance of both MySQL and
MongoDB databases. Residual value is the metric that is used
in this comparison.

3) LINEAR REGRESSION
FIGURE 13. Select query Instance T3.xlarge.
A simple Linear Regression is used to illustrate the relation
between the dependent variable y and the independent vari-
able x based on the regression equation [36].
y = a1 x1 + a0 (1)
The proposed evaluation needs to find a relation between
three variables: latency, data size and instance performance.
The dependent variable (latency) is related to two indepen-
dent variables (data size and instance performance). In this
case, the Multiple Linear Regression can be implemented as
follows [35]:
y = a2 x2 + a1 x1 + a0 (2)

FIGURE 14. Select query Instance T3.2xlarge. where x1 is the data size, x2 is the instance performance and y
is the latency. Table 9 shows the parameters notations which
are used in this evaluation.
database dealing with a wide range of IoT data. In addition, The determiner method is used to solve the previous equa-
the results show that MySQL can work well with high- tion and get the final equation of prediction for both Mon-
performance instance, unlike MongoDB which works very goDB and MySQL latency as [36]:
well with all instance specifications. 
 y x1 x2 1 

 n n n
 P P P 
V. STATISTICAL ANALYSIS 
 i=1 y x 1 x2 n 
i=1 i=1
In this section, a statistical analysis is introduced to estimate  n n n n

=0
P P 2
P P (3)
the latency of data from a measured data size and instance 
 x1 y x1 x1 x2 y 
performance using two approaches: Multiple Linear Regres-  i=1
n
i=1
n
i=1
n
i=1 
n
P 
x22
P P P
sion and Multiple Non-linear Regression. This estimation 
 x2 y x1 x2 x2 
is implemented on both database types, MySQL and Mon- i=1 i=1 i=1 i=1

goDB, for twofold aims. The first aim is to compare the By substituting in equation (3) with values from Table 8 to
two types of databases in terms of latency. The second aim get the values a2 , a1 , and a0 .

110664 VOLUME 8, 2020

M. Eyada et al.: Performance Evaluation of IoT Data Management using MongoDB versus MySQL Databases

TABLE 8. Dataset information: latency, data Size, and Instance TABLE 9. Parameters’ notations used in the proposed evaluation.
performance.

The determiner method is used to solve the previous equa-

tion and get the final equation of prediction for both Mon-
goDB and MySQL latency [37] as shown in equation (7).
By substitution; in equation (7), with values from Table 8
to get the values a5 , a4, a3 , a2 , a1 , and a0 .
MongoDB equation will be:
y = 0.446x22 + 0.004x1 x2 + 0.093x1 −11.305x2 +76.0968
MongoDB equation will be:
(8)
y = −3.83x2 + 0.111x1 + 35.35 (4)
MySQL equation will be:
MySQL equation will be: y = 3.673x22 −0.0424x1 x2 +0.694x1 −98.236x2 +544.357
y = −27.257x2 + 0.581x1 + 310.345 (5) (9)

5) R-SQUARED AND ADJUSTED R-SQUARED

4) NON-LINEAR REGRESSION R-Squared is a statistical measure that predicts the future
Non-Linear Regression is another approach that can be used outcome of an investment and how closely it aligns to a single
to estimate the database latency based on the perception of measured model [8], [39]. It is represented as the difference
both data size and instance performance. In this case, the Mul- between the observed value of the dependent variable for
tiple Non-linear Regression can be implemented as follows the observation yi and the estimated value of the dependent
[7]: variable for the observation ŷi [7]:
n
y = a5 x12 + a4 x22 + a3 x1 x2 + a2 x1 + a1 x2 + a0 (ŷi − ȳ)2
P
(6)
SSE i=1
R2 = = n (10)
where x1 is the data size, x2 is the instance performance and SST
(yi − ȳ)2
P
y is the latency. i=1

x12 x22
 
 y x1 x2x1 x2 1 
 n n n n n n
 P 
x12 x22
P P P P P
y x1 ∗ x2 x1 x2 n

 
 i=1 i=1 i=1 i=1 i=1 i=1 
 n n n n n n n 
 P 2
x14 x12 x22 x12 x1 x2 x12 x1 x12 x2 2
P P P P P P 
x y x1 y 
 i=1 1

i=1 i=1 i=1 i=1 i=1 i=1

 n n n n n n n

 P 2
x12 x22 x24 x22 x1 x2 x22 x1 x22 x2 x22 y 
P P P P P P 
 x2 y
=0 (7)

 ni=1 i=1
n n
i=1 i=1
n
i=1
n
i=1
n
i=1
n
x12 x1 x2 x22 x1 x2 x12 x22 x2 x12 x1 x22
P P P P P P P 

 x1 x2 y x1 x2 y 
 i=1 i=1 i=1 i=1 i=1 i=1 i=1 
 P n n n n n n n 
x12 x1 x22 x1 x12 x2 x12
P P P P P P

 x1 y x1 x2 x1 y 
 i=1 i=1 i=1 i=1 i=1 i=1 i=1 
 P n n n n n n n 
x12 x2 x22 x2 x22 x1 x22
P P P P P P

 x2 y x1 x2 x2 y 
i=1 i=1 i=1 i=1 i=1 i=1 i=1

VOLUME 8, 2020 110665

 M. Eyada et al.: Performance Evaluation of IoT Data Management using MongoDB versus MySQL Databases

TABLE 10. The values of R2adj in different cases. 2 in MYSQL cases.

TABLE 11. The values of Radj

results show that the Non-linear Regression approach pro-

where R2 is R-Squared, ȳ is the average of the estimated vides better estimation results for MongoDB. On the other
values, SSE is the sum of squares error and SST is the sum of hand, after dividing the dataset into two sets, the two Non-
squares total. linear Regression equations of the divided dataset is the best
SSE and SST are calculated as follows [7]: for MySQL with adjusted R-squared 0.959, and 0.951 for
n
X both the sets of data.
SSE = (ŷi − ȳ)2 (11)
i=1 VI. CONCLUSION AND FUTURE WORK
Xn In this paper, a comparison between MongoDB and MySQL
SST = (yi − ȳ)2 (12) has been implemented to evaluate the performance of both
i=1 databases to deal with a large scale of heterogeneous IoT
The adjusted R-squared compares the correlation of the data. Several scenarios were introduced to perform this target.
investment to several measured models as [7]: The results showed that increasing workloads in case of using
MySQL leads to a considerable loss in performance greater
(1 − R2 )(n − 1)
 
2 than the case of using MongoDB. Three instances with
Radj = 1 − (13)
n−k −1 different capabilities are used to compare the performance
where R2adj is the adjusted R-squared. improvement for both databases. The proposed evaluation
Table 10 shows the adjusted R-squared values for both helps to differentiate between the two types of databases
databases based on the two approaches of regressions. High based on database workloads, the available resources capa-
adjusted R-squared means good performance in the estima- bilities, the number of network connected-sensors and the
tion processand vice versa. The results show that both the needed requirements. The results showed that there are cases
Linear Regression and Non-linear Regression approaches where MongoDB is better than MySQL in terms of both
provide better estimation in case of MongoDB than in case latency and database size. On the other hand, there are other
of MySQL. Unfortunately, adjusted R-squared is very low cases where MongoDB is better in terms of latency and
0.636 in case of MySQL based on Linear Regression and MySQL is better in terms of database size; based on the
cannot be used. To solve this problem, the used dataset is database workload and the number of connected sensors.
divided in two sets: the first one is from 1 to 6 sensors and The differentiation in these cases is based on the available
the second set is from 7 to 12 sensors. resources capabilities.
MySQL first set (1:6) Linear Regressions Equation: Two data models of the MongoDB are introduced: refer-
ence and hybrid. The results show that the hybrid model is
y = −6.238x2 + 0.366x1 + 135.383 (14) better than the reference model and also better than MySQL
in terms of database size. Two prediction models of estimat-
MySQL second set (7:12) Linear Regressions Equation:
ing the latency from measured data were introduced based
y = −48.276x2 + 0.827x1 + 220.148 (15) on two approaches: linear Regression and Non-linear Regres-
sion. The results showed that Non-linear Regression is better
MySQL first set (1:6) Non-linear Regressions Equation: than Linear Regression for estimating the latency in both
y = 0.473x22 − 0.015x1 x2 + 0.399x1 −12.659x2 +141.239 MongoDB and MySQL. A benefit from applying the regres-
sions approach is to select the appropriate DBMS based on
(16)
the low estimated latency.
MySQL second set (7:12) Non-linear Regressions Equa- As a future work for this paper, a hybrid model between
tion: MongoDB and MySQL can be implemented and compared
against MySQL and MongoDB. Also, the evaluation process
y = 6.874x22 −0.073x1 x2 +0.9264x1 −150.557x2 +591.552 can be performed using big-data workloads. Finally, the esti-
(17) mation process can be implemented using the neural network
instead of the Regression process.
Table 11 shows that dividing the dataset into two sets
improving the adjusted R-squared and therefore the perfor-
REFERENCES
mance of estimation in the case of MySQL. However Non-
[1] L. Gutierrez-Madronal, L. La Blunda, M. F. Wagner, and I. Medina-Bulo,
linear Regression approach still provides good performance ‘‘Test event generation for a fall-detection IoT system,’’ IEEE Internet
in estimation 0.974 than the Linear Regression approach. The Things J., vol. 6, no. 4, pp. 6642–6651, Aug. 2019.

110666 VOLUME 8, 2020

M. Eyada et al.: Performance Evaluation of IoT Data Management using MongoDB versus MySQL Databases

[2] B. Diene, J. Rodrigues, O. Diallo, E. Ndoye, and V. V. Korotaev, ‘‘Data [26] B. Maity, S. Sen, and N. C. Debnath, ‘‘Retracted: Challenges of implement-
management techniques for Internet of Things,’’ Mech. Syst. Signal Pro- ing data warehouse in MongoDB environment,’’ J. Fundam. Appl. Sci.,
cess., vol. 138, Apr. 2020, Art. no. 106564. vol. 10, no. 4S, pp. 222–228, 2018.
[3] S. Kontogiannis, C. Asiminidis, and G. Kokkonis, ‘‘Comparing relational [27] C. Gyorödi, R. Gyorödi, and R. Sotoc, ‘‘A comparative study of relational
and NoSQL databases for carrying IoT data,’’ J. Sci. Eng. Res., vol. 6, no. 1, and non-relational database models in a Web-based application,’’ Int.
pp. 125–133, 2019. J. Adv. Comput. Sci. Appl., vol. 6, no. 11, pp. 78–83, 2015.
[4] R. Čerešnák and M. Kvet, ‘‘Comparison of query performance in relational [28] L. Kumar, S. Rajawat, and K. Joshi, ‘‘Comparative analysis of NoSQL
a non-relation databases,’’ Transp. Res. Procedia, vol. 40, pp. 170–177, (MongoDB) with MySQL database,’’ Int. J. Modern Trends Eng. Res.,
Jan. 2019. vol. 2, no. 5, pp. 120–127, May 2015.
[5] B. Jose and S. Abraham, ‘‘Analysis of aggregate functions in relational [29] Z. Bicevska and I. Oditis, ‘‘Towards NoSQL-based data warehouse solu-
databases and NoSQL databases,’’ Int. J. Comput. Sci. Eng., vol. 6, no. 6, tions,’’ Procedia Comput. Sci., vol. 104, pp. 104–111, Jan. 2017.
pp. 74–79, Jul. 2018. [30] C. Li and J. Gu, ‘‘An integration approach of hybrid databases based on
[6] W. Ali, M. U. Shafique, M. A. Majeed, and A. Raza, ‘‘Comparison between SQL in cloud computing environment,’’ Softw., Pract. Exper., vol. 49, no. 3,
SQL and NoSQL databases and their relationship with big data analytics,’’ pp. 401–422, Mar. 2019.
Asian J. Res. Comput. Sci., pp. 1–10, Oct. 2019. [31] Y. Rasheed, M. Qutqut, and F. Almasalha, ‘‘Overview of the current status
[7] J. Dizdarević, F. Carpio, A. Jukan, and X. Masip-Bruin, ‘‘A survey of of NoSQL database,’’ Int. J. Comput. Sci. Netw. Secur., vol. 19, no. 4,
communication protocols for Internet of Things and related challenges of pp. 47–53, Apr. 2019.
fog and cloud computing integration,’’ ACM Comput. Surv., vol. 51, no. 6, [32] C. Asiminidis, G. Kokkonis, and S. Kontogiannis, ‘‘Database systems
pp. 1–29, Feb. 2019. performance evaluation for IoT applications,’’ Int. J. Database Manage.
[8] Y. Liu, K. Akram Hassan, M. Karlsson, Z. Pang, and S. Gong, ‘‘A data- Syst., vol. 10, no. 06, pp. 1–14, Dec. 2018.
centric Internet of Things framework based on azure cloud,’’ IEEE Access, [33] T3. US. Accessed: Jun. 25, 2019. [Online]. Available: https://aws.amazon.
vol. 7, pp. 53839–53858, 2019. com/ec2/instance-types/t3
[9] A. Celesti, A. Galletta, L. Carnevale, M. Fazio, A. Lay-Ekuakille, and [34] Romania. Pollution Measurements for the City of Brasov in
M. Villari, ‘‘An IoT cloud system for traffic monitoring and vehicular acci- Romania. Accessed: Jun. 25, 2019. [Online]. Available: http://iot.ee.
dents prevention based on mobile sensor data processing,’’ IEEE Sensors surrey.ac.uk:8080/datasets.html#weather
J., vol. 18, no. 12, pp. 4795–4802, Jun. 2018. [35] J. Moon, S. Kum, and S. Lee, ‘‘A heterogeneous IoT data analysis frame-
work with collaboration of edge-cloud computing: Focusing on indoor
[10] Z. Daher and H. Hajjdiab, ‘‘Cloud storage comparative analysis Amazon
PM10 and PM2.5 status prediction,’’ Sensors, vol. 19, no. 14, p. 3038,
simple storage vs. microsoft azure blob storage,’’ Int. J. Mach. Learn.
Jul. 2019.
Comput., vol. 8, no. 1, pp. 85–89, Feb. 2018.
[36] D. J. Hand, ‘‘Statistical challenges of administrative and transaction data,’’
[11] V. M. Ionescu and J. M. Lopez-Guede, ‘‘Comparing Google Cloud
J. Roy. Stat. Soc., A (Statist. Soc.), vol. 181, no. 3, pp. 555–605, Jun. 2018.
and Microsoft Azure platforms for undergraduate laboratory use,’’ in
[37] B. Shyti and D. Valera, ‘‘The regression model for the statistical analysis of
Proc. Int. Workshop Soft Comput. Models Ind. Environ. Appl. (SOCO),
albanian economy,’’ Int. J. Math. Trends Technol., vol. 62, no. 2, pp. 90–96,
San Sebastián, Spain, Oct. 2016, pp. 795–802.
Oct. 2018.
[12] J. Kaur and M. Sharma, ‘‘Extending IoTs into the cloud-based platform for [38] M. El Genidy, ‘‘Multiple nonlinear regression of the Markovian arrival
examining Amazon Web services,’’ in Examining Cloud Computing Tech- process for estimating the daily global solar radiation,’’ Commun. Statist.-
nologies Through the Internet of Things. Hershey, PA, USA: IGI Global, Theory Methods, vol. 48, no. 22, pp. 5427–5444, Oct. 2018.
2018, pp. 216–227. [Online]. Available: http://www.igi-global.com [39] M. M. El Genidy, ‘‘Multiple non linear regression model for the maximum
[13] M. Laaziri, K. Benmoussa, S. Khoulji, and M. L. Kerkeb, ‘‘A Compara- number of migratory bird types during migration years,’’ Commun. Statist.-
tive study of PHP frameworks performance,’’ Procedia Manuf., vol. 32, Theory Methods, vol. 46, no. 16, pp. 7969–7975, Aug. 2017.
pp. 864–871, Jan. 2019.
[14] J. M. Volk and M. A. Turner, ‘‘PRMS-Python: A Python framework for
programmatic PRMS modeling and access to its data structures,’’ Environ.
Model. Softw., vol. 114, pp. 152–165, Apr. 2019.
[15] D. Laksono, ‘‘Testing spatial data deliverance in SQL and NoSQL database
using NodeJS fullstack Web app,’’ in Proc. 4th Int. Conf. Sci. Technol.
(ICST), Yogyakarta, Indonesia, Aug. 2018, pp. 1–5.
[16] M. Ohyver, J. V. Moniaga, I. Sungkawa, B. E. Subagyo, and I. A. Chandra, MAHMOUD EYADA received the B.Sc. degree in
‘‘The comparison firebase realtime database and MySQL database perfor- computer and mathematical science from the Fac-
mance using Wilcoxon signed-rank test,’’ Procedia Comput. Sci., vol. 157, ulty of Science, Port Said University, in 2016. He
pp. 396–405, Jan. 2019. is a Server Side Back-END Developer, a Database
[17] L. Bienvenu and R. Downey, ‘‘On low for speed oracles,’’ J. Comput. Syst. Analyzer, and an IoT Systems Designer work
Sci., vol. 108, pp. 49–63, Mar. 2020. as a freelancer with worldwide projects since
[18] P. Senellart, L. Jachiet, S. Maniu, and Y. Ramusat, ‘‘ProvSQL: Provenance his last year in college. His research interests
and probability management in postgreSQL,’’ Proc. VLDB Endowment, include database management systems, data anal-
vol. 11, no. 12, pp. 2034–2037, Aug. 2018. ysis, the Internet of Things, Internet protocols,
[19] R. R. Parmar and S. Roy, ‘‘MongoDB as an efficient graph database: wireless sensor networks, and embedded systems.
An application of document oriented NOSQL database,’’ Data Intensive
Comput. Appl. Big Data, vol. 29, pp. 331–358, Feb. 2018.
[20] M. Ben Brahim, W. Drira, F. Filali, and N. Hamdi, ‘‘Spatial data extension
for cassandra NoSQL database,’’ J. Big Data, vol. 3, no. 1, Dec. 2016.
[21] H. V. Le and A. Takasu, ‘‘G-HBase: A high performance geographical
database based on HBase,’’ IEICE Trans. Inf. Syst., vol. E101.D, no. 4,
pp. 1053–1065, 2018.
[22] C. Asiminidis, G. Kokkonis, and S. Kontogiannis, ‘‘Managing IoT data
WALAA SABER received the B.Sc. and M.Sc.
using relational schema and JSON fields, a comparative study,’’ IOSR degrees in computer and control engineering from
J. Comput. Eng., vol. 20, no. 6, pp. 46–52, 2019. Suez Canal University in 2001 and 2008, respec-
[23] V. Jain, ‘‘MongoDB and NoSQL databases,’’ Int. J. Comput. Appl., tively, and the Ph.D. degree in computer and con-
vol. 167, no. 10, pp. 8887–8975, 2017. trol engineering from Port Said University, Egypt,
[24] J. Fjällid, ‘‘A comparative study of databases for storing sensor data,’’ in 2014. She is an Assistant Professor with Electri-
M.S. thesis, Dept. Comp. Sci., Tech. Univ., Stockholm, Sweden, 2019. cal Engineering Department, Port Said University.
[25] Y.-S. Kang, I.-H. Park, J. Rhee, and Y.-H. Lee, ‘‘MongoDB-based repos- Her research interests include computer network-
itory design for IoT-generated RFID/Sensor big data,’’ IEEE Sensors J., ing, including cloud computing, clustering, and the
vol. 16, no. 2, pp. 485–497, Jan. 2016. Internet of Things.

VOLUME 8, 2020 110667

 M. Eyada et al.: Performance Evaluation of IoT Data Management using MongoDB versus MySQL Databases

MOHAMMED M. EL GENIDY received the FATHY AMER received the B.Sc. degree in mil-
Ph.D. degree in statistics and computer science itary science and the B.Sc. degree in electrical
from Mathematics Department, Faculty of Sci- and communication engineering from the Military
ence, Mansoura University, Mansoura, Egypt, Technical College (MTC), Cairo, Egypt, in 1970,
in 2001. He is currently an Assistant Professor the M.Sc. degree in electrical, mechatronics, and
of statistics with the Faculty of Science, Mathe- communication engineering (major area: electron-
matics and Computer Science Department, Port ics and communication) from Azhar University,
Said University, Port Said, Egypt. His research Cairo, in 1985, and the Ph.D. degree in philoso-
interests include statistics, probability, order statis- phy in computer science from Computer Science
tics, queues theory, regression, estimation, distri- Department, Azhar University. From May 1970 to
butions, mathematical programming, statistical tests, and hypotheses test. June 1993, he was an Engineer Officer and a Lecturer with the Military
Technical College and various places in the armed forces in the field of
computers, information technology, electrical engineering, communications,
and electronic insurance. From September 2012 to August 2013, he was the
Dean of the Higher Institute of Engineering and Technology, Obour, Egypt.
From September 2013 to September 2016, he was a Professor Emeritus with
the Department of Information Technology, Faculty of Computers and Infor-
mation, Cairo University. Since October 2016, he has been the Vice Dean of
Community Affairs and the Environment, Misr University for Science and
Technology. He is currently a Professor with the 6th of October University.
His research interests include local, expanding and international networking,
databases, multimedia, the Internet of Things, and information technology.

110668 VOLUME 8, 2020