See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/311488672

SQL: From Traditional Databases to Big Data

Conference Paper · February 2016

DOI: 10.1145/2839509.2844560

CITATIONS READS
55 15,271

3 authors, including:

Michell Queiroz
University of Antwerp
5 PUBLICATIONS 79 CITATIONS

SEE PROFILE

All content following this page was uploaded by Michell Queiroz on 14 April 2020.

The user has requested enhancement of the downloaded file.

 SQL: From Traditional Databases to Big Data
Yasin N. Silva Isadora Almeida Michell Queiroz
Arizona State University Arizona State University Arizona State University
[email protected] [email protected] [email protected]

ABSTRACT Standardization (ISO); and (4) it is, to some extent, portable among
The Structured Query Language (SQL) is the main programing different database systems. Even though SQL was initially
language designed to manage data stored in database systems. proposed for traditional relational databases, it has also been found
While SQL was initially used only with relational database to be an effective language in several new types of database
management systems (RDBMS), its use has been significantly systems, particularly Big Data management systems (BDMSs) that
extended with the advent of new types of database systems. process large, fast and diverse data. While BDMSs have
Specifically, SQL has been found to be a powerful query language significantly changed the database landscape and traditional
in highly distributed and scalable systems that process Big Data, RDBMs represent now only a small fraction of it, most books and
i.e., datasets with high volume, velocity and variety. While database courses only present SQL in the context of RDBMs.
traditional relational databases represent now only a small fraction In this paper we propose learning SQL as a powerful language that
of the database systems landscape, most database courses that cover can be used to query a wide range of database systems, from
SQL consider only the use of SQL in the context of traditional traditional relational databases to modern Big Data systems. The
relational systems. In this paper, we propose teaching SQL as a main contributions of this paper are:
general language that can be used in a broad range of database
systems from traditional RDBMSs to Big Data systems. This paper  The description of the new database landscape and the
presents well-structured guidelines to introduce SQL in the context identification of broad types of Big Data systems where SQL
of new types of database systems including MapReduce, NoSQL can be effectively used.
and NewSQL. A key contribution of this paper is the description of  The presentation of well-structured guidelines to prepare course
an array of course resources, e.g., virtual machines, sample units that focus on the study of SQL on new types of database
projects, and in-class exercises, to enable a hands-on experience systems including MapReduce, NoSQL and NewSQL.
with SQL across a broad set of modern database systems.  A detailed description of class resources for each unit to enable
a hands-on experience with SQL across a broad set of modern
Categories and Subject Descriptors database systems. These resources include virtual machines,
K.3.2 [Computers and Education]: Computer and Information
data generators, sample programs, projects, and in-class
Science Education
exercises.
General Terms  Files of all the resources made available to instructors [1]. The
Design, Experimentation goal is to enable instructors to use and extend these resources
based on their specific needs.
Keywords The rest of the paper is organized as follows. Section 2 describes
Databases curricula; SQL; structured query language; Big Data
the new database landscape and the types of BDMSs where SQL
1. INTRODUCTION can be used. Sections 3, 4, and 5 present the details of how SQL
The Structured Query Language (SQL) is the most extensively used can be introduced in the context of the new types of database
database language. SQL is composed of a data definition language systems (MapReduce, NoSQL, and NewSQL). These sections
(DDL), which allows the specification of database schemas; a data describe many class resources which will be made available in [1].
manipulation language (DML), which supports operations to Section 6 presents a discussion of the integration of the proposed
retrieve, store, modify and delete data; and a data control language units into database courses. Section 7 concludes the paper.
(DCL), which enables database administrators to configure security
access to databases. Among the most important reasons for SQL’s
2. NEW DATABASE LANDSCAPE
The development and widespread use of highly distributed and
wide adoption are that: (1) it is primarily a declarative language,
scalable systems to process Big Data is considered as one of the
that is, it specifies the logic of a program (what needs to be done)
recent key technological developments in computer science [3, 4,
instead of the control flow (how to do it); (2) it is relatively simple
5, 6]. These systems have significantly extended the database
to learn and understand because it is declarative and uses English
landscape and are currently used in many application scenarios,
statements; (3) it is a standard of the American National Standards
e.g., social media data mining, scientific data analysis,
Institute (ANSI) and the International Organization for
recommendation systems, and web service analysis. BDMSs are
usually composed of clusters of commodity machines and are often
Permission to make digital or hard copies of all or part of this work for
dynamically scalable, i.e., nodes can be added or removed as
personal or classroom use is granted without fee provided that copies are needed. The area of BDMSs has quickly developed and there are
not made or distributed for profit or commercial advantage and that copies now dozens of BDMS-related open-source and commercial
bear this notice and the full citation on the first page. To copy otherwise, products. Some of these systems have proposed their own query
or republish, to post on servers or to redistribute to lists, requires prior languages or application program interfaces but several others have
specific permission and/or a fee. recognized the benefits of using SQL and support it as a query
SIGCSE’16, March 2–5, 2016, Memphis, Tennessee, USA. language. The use of SQL in DBMSs is expanding and some top
Copyright 2016 ACM 1-58113-000-0/00/0010 …$15.00.
database researchers believe that many more systems will move to 3.1 Hive: SQL Queries on Hadoop
support SQL [2]. The authors propose that SQL should be taught in Apache Hive [8] is a system that supports the processing and
the context of the following types of BDMSs in addition to analysis of data stored in Hadoop. Hive allows projecting structure
traditional RDBMSs: onto this data and querying the data using HiveQL, a SQL-like
 MapReduce. It is considered to be one of the main frameworks query language. One of the key features of Hive is that it
for Big Data processing. It enables building highly distributed transparently converts queries specified in HiveQL to MapReduce
programs that run on failure-tolerant and scalable clusters. programs. This enables the user to focus on specifying what the
Apache Hadoop [7] is the most widely used MapReduce query should retrieve instead of specifying a procedural
implementation. Examples of systems that support SQL to MapReduce program in Java. Hive uses indexing structures to
query data in Hadoop are: Hive [8] and Spark (Spark SQL) [9]. accelerate the execution of queries and was particularly built to
support data warehouse applications, which require the analysis of
 NoSQL. These data stores have been designed to provide primarily read-only massive datasets (data that does not change
higher scalability and availability than conventional relational over time). While HiveQL is Hive’s main query language, Hive
databases while supporting a simplified transaction and also allows the use of custom map and reduce functions when this
consistency model. Some examples of NoSQL data stores are: is a more convenient or efficient way to express a given query logic.
MongoDB [10], Apache HBase [11], Google’s BigTable [12],
and Apache Cassandra [13]. While many NoSQL systems do HiveQL supports many of the features of SQL but it does not
not support SQL natively, several systems have been proposed strictly follow a full SQL standard. Hive supports multiple DDL
to enable SQL querying on these systems. Examples of such and DML commands such as CREATE TABLE, SELECT,
systems are: Impala [14], Presto [15] and SlamData [16]. INSERT, UPDATE and DELETE. Moreover, starting with Hive
 NewSQL. These systems aim to have the same levels of 0.13, it is possible to support transactions with full ACID semantics
scalability and availability of NoSQL systems while at the row (record) level.
maintaining the ACID properties (Atomicity, Consistency, We propose the following in-class activity to introduce HiveQL.
Isolation and Durability), relational schema, and SQL query
language of traditional relational databases. Examples of Using Virtual Machines. Since many Big Data systems rely on
NewSQL systems are: VoltDB [17], MemSQL [18], NuoDB distributed architectures, a computer cluster is needed to enable a
[19], and Clustrix [20]. direct interaction with these technologies. Many institutions,
however, do not have such clusters available for teaching. A
3. SQL IN MAPREDUCE SYSTEMS solution that the authors recommend is the use of virtual machines
MapReduce is an extensively used programming framework for (VM) with all the required packages already installed and
processing very large datasets. Apache Hadoop is its most popular configured. In the case of Hive, the authors recommend the use of
implementation. A MapReduce program divides a large dataset into Cloudera’s VM [23] which includes: (1) CentOS Linux as the
independent chunks that can be processed in a parallel fashion over operating system, (2) Hadoop, (3) Hive, and (5) Hue, a web-based
dynamic computer clusters. The overall processing is divided into application that can be used to write and run HiveQL queries.
two main phases: map and reduce, and the framework user needs
to provide the code for each phase. The map and reduce functions Data Generators and Datasets. The queries presented in this
have the following form: section use MStation2, a synthetic dataset of meteorological station
data prepared by the authors. The dataset has two tables: Station
map: (k1,v1) → list(k2,v2) (stationID, zipcode, latitude, longitude) and WeatherReport
reduce: (k2,list(v2)) → list(k3,v3) (stationID, temperature, precipitation, humidity, year, month). The
The input dataset is usually stored on an underlying distributed file data generator and a sample dataset are available in [1]. The
system and different chunks of this dataset are processed in parallel generator can be modified to produce datasets of different sizes and
by different map tasks. Each map task processes an input chunk one data distributions for additional exercises or projects.
record or line at the time. Each map function call receives a key- Loading and Querying the Data. The commands for this activity
value pair (k1,v1) and generates a list of (k2,v2) pairs. Each are listed in Fig. 1. To start interacting with Hive, students can open
generated key-value pair is sent over the network to a reduce node a terminal window using Cloudera’s VM and start the Hive console
(shuffle phase). The framework guarantees that all the intermediate using command C1. Students can then create the database MStation
pairs with the same key (k2) will be sent to the same reduce node (C2) and use the DDL commands C3 and C4 to create tables
where they will form a single group. Each reduce call processes a stationData and weatherReport, respectively. Students can then
group (k2,list(v2)) and outputs a list of (k3,v3) pairs, which load the data generated by the MStation2 data generator into both
represent the overall output of the MapReduce job. tables using commands C5 and C6. Both, the Hive console and the
While MapReduce is a powerful framework to build highly Hue application, can be used next to run SQL queries. Students can
distributed and scalable programs, it is also complex and difficult use the available tables to write queries such as (Q1-Q4 in Fig. 1):
to learn. In fact, even simple data operations, like joining two  Q1: For each zipcode, compute the average precipitation level.
datasets or identifying the top-K records, require relatively
complex MapReduce programs. This is the case because  Q2: Group the data by station and output the average humidity
MapReduce requires users to build a program using a procedural level at each station.
language that needs a detailed specification of how a processing  Q3: Identify the minimum and maximum temperatures reported
task should be carried out. Considering this limitation, several by each station considering years greater than 2000.
systems have been proposed to: (1) enable the use of SQL-like  Q4: Compute the tenth highest temperatures ever reported. List
languages on top of MapReduce-based systems, e.g., Apache Hive the temperature, zip code, month and year.
[8] and Apache Pig [22], and (2) integrate SQL with MapReduce-
based computations, e.g., Spark SQL [9]. Fig. 2 shows the result of queries using the Hive console and Hue.
C1: sudo hive C1: wget http://real‐chart.finance.yahoo.com/table.csv?
s=AAPL&d=6&e=4&f=2015&g=d&a=11&b=12&c=1980&ignore=.csv
C2: CREATE database MStation;
C2: mv table.csv?s=AAPL table.csv
C3: CREATE TABLE stationData (stationed int, zipcode int,
latitude double, longitude double, stationname string); C3: hadoop fs ‐put ./table.csv /data/
C4: CREATE TABLE weatherReport (stationid int, temp double, C4: spark‐shell ‐‐master yarn‐client ‐‐driver‐memory 512m ‐‐
humi double, precip double, year int, month string); executor‐memory 512m
C5: LOAD DATA LOCAL INPATH '/home/cloudera/datastation/ C5: import org.apache.spark.sql._
stationData.txt' into table stationData;
C6: val base_data = sc.textFile("hdfs://sandbox.hortonworks.
C6: LOAD DATA LOCAL INPATH '/home/cloudera/datastation/ com:8020/data/table.csv")
weatherReport.txt' into table weatherReport;
C7: val attributes = base_data.first
Q1: SELECT S.zipcode, AVG(W.precip) FROM stationData S JOIN
C8: val data = apple_stocks.filter(_(0) != attributes(0))
weatherReport W ON S.stationid = W.stationid GROUP BY
S.zipcode; C9: case class AppleStockRecord(date: String, open: Float,
high: Float, low: Float, close: Float, volume: Integer,
Q2: SELECT stationid, AVG(humi) FROM weatherReport GROUP BY
adjClose: Float)
stationid;
C10: val applestock = data.map(_.split(",")).map(row =>
Q3: SELECT stationid, MIN(temp), MAX(temp) FROM weatherReport
AppleStockRecord(row(0), row(1).trim.toFloat,
WHERE year > 2000 GROUP BY stationid;
row(2).trim.toFloat, row(3).trim.toFloat, row(4).trim.toFloat,
Q4: SELECT S.zipcode, W.temp, W.month, W.year FROM row(5).trim.toInt, row(6).trim.toFloat,
stationData S JOIN weatherReport W ON S.stationid = row(0).trim.substring(0,4).toInt)).toDF()
W.stationid ORDER BY W.temp DESC LIMIT 10;
C11: applestock.registerTempTable("applestock")
Figure 1. Hive Commands and Queries C12: applestock.show
C13: applestock.count
C14: output.map(t => "Record: " + t.toString).collect().
foreach(println)
Q1: val output = sql("SELECT * FROM applestock WHERE close >=
open")
Q2: val output = sql("SELECT MAX(close‐open) FROM applestock")
Q3: val output = sql("SELECT date, high FROM applestock ORDER
BY high DESC LIMIT 10")
Q4: val output = sql("SELECT year, AVG(Volume) FROM applestock
WHERE year > 1999 GROUP BY year")

Figure 3. Spark Commands and SQL Queries

We describe next an in-class activity with Spark SQL.

Hortonworks Sandbox [24] is an excellent virtualization
environment to provide students with hands-on experience in Spark
SQL. This virtual machine comes with Apache Spark fully installed
and enables the execution of commands from a terminal. The
Figure 2. Executing SQL queries in Hive dataset to be used in this activity contains the historical prices of
the Apple stock (AAPL, 1980-2015) and was obtained from the
3.2 Spark: Integrating SQL and MapReduce Yahoo! Finance website [25]. The dataset can be downloaded as a
Apache Spark [9] is a highly distributed data processing comma-separated values (CSV) file. Each record contains the stock
framework. In contrast to Hadoop’s two-stage disk-based values of a single date and includes the following attributes: date,
MapReduce approach, Spark works by loading the data into a open price, high price (highest stock value), low price (lowest stock
cluster's memory and querying it using a wider set of processing value), close price, volume, and adjClose (close price adjusted for
primitives (which also include operations similar to Map and dividends and splits). An additional attribute year (extracted from
Reduce). Spark's performance has been reported to be up to one date) is added to enable de specification of some interesting queries.
hundred times faster than Hadoop’s for some applications [9]. For
distributed storage, Spark can work with Hadoop Distributed File Fig. 3 shows the sequence of commands to load and query the data
System (HDFS), OpenStack, Amazon S3, etc. Spark uses the using the Hortonworks VM. The student downloads first the dataset
notion of Resilient Distributed Datasets (RDD). An RDD is an (C1), renames it to table.csv (C2), copies the file into Hadoop’s
immutable collection of objects that are partitioned and distributed HDFS (C3), and starts the Spark shell (C4). Then, C5 is executed
across multiple physical nodes of a cluster, which allows the data to import a library that is needed to use Spark SQL. Command C6
to be processed in parallel. Once an RDD has been instantiated, the creates the RDD base_data using the dataset stored in the HDFS.
user can apply two types of operations: transformations and actions. C7 and C8 create a new RDD (data) which contains all the records
Transformations return new RDDs while actions generate outputs. in base_data with the exception of the record with the headers. C9
Spark SQL is one of the key tools supported by Spark. Spark SQL creates the class AppleStockRecord which specifies the attributes
is the Spark module for structured data processing, and uses and data types of the Apple stock dataset. C10 parses the records of
DataFrames (equivalent to relational database tables) as its main data, creates an instance of AppleStockRecord for each record, and
programming abstraction. The data in a DataFrame is organized populates the RDD applestock with the created instances. C11
into named columns. DataFrames can be created from already registers applestock as a DataFrame. This DataFrame can now be
existing RDDs, Hive tables, or JSON files. Spark SQL supports used as a relational table. C12 and C13 show the top 20 records and
many of the features of SQL and, therefore, it is a powerful tool to the number of records of applestock, respectively. Students can
learn SQL while working with big datasets. now be assigned to write SQL queries such as (Q1-Q4 in Fig. 3):
  Q1: Report the records where the close price is greater or equal C1: hbase shell
than the open price. C2: create 'OccupationsData', 'code', 'description',
'total_emp', 'salary'
 Q2: Identify the record with the largest difference between the
close and the open prices. C3: exit
C4: hive
 Q3: Report the ten records with the highest high prices.
C5: CREATE EXTERNAL TABLE Occupations (key STRING, description
 Q4: Report the average stock volume per year, considering only STRING, total_emp int, salary int) ROW FORMAT DELIMITED FIELDS
years greater than 1999. TERMINATED BY ',' STORED BY 'org.apache.hadoop.hive.hbase.HBa
seStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mappin
A key feature of Spark is that the output of SQL queries can be used g" = ":key, description:description,total_emp:total_emp,salary
as the input of MapReduce-like computations. This is, in fact, done :salary") TBLPROPERTIES("hbase.table.name"="OccupationsData");
in C14 to print the results of the SQL queries. C6: hbase.org.apache.hadoop.hbase.mapreduce.ImportTsv '‐
Dimporttsv.separator=,' ‐Dimporttsv.columns=HBASE_ROW_KEY,
4. SQL IN NOSQL SYSTEMS details:code,details:description,details:total_emp,details:sal
ary OccupationsData /home/cloudera/occupations.csv;
NoSQL (Not only SQL) is a broad class of data stores that aims to
provide higher scalability and availability than traditional relational C7: impala‐shell
databases. NoSQL systems have the following core properties: (1) C8: invalidate metadata [[default.]Occupations]
they usually use a distributed and fault-tolerant architecture where Q1: SELECT description, salary FROM Occupations ORDER BY
more computers can be easily added, (2) data is partitioned among salary DESC LIMIT 10;
different computers, (3) data is replicated to provide fault tolerance, Q2: SELECT description, salary/total_emp AS PerPersonSalary
and (4) they often support a simplified transaction/consistency FROM Occupations ORDER BY PerPersonSalary DESC LIMIT 10;
model, for instance eventual consistency (given a sufficiently long Q3: SELECT SUM(salary)/SUM(total_emp) AS AvgSalary FROM
period of time in which no changes are made, all the previous occupationsIncomeHBase WHERE description LIKE '%computer%';
updates are expected to eventually propagate through the system). Q4: SELECT code, description, salary FROM Occupations WHERE
There are different types of NoSQL such as: (1) Key-Value stores, Salary IN (SELECT MAX(salary) FROM Occupations);
e.g., Cassandra and Riak, which store data in a schema-less way Figure 4. Impala Commands and SQL Queries
using key-value pairs; (2) Document stores, e.g., MongoDB and
CouchDB, which store documents in a given format (XML, JSON, At this point, students should switch into Hive (C3 and C4), create
binary, etc.); and (3) Tabular data stores, e.g., HBase and BigTable, an external table (Occupations) that is linked to OccupationsData
which store tabular data that can have many attributes and records. (C5), and load the data (C6). After this, students can start the Impala
shell (C7) where they can write and execute SQL queries. Because
While originally most of the NoSQL systems did not adhere to the Impala and Hive share the same metadata database, once the table
relational database model and did not support SQL for data is created in Hive, it can be queried in Impala. Command C8 is
manipulation, one of the key recent developments in this area has needed to make Impala aware of the recently created table. Students
been the recognition that a declarative, highly expressive and can now write an execute queries such as (Q1-Q4 in Fig. 4):
standardized query language like SQL can be an effective way to
query NoSQL systems [2]. This is reflected in a number of systems Q1: Show the occupations with the 10 highest combined salaries
that enable SQL querying on common NoSQL data stores, for Q2: Show the occupations with the 10 highest per-person salaries.
instance Impala [14], SlamData [16] and Presto [15] support SQL Q3: Show the average per-person income of the occupations in the
queries on HBase, MongoDB, and Cassandra, respectively. Computer field.
4.1 Impala: SQL to Query HBase Tables Q4: Show the occupation that has the highest combined income.
Impala [14] is an open-source query engine that allows running
SQL queries on top of HBase tables (Impala also supports HDFS
4.2 SlamData: SQL on MongoDB
as storage sub-system). HBase [11] is an open-source NoSQL SlamData [16] is a system that allows running SQL statements on
database that runs on top of HDFS and provides a fault-tolerant way MongoDB [10], an extensively used document-oriented and open-
of storing and processing large amounts of sparse tabular data. source NoSQL database. MongoDB does not natively support SQL
HBase provides efficient random reads and writes on top of HDFS or multi-object transactions and works with documents as basic
(HDFS does not directly support random writes). Impala has been units of data. Documents use a JSON-like structure (BSON) and
built leveraging and extending key components of Hive, e.g., SQL are equivalent to rows in relational databases. A set of documents
syntax, metadata and schema. While both system support SQL, forms a collection (table). Each document is composed of field and
Hive is best suited for long-running batch processing and Impala value pairs similar to JSON objects. The values of fields can be
provides a better support for dynamic analytic queries. other documents, arrays, and arrays of documents. These embedded
documents and arrays reduce the need for joins.
We propose the following in class activity to introduce SQL in
Impala. For this activity students can use the Cloudera VM [14] We propose the following in-class exercise to introduce the use of
which has Impala, HBase and Hive already installed. The list of SQL in the context of MongoDB and SlamData. The commands for
commands is presented in Fig. 4. The dataset used in this activity this exercise using MS Windows are listed in Fig. 5. The dataset is
contains information about occupations in the U.S. The dataset can the 2012 U.S. Income Tax Statistics per ZIP code reported by the
be found in the VM under the dataset folder (sample_007.csv). It IRS (CSV file) [26]. We use the following attributes: STATE,
has four attributes: code (occupation ID), description, total_emp ZIPCODE (99999 is the aggregation of all ZIP codes with less than
(number of employees), and salary (combined income). We 100 returns, 0 represents the whole state), N1 (number of returns,
renamed the data file as occupations.csv. Students should first open which approximates the number of households), N2 (number of
a terminal window in the VM and start the HBase shell (C1). Next, exemptions, which approximates the population), A00100 (AGI),
the HBase table OccupationsData is created using command C2. and A00200 (salaries and wages, in thousands of dollars).
The first step is to download [10] and install MongoDB (we C1: md \data\db
installed it in C:\mongodb). A required data directory is created in C2: C:\mongodb\bin\mongod.exe
C1. MongoDB is started in C2 and a connection through the C3: C:\mongodb\bin\mongo.exe
mongo.exe shell is stablished in C3. C4 can be used to list the C4: show dbs
available databases (initially, there should be one called local). At
C5: C:\mongodb\bin\mongoimport.exe ‐h <host:port> ‐d <your
this point, SlamData should be downloaded [16] and installed. To database name> ‐c <collection> ‐u <dbuser> ‐p <dbpassword> ‐‐
run queries, students need to mount a MongoDB cluster in type csv ‐‐file <path to .csv file> –headerline
SlamData and host it somewhere. We will do this using the free tier C6: mongodb://<dbuser>:<dbpassword>@<host:port>/<dbname>
of MongoLab [27]. After creating a MongoLab account, students
Q1: SELECT N1 FROM "/project/taxes" WHERE ZIPCODE=85304
should create a new MongoDB deployment (select single-node and
standard line) and provide the database name (project). After the Q2: SELECT N2, ZIPCODE FROM "/project/taxes" WHERE STATE='NY'
deployment is created, students can set the database user name and Q3: SELECT ZIPCODE, A00100 FROM "/project/taxes" WHERE
password by clicking on the new deployment. Then, students will STATE='NY' AND ZIPCODE!=0 ORDER BY (A00100) DESC LIMIT 10
import the CSV file into the MongoDB deployment. This can be Q4: SELECT DISTINCT STATE, AVG(A00200) FROM "/project/taxes"
WHERE ZIPCODE!=0 AND ZIPCODE!=99999 GROUP BY STATE
done using one of the pre-filled commands provided by MongoLab
under the Tools/Import-Export tab. The CSV section in this tab Figure 5. SlamData Commands and SQL Queries
provides a command formatted as C5. Students need to run this VoltDB can be installed on Linux and Mac OS X computers and
command on a MS Command Prompt window. Next, students will provides client libraries for Java, Python, PHP, C++, C#, etc.
open the GUI-based SlamData application and mount the
MongoDB database. To do this, students will click on the Mount VoltDB can be used to learn many aspects of SQL using hands-on
icon in SlamData and use a connection URI (formatted as C6) that class activities. For instance it can be used to learn the SQL data
can be obtained in the recently created MongoDB deployment in definition and data manipulation languages. Using a VM is also an
MongoLab. Students can now use the SlamData application to effective way to enable a hands-on interaction with NewSQL
write SQL queries such as (Q1-Q4 in Fig. 5): systems like VoltDB. In this case, instructors can use the VM
available at VoltDB’s web site [17]. The VM includes the installed
 Q1: Identify the number of households in your ZIP code. VoltDB software and a set of sample applications and tools for
 Q2: Estimate the population in each ZIP code of New York. application development. The sample applications are useful
 Q3: Identify the zip codes with the 10 largest total gross income resources to learn about VoltDB. They include: (1) a system that
in the state of NY. simulates a telephone-based voting process, (2) a memcache-like
 Q4: Compute the average wages and salaries amount per state. cache implementation using VoltDB, (3) a Key-Value store backed
by VoltDB, and (4) a system that uses a flexible schema and JSON.
5. SQL IN NEWSQL SYSTEMS VoltDB also made available an application gallery [21] that
NewSQL is a class of database systems that not only offers the contains many applications that can be used to develop learning
same levels of scalability and availability of NoSQL systems but activities. We propose the use of the Ad Performance application
also preserves the ACID guarantees, relational data model, and that can be downloaded from [21] and installed in VoltDB’s VM.
SQL query language of traditional relational databases. NewSQL This application simulates a high velocity stream of ad events
systems are divided into three groups: (1) New architectures, e.g., (impressions, clickthroughs and conversions) that are augmented
VoltDB and NuoDB, these are new systems designed to work on a and stored in real time. Instructors can use the following in-class
distributed cluster of shared-nothing nodes that can dynamically activity working with this application. Students first analyze and
scale; (2) Optimized SQL Engines, e.g., MySQL Cluster and execute the provided scripts that start the database server, create
Infobright, these systems support the same programming interface tables and views, and run a client application that creates the stream
of traditional systems like MySQL but have better scalability; and of ad events. Fig. 6 shows the statement (C1) that creates the table
(3) Transparent sharding, e.g., dbShards and ScaleBase, these event_data, which stores all the received and processed events.
systems provide a middleware layer to automatically fragment and During the execution of the client application, students use the
distribute databases over multiple nodes. provided web-based interface, shown in Fig. 7, to monitor in real-
Most NewSQL databases can be used to learn SQL considering the time the stream of events and the costs and rates associated with ad
application scenarios commonly used to teach traditional RDBMs, campaigns. Next, students can use VoltDB’s Management Center
e.g., university, employee, or inventory databases. A better to write and run SQL queries. Fig. 6 shows several queries prepared
approach, however, is to consider application scenarios that require by the authors to compute various ad real-time statistics.
the capabilities of NewSQL systems, e.g., the need of processing  Q1: For each website that shows ads, compute the number of
large or fast data in the stock market, social media networks and received ad events, impressions, clicks and conversions, and the
online games. Moreover, educators should consider applications total ad cost.
that take advantage of the features of the NewSQL system used in
class, e.g., an application that requires processing a high number of  Q2: For each advertiser, get the number of processed ad events,
impressions, clicks and conversions, and the total ad cost.
short transactions per second would be a good fit for VoltDB.
 Q3: For each advertiser, compute the daily number of ad events,
5.1 Learning SQL with VoltDB impressions, clicks and conversions, and the daily ad cost.
VoltDB [17] is an in-memory and ACID-compliant NewSQL
 Q4: Identify the 10 ads with the largest associated cost.
database system that uses a shared-nothing architecture as well as
partitioning and replication mechanisms to achieve high transaction This learning activity also provides a good opportunity to highlight
throughputs. The system is open-source and was designed by well- some of the key properties of this type of NewSQL system, e.g.,
known database researchers. VoltDB aims to give solutions to real- high transaction throughput (transactions/second) and low latency
time analytic problems, extracting insights from fast flowing data. (time to process the events).
 C1: CREATE TABLE event_data (utc_time TIMESTAMP NOT NULL, Many of these systems have recognized the strengths of SQL as a
creative_id INTEGER NOT NULL, cost DECIMAL, campaign_id query language. Most database courses, however, focus solely on
INTEGER NOT NULL, advertiser_id INTEGER NOT NULL, site_id
INTEGER NOT NULL, is_impression INTEGER NOT NULL, traditional relational databases. In this paper we propose that SQL
is_clickthrough INTEGER NOT NULL, is_conversion INTEGER should be taught consider the new and broader database landscape.
NOT NULL); This exposure will enable students to get better insight from the
Q1: SELECT site_id, COUNT(*) AS records, SUM(is_impression) data on a wider array of systems and applications. This paper
AS impressions, SUM(is_clickthrough) AS clicks, presents a set of guidelines and a wide array of class resources to
SUM(is_conversion) AS conversions, SUM(cost) AS cost FROM
event_data GROUP BY site_id;
integrate the study of SQL with three core types of Big Data
systems: MapReduce, NoSQL, and NewSQL.
Q2: SELECT advertiser_id, COUNT(*) AS records,
SUM(is_impression) AS impressions, SUM(is_clickthrough) AS
clicks, SUM(is_conversion) AS conversions, SUM(cost) AS cost 8. REFERENCES
FROM event_data GROUP BY advertiser_id; [1] ASU. SQL: From Traditional Databases to Big Data - course
Q3: SELECT advertiser_id, TRUNCATE(DAY,utc_time) AS utc_day, resources. http://www.public.asu.edu/~ynsilva/iBigData.
COUNT(*) AS records, SUM(is_impression) AS impressions,
[2] Barron's. Michael Stonebraker Explains. http://blogs.barrons.
SUM(is_clickthrough) AS clicks, SUM(is_conversion) AS
conversions, SUM(cost) AS spent FROM event_data GROUP BY com/techtraderdaily/2015/03/30/michael-stonebraker-describ
advertiser_id, TRUNCATE(DAY,utc_time); es-oracles-obsolescence-facebooks-enormous-challenge/.
Q4: SELECT creative_id AS ad, SUM(cost) AS cost FROM [3] D. Kumar. Data science overtakes computer science? ACM
event_data GROUP BY creative_id ORDER BY cost DESC LIMIT 10;
Inroads, 3(3):18–19, Sept. 2012.
Figure 6. VoltDB SQL Queries [4] A. Cron, H. L. Nguyen, and A. Parameswaran. Big data.
XRDS, 19(1):7–8, Sept. 2012.
[5] A. Sattar, T. Lorenzen, and K. Nallamaddi. Incorporating
nosql into a database course. ACM Inroads, 4(2):50–53, June
2013.
[6] Y. N. Silva, S. W. Dietrich, J. Reed, L. Tsosie. Integrating
Big Data into the Computing Curricula. In ICDE 2015.
[7] Apache. Hadoop. http://hadoop.apache.org/.
[8] Apache. Hive. https://hive.apache.org/.
[9] Apache. Spark SQL. https://spark.apache.org/sql/.
[10] 10gen. Mongodb. http://www.mongodb.org/.
[11] Apache. Hbase. http://hbase.apache.org/.
[12] F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A.
Wallach, M. Burrows, T. Chandra, A. Fikes, and R. E.
Gruber. Bigtable: A distributed storage system for structured
Figure 7. Executing SQL queries in VoltDB data. ACM Trans. Comput. Syst., 26(2):1–26, 2008.

6. DISCUSSION [13] Apache. Cassandra. http://cassandra.apache.org/.

The proposed modules can be integrated into several introductory [14] Cloudera. Impala. http://impala.io/.
and advanced database-related courses. One approach is to adopt [15] Presto. https://prestodb.io/.
Big Data management systems as a replacement of the traditional
relational databases currently used in most introductory database [16] SlamData. SlamData. http://slamdata.com/.
courses. We recommend using a NewSQL system, e.g., VoltDB, [17] VoltDB. Voltdb VM. https://voltdb.com/run-voltdb-vmware.
for this approach since it supports both the relational model and
[18] MemSQL. MemSQL. http://www.memsql.com/.
SQL. An alternative approach for introductory database courses is
to complement the use of a traditional database with a Big Data [19] NuoDB. Nuodb. http://www.nuodb.com/.
system. In this case, instructors can teach database fundamentals [20] Clustrix. Clustrix. http://www.clustrix.com/.
using traditional databases and then show how the same or similar
concepts can be applied to modern Big Data systems. Another [21] VoltDB. Application gallery.
approach it to integrate the study of SQL beyond traditional http://voltdb.com/community/applications.
databases as part of a course in Big Data. This course can study the [22] Apache. Pig. https://pig.apache.org/.
use of SQL in MapReduce, NoSQL and NewSQL systems, and
[23] Cloudera. Cloudera VM 4.7.0. http://www.cloudera.com/
compare it with other native query languages. The authors have
content/cloudera/en/downloads.html.
previously integrated the study of NewSQL systems in a Big Data
course and are currently integrating the study of SQL in a Big Data [24] Hortonworks. Hortonworks Sandbox on a VM, HDP 2.3.
system as part of an undergraduate database course. We plan to http://hortonworks.com/products/hortonworks-sandbox.
report the results of these experiences in a future work. [25] Yahoo Finance. Historical Prices - AAPL.
http://finance.yahoo.com/q/hp?s=AAPL+Historical+Prices.
7. CONCLUSIONS
[26] IRS. Tax Stats. http://www.irs.gov/uac/SOI-Tax-Stats-Indivi
Many application scenarios require processing very large datasets
dual-Income-Tax-Statistics-2012-ZIP-Code-Data-(SOI).
in a highly scalable and distributed fashion and different types of
Big Data systems have been designed to address this challenge. [27] Mongolab. Mongolab. https://mongolab.com/.

View publication stats