Introduction

• Why SQL?

• What about Python? R?

• Data Analytics

• Relational Database

• What is a database?

• Terminology

• SQLite

• Exercise 1

• SQL

• Data Definition Language (DDL)

• Exercise 2

• Data Manipulation Language (DML)

• Exercise 3 • Open Data Portal

• How I prepared for today

Why SQL? :
• Simple • Accessible • Applicable • Powerful • Pervasive • Valuable • Universal
Why not Python? R? :
Difficult for beginners • Complicated syntax • Requires programming knowledge (logic,
algorithms) • Is SQL better than Python or R? • SQL is good for some things • Python/R is
good for other things • Compliment each other • SQL is a great starting point.

Data
Analytics :
• Analytics is the discovery, interpretation, and communication of meaningful patterns in
data, and the process of applying those patterns towards effective decision making.

• Organizations may apply analytics to business data to describe, predict, and improve
business performance.

What is a database?
A relational “database” management system (RDBMS) organizes data.

• The logical structure of the database is based upon the information needs of an
organization.

• Entities (“things” of interest to the organization), AND.

• Relationships (how the Entities are associated with each other).

Advantages of an RDBMS :
• Establish a centralized, logical view of
data

• Minimizes data duplication (i.e.,

“redundancy”)

• Promote data accuracy and integrity

• Capacity of the database

• Superior multi-user or concurrent access

• Security

• Retrieve information quickly

• Inter-operability

Database Terminology:
• Table - Entity, Relation (similar to an Excel Worksheet)

• Row - Record, Instance

• Column - Field, Attribute

• Primary Key – unique and mandatory

• Foreign Key – a cross-reference between tables because it references the primary key of
another table

• Relationship – created through foreign keys

How to introduce SQL?
• Microsoft Access : https://products.office.com/en- ca/access

• Microsoft SQL Server : https://www.microsoft.com/en- us/sql-server/sql-server-2017

• MariaDB, MySQL : https://mariadb.org/ , https://www.mysql.com/

• PostgreSQL: https://www.postgresql.org/

• Oracle: https://www.oracle.com/database/

• Hadoop, Spark, Hive, Pig: https://hadoop.apache.org/

A database that :
• Has billions and billions of deployments

• Is a single-file database

• Has public domain source code

• Small footprint

• Has a max DB size of 140 terabytes

• Has a max row size of 1 gigabyte

• Is faster than direct file access

• Aviation-grade quality and testing

• Zero-configuration

• Has ACID (Atomic, Consistent, Isolated, and Durable) transactions, even after power loss

• Has a stable, enduring file format

• Has extensive, detailed documentation

• Has long-term support (to the year 2050)

SQLite :
• “SQLite is the most widely deployed database in the world with more applications than we
can count, including several high-profile projects.”

• https://www.sqlite.org/famous.html

• “SQLite is an in-process library that implements a self-contained, serverless, zero-

configuration, transactional SQL database engine.”

• https://www.sqlite.org/about.html

• Perfect for learning SQL (the foundation of data analytics)

Exercise 1: Download and Run SQLite :

• Extract the ZIP archive to the Desktop

• Start SQLite • SQLiteDatabaseBrowserPortable.exe

• Create a New database

• open_data_day_2019.db

• Save the database in the Data folder

• Click Cancel when prompted to create a table

• Done!

What is SQL?
• SQL stands for Structured Query Language.

• SQL is pronounced S-Q-L or sequel.

• SQL is a standard language for managing, manipulating, and

querying databases.

• Developed at IBM in the early 1970s.

• In 1986, ANSI and ISO standard groups officially adopted the standard “Database Language
SQL” definition.

• Most SQL databases have their proprietary extensions in addition to the SQL standard.

• SQL is the language used to ask questions (queries) of a database, which will return
answers (results).
 Why is SQL the foundation of Data
Analytics?
• Data engineers and database administrators will use SQL to ensure that everybody in their
organization can access the needed data.

• Data scientists will use SQL to load data into their models.

• Data analysts will use SQL to query tables of data and derive insights from the.

Components of SQL :
• SQL consists of three components that offer everything required to manage, maintain,
and use a database

1. Data Definition Language 2. Data Manipulation Language 3. Data Control Language

Data Definition Language (DDL)
• This component is used to define the
structure (or schema) of the database.

• For tables, there are three main commands.

• CREATE TABLE table_name

• To create a table in the database

• ALTER TABLE table_name

• To add or remove columns from a table in the database

• DROP TABLE table_name

• To remove a table from the database

Exercise 2: Data Definition Language

• Select the Execute SQL tab in SQLite.

• Type or copy/paste the CREATE TABLE statement into the empty SQLite Execute SQL
window.

• Click the Execute SQL button on the toolbar.

• If the table is created successfully, you should receive the following message:

• Query executed successfully: CREATE TABLE "MOSQUITO_TRAP_DATA“ .

• Click Write Changes to commit the changes permanently.

• View the changes in the Database Structure tab.

CREATE TABLE "MOSQUITO_TRAP_DATA" ( `SAMPLEID` INTEGER PRIMARY KEY

AUTOINCREMENT,

`TRAP_DATE` NUMERIC,

`GENUS` TEXT,

`SPECIES` TEXT,

`TYPE` TEXT,

`GENDER` TEXT)

• Select the Execute SQL tab in SQLite.

• Type or copy/paste the ALTER TABLE statements into the empty SQLite Execute SQL
window.

• Click the Execute SQL button on the toolbar.

• If the table is created successfully, you should receive the following message:

• Query executed successfully: ALTER TABLE "MOSQUITO_TRAP_DATA“

• Click Write Changes to make the changes permanent.

• View the changes in the Database Structure tab.

ALTER TABLE "MOSQUITO_TRAP_DATA" ADD COLUMN `RURALNORTHWEST` INTEGER; ALTER

TABLE "MOSQUITO_TRAP_DATA" ADD COLUMN `RURALNORTHEAST` INTEGER; ALTER TABLE
"MOSQUITO_TRAP_DATA" ADD COLUMN `RURALSOUTHEAST` INTEGER; ALTER TABLE
"MOSQUITO_TRAP_DATA" ADD COLUMN `RIVERVALLEYEAST` INTEGER; ALTER TABLE
"MOSQUITO_TRAP_DATA" ADD COLUMN `RIVERVALLEYWEST` INTEGER; ALTER TABLE
"MOSQUITO_TRAP_DATA" ADD COLUMN `RESIDENTIALNORTH` INTEGER; ALTER TABLE
"MOSQUITO_TRAP_DATA" ADD COLUMN `RURALSOUTHWEST` INTEGER; ALTER TABLE
"MOSQUITO_TRAP_DATA" ADD COLUMN `LAGOON` INTEGER; ALTER TABLE
"MOSQUITO_TRAP_DATA" ADD COLUMN `GOLFCOURSE` INTEGER; ALTER TABLE
"MOSQUITO_TRAP_DATA" ADD COLUMN `INDUSTRIALPARK` INTEGER; ALTER TABLE
"MOSQUITO_TRAP_DATA" ADD COLUMN `RESIDENTIALSOUTH` INTEGER; ALTER TABLE
"MOSQUITO_TRAP_DATA" ADD COLUMN `TOTAL` INTEGER;

• Select the Execute SQL tab in SQLite

• Type or copy/paste the DROP TABLE statement into the empty SQLite Execute SQL window
• Click the Execute SQL button on the toolbar

• If the table is created successfully, you should receive the following message:

• Query executed successfully: DROP TABLE "MOSQUITO_TRAP_DATA"

• Click Write Changes to make the changes permanent

• View the changes in the Database Structure tab

DROP TABLE "MOSQUITO_TRAP_DATA";

• Create the MOSQUITO_TRAP_DATA table again using the DDL on the next slide.

• Click Write Changes to make the changes permanent.

• View the changes in the Database Structure tab

• Done!
CREATE TABLE "MOSQUITO_TRAP_DATA" (

`SAMPLEID` INTEGER PRIMARY KEY AUTOINCREMENT,

`TRAP_DATE` NUMERIC, `GENUS` TEXT,

`SPECIES` TEXT, `TYPE` TEXT,

`GENDER` TEXT,

`RURALNORTH` INTEGER,

`RURALNORTHEAST` INTEGER,

`RURALSOUTHEAST` INTEGER,

`RIVERVALLEYEAST` INTEGER,

`RIVERVALLEYWEST` INTEGER,

`RESIDENTIALNORTH` INTEGER,

`RURALSOUTHWEST` INTEGER,

`LAGOON` INTEGER,

`GOLFCOURSE` INTEGER,

`INDUSTRIALPARK` INTEGER,

`RESIDENTIALSOUTH` INTEGER,

`TOTAL` INTEGER
Data Manipulation Language
• This component is used to manipulate data within a table. There are four main commands:
• SELECT - To select rows of data from a table.

• INSERT - To insert rows of data into a table.

• UPDATE - To change rows of data in a table.

• DELETE - To remove rows of data from a table

SELECT Data Manipulation Language :

• Select the Execute SQL tab in SQLite.

• Type or copy/paste the SELECT statement into the empty SQLite Execute SQL window.

• SELECT COUNT(*) FROM MOSQUITO_TRAP_DATA;

• Click the Execute SQL button on the toolbar.

• Do you get an answer? Why not?

Exercise 3: INSERT Data Manipulation Language

• Add some data to the MOSQUITO_TRAP_DATA table created in Exercise 2

• Type or copy/paste the INSERT statement into the empty SQLite Execute SQL window

• Click the Execute SQL button on the toolbar

• Click Write Changes to make the changes permanent

• View the changes in the Browse Data tab

• The MOSQUITO_TRAP_DATA table now has seven rows of data.

INSERT INTO "MOSQUITO_TRAP_DATA" (TRAP_DATE, GENUS, SPECIES, TYPE, GENDER,

RURALNORTHWEST, RURALNORTHEAST, RURALSOUTHEAST, RIVERVALLEYEAST,
RIVERVALLEYWEST, RESIDENTIALNORTH, RURALSOUTHWEST, LAGOON, GOLFCOURSE,
INDUSTRIALPARK, RESIDENTIALSOUTH, TOTAL) VALUES ('2014- 07-
01','Aedes','spencerii','Black legs','Female',0,0,0,0,0,1,0,0,0,1,1,3); INSERT INTO
"MOSQUITO_TRAP_DATA" (TRAP_DATE, GENUS, SPECIES, TYPE, GENDER,
RURALNORTHWEST, RURALNORTHEAST, RURALSOUTHEAST, RIVERVALLEYEAST,
RIVERVALLEYWEST, RESIDENTIALNORTH, RURALSOUTHWEST, LAGOON, GOLFCOURSE,
INDUSTRIALPARK, RESIDENTIALSOUTH, TOTAL) VALUES ('2014- 07-
01','Aedes','dorsalis','Banded legs','Female',0,1,0,0,0,0,2,0,0,0,0,3); INSERT INTO
"MOSQUITO_TRAP_DATA" (TRAP_DATE, GENUS, SPECIES, TYPE, GENDER,
RURALNORTHWEST, RURALNORTHEAST, RURALSOUTHEAST, RIVERVALLEYEAST,
RIVERVALLEYWEST, RESIDENTIALNORTH, RURALSOUTHWEST, LAGOON, GOLFCOURSE,
INDUSTRIALPARK, RESIDENTIALSOUTH, TOTAL) VALUES ('2014- 07-
01','Aedes','euedes','Banded legs','Female',1,1,0,0,2,0,0,0,0,0,0,4); INSERT INTO
"MOSQUITO_TRAP_DATA" (TRAP_DATE, GENUS, SPECIES, TYPE, GENDER,
RURALNORTHWEST, RURALNORTHEAST, RURALSOUTHEAST, RIVERVALLEYEAST,
RIVERVALLEYWEST, RESIDENTIALNORTH, RURALSOUTHWEST, LAGOON, GOLFCOURSE,
INDUSTRIALPARK, RESIDENTIALSOUTH, TOTAL) VALUES ('2014- 07-
01','Aedes','excrucians','Banded legs','Female',1,2,0,0,2,1,0,0,0,1,0,7); INSERT INTO
"MOSQUITO_TRAP_DATA" (TRAP_DATE, GENUS, SPECIES, TYPE, GENDER,
RURALNORTHWEST, RURALNORTHEAST, RURALSOUTHEAST, RIVERVALLEYEAST,
RIVERVALLEYWEST, RESIDENTIALNORTH, RURALSOUTHWEST, LAGOON, GOLFCOURSE,
INDUSTRIALPARK, RESIDENTIALSOUTH, TOTAL) VALUES ('2014- 07-
01','Aedes','fitchii','Banded legs','Female',0,2,0,0,1,0,0,0,0,0,4,7); INSERT INTO
"MOSQUITO_TRAP_DATA" (TRAP_DATE, GENUS, SPECIES, TYPE, GENDER,
RURALNORTHWEST, RURALNORTHEAST, RURALSOUTHEAST, RIVERVALLEYEAST,
RIVERVALLEYWEST, RESIDENTIALNORTH, RURALSOUTHWEST, LAGOON, GOLFCOURSE,
INDUSTRIALPARK, RESIDENTIALSOUTH, TOTAL) VALUES ('2014- 07-
01','Aedes','flavescens','Banded legs','Female',6,5,8,0,0,0,5,0,0,3,1,28); INSERT INTO
"MOSQUITO_TRAP_DATA" (TRAP_DATE, GENUS, SPECIES, TYPE, GENDER,
RURALNORTHWEST, RURALNORTHEAST, RURALSOUTHEAST, RIVERVALLEYEAST,
RIVERVALLEYWEST, RESIDENTIALNORTH, RURALSOUTHWEST, LAGOON, GOLFCOURSE,
INDUSTRIALPARK, RESIDENTIALSOUTH, TOTAL) VALUES ('2014- 07-
01','Aedes','vexans','Banded legs','Female',3,168,1,21,38,8,16,0,0,3,32,290);

Type or copy/paste the SELECT statement into the empty SQLite Execute SQL window.

• SELECT COUNT(*) FROM MOSQUITO_TRAP_DATA;

• Click the Execute SQL button on the toolbar.

• When you execute the query, you ask the database a question.

• Can you tell me the number of rows in the MOSQUITO_TRAP_DATA table?.

• The database gives you an answer (the result) and you should have received the following
message:

• 7 rows returned in 1ms from: SELECT * FROM MOSQUITO_TRAP_DATA;

What if you want to see all the rows in your database?.

• SELECT * FROM MOSQUITO_TRAP_DATA;

• Returns all columns and rows in a table.

• What if you only want to see each row's Genus, Species, and Total?

• SELECT GENUS, SPECIES, TOTAL FROM MOSQUITO_TRAP_DATA;

• Returns only the GENUS, SPECIES, and TOTAL columns for each row in a table.

Data Manipulation Language :

• The WHERE clause.

• Uses operators to extract only those

records that fulfill a specified condition.

• Used to ask more complicated questions.

• SQL will do exactly what you ask, not

always what you expect.

• “I do not think it means what you think it

means.”
• Inigo Montoya.

• Show the rows that have a mosquito TYPE of “Black legs”

• SELECT * FROM MOSQUITO_TRAP_DATA WHERE TYPE = 'Black legs';

YOUR TURN :
• Write and execute a DML statement to answer the question below:

• Which mosquito species were caught in the traps placed in the west river valley?

UPDATE Data Manipulation Language :

• Select the Execute SQL tab in SQLite.

• Type or copy/paste the UPDATE statement into an empty SQLite Execute SQL window.

• Click the Execute SQL button on the toolbar.

• You should receive the following message:

• Query executed successfully: … (took 1ms, 4 rows affected)

UPDATE MOSQUITO_TRAP_DATA SET GENDER = 'Male‘ WHERE SAMPLEID IN (1,3,5,7);

• The GROUP BY clause - Used in

collaboration with the SELECT statement to
arrange identical data into groups.

• The GROUP BY statement is often used

with aggregate functions.

YOUR TURN
• Write and execute a DML statement to answer the question below:

• How many mosquitoes of each gender were caught in traps throughout the city?

• Select the Execute SQL tab in SQLite.

• Type or copy/paste the DELETE statement into an empty SQLite Execute SQL window.

• Click the Execute SQL button on the toolbar.

• You should receive the following message:

• Query executed successfully: … (took 0ms, 4 rows affected).

DELETE FROM MOSQUITO_TRAP_DATA WHERE GENDER = "Male";

YOUR TURN

• Write and execute a DML statement to answer the question below:

• At which traps were more mosquitoes caught? Rural north east or rural north west?

• Done!

Advanced SQL
• The MOSQUITO database only has one table.

• Databases with more than one table require tables to be joined.

• Foreign keys create relationships between tables and must be joined in a DML statement.

• Download the LED Streetlight Conversion database called odd_streetlight.db.

• Execute the query below: SELECT LED_STREETLIGHT.STREETLIGHT_ID,

LED_STREETLIGHT.TYPE, LOCATION.LOCATION FROM LED_STREETLIGHT, LOCATION WHERE
LED_STREETLIGHT.STREETLIGHT_ID = LOCATION.STREETLIGHT_ID AND
LED_STREETLIGHT.STREETLIGHT_ID = 12;

Using the Open Data Portal

• https://data.edmonton.ca/

• Data sets are usually available in comma-separated value (CSV) format.

• To use the dataset requires cleaning, importing, exploring, and understanding the dataset
• Workshop: Exploring & Cleaning Data with Open Refine.

• Requires work.
Data Work Flow :

How I prepared the data sets for today:

• Selected data sets from the Open Data Portal.

• Downloaded the CSV and surveyed in Google Sheets.

• Cleaned the data set.

• E.g., reformatted dates from MMM DD YYYY to YYYYMM-DD.

• Imported directly into SQLite tables.

• Added primary keys.

• Explored the dataset using DML.

Some “Mosquito Trap Data” questions

• How many mosquitoes were caught in 2014? SELECT strftime ('%Y', TRAP_DATE) as YEAR,
SUM(TOTAL) FROM MOSQUITO_TRAP_DATA

WHERE TOTAL <> '' AND TOTAL > 0 GROUP BY YEAR;

• How many mosquitoes of each species were caught?

• Which traps caught the most mosquitoes?

Some “LED Streetlight Conversion” questions

• How many total streetlights?

• How many streetlights are converted to LED?

• How many streetlights were converted by year? SELECT strftime('%Y', STARTDATE) as YEAR,
TYPE, COUNT(STREETLIGHT_ID)
FROM LED_STREETLIGHT

WHERE TYPE = "LED"

GROUP BY YEAR;

SQL and Climate Change

• Connecting and linking various data sets.

• Builds an understanding of what that data means.

•Data is a universal language, and climate change is a global problem.

Next steps

• Playing with data and SQL forces you to think and understand the data (builds knowledge)
• The relationships between data

• The meaning of those relationships

• The validity of the data

• SQL is iterative, often a “trial and error” process

• Don’t be afraid to make mistakes

• Team sport – discuss, share, question, collaborate

• Data is everywhere, which raises questions of privacy, security, and ethics