Week 1： Getting Started and Selecting &

Retrieving Data with SQL

Introduction
What is SQL?

Structured Query Language (SQL) is a standard computer language for relational database
management and data manipulation.

Used to query, insert, update and modify data.

Pronounced as “sequel” or S-Q-L.

Used to communicate with databases;

SQL is non-procedural language:

 Cannot write complete applications

 Simple, but powerful

How is SQL used?

Read/retrieve data

Write data - add data to a table;

Update data - insert new data;

Who uses SQL?

 Backend Developer
 Data Architect
 QA Engineer
 ETL Developer
 Database Admin (DBA)
 Data analyst
 System Admin
 System Engineer
 Data Scientist

DBA or Data Scientist

DBA

Manages/governs entire database End user of a database

Gives permissions to users Use SQL to query and retrie

Determines access to data

Manages and creates tables

Uses SQL to query and retrieve data

HOW DO DATA SCIENTISTS USE SQL?

Retrieve data;

May create their own table or test enviroments;

Combine multiple sources together;

Write complex queries for analysis.

Relational Database Management Systems

 SQL Server
 IBM DB2 Oracle
 Sybase ASE
 PostgreSQL
 MySQL
 Microsoft SQL Server
 Apache Open Office Base
 SQLite
Write syntax depends on what DBMS you are using.

Each DBMS has its own dialect.

SQL can translate.

Tweak based on the dialect your DBMS speaks.

Data Models
Thinking about your data

Understand the business process or subject matter the data is modeled after

Know the bussiness rules

Understand how your data is organized and structured in the table(modeled).

Why this is worthwhile

Get more accurate results,

Speed up your work,

Have less rework.

Some concepts
 Name Description

Database A container to store organized data

Tables A structured list of data or a specific type

Column A single field in a table

Row A record in a table

What is data modeling?

Organizes and structures information into multiple, related tables.

Can represent a bussiness process or show relationships between bussiness processes.

Should closely represent real world.

Types of data modeling

Models for prediction built by data scientist.

Data model as data tables represented and organized in a database.

Evolution of data models

Time Types

1960 Hierarchical

1969 Network
 Time Types

1970 Relational

1976 Entity relationship

1978 Semantic

1985 Object-oriented

1990 Extended relational(O/R DBMS)

2009 NoSQL - Not only SQL

Relational vs. Transactional Models

Relational Model

Allows for easy querying and data manipulation in an easy, logical and intuitive way. Operational database

Data model building blocks

Name Description

Entity Person, place thing or event, Distinguishable, unique and distinct

 Name Description

Attribute A characteristic of an entity

Describes association among entities.

Relationship
One-to-many,Many-to-many,One-to-one

 One-to-many: customer to invoices.

 Many-to-many: student to classes.
 One-to-one: manager to store.

ER Diagrams
ER Model

 Show relationships
 Bussiness process
 Represented visually
 Show links (Primary keys)

Some examples of ER Diagram

Retrieving data with SELECT statement

1SELECT prod_name
2FROM Products;

For multiple columns, be sure to use a comma.

 1SELECT prod_name,prod_id,prod_price
2FROM Products;

The following expression is more clearly.

1SELECT prod_name
2 ,prod_id
3 ,prod_price
4FROM Products;

Retrieving multiple columns using a wildcard

Request all columns by using a asterisk(*) wildcard character.

1SELECT *
2FROM Products;

Limit results

If the database is very large, we can use LIMIT to limit the number of retrieved results.
1SELECT *
2FROM Products
3LIMIT 5;
 Different Syntaxes

Database Codes

prod_name FROM products LIMIT 5;```|

1
2|Oracle|```SELECT prod_name FROM products WHERE ROWNUM <=5;```|
3|DB2|```SELECT prod_name FROM products FETCH FIRST 5 ROWS ONLY;`
4## Creating Tables
5```sql
SQLite 6CREATE TABLE Shoes
7(
8Id char(10) PRIMARY KEY,
9Brand char(10) NOT NULL,
10Price decimal(8,2) NOT NULL,
11Desc Varchar(750) NULL
12);
13
14
NULL and Primary Keys

Every column is either NULL or NOT NULL.

Primary keys cannot be null, they must have a value.

ADDING DATA TO THE TABLE

1INSERT INTO Shoes

2(Id
3,Brand
4,Price
5,Desc
6)
7VALUES
8('12345'
9,'Gucci'
10,'695.00'
11,NULL
12)
13;

CREATE A TEMPLATE TABLE

1CREATE TEMPORARY TABLE Sandals AS

2(
3SELECT *
4FROM Shoes
5WHERE shoe_type = 'sandals'
6);

Adding comments
Single line

1SELECT shoe_id
2--,brand_id
3,shoe_name
4FROM Shoes;

Section

1SELECT shoe_id
2/*,brand_id
 3,shoe_name
4*/
5FROM Shoes;

Week 2: Filtering, Sorting, and Calculating

Data with SQL
Introductions
Clauses and Operators in SQL

 WHERE
 BETWEEN
 IN
 OR
 NOT
 LIKE
 ORDER BY
 GROUP BY

Wildcards

Math Operators

 AVERAGE
 COUNT
 MAX
 MIN

Basics of Filtering with SQL

Why?

 Be specific about the data you want to retrieve

 Reduce the number of records you retrieve
 Increase query performance
 Reduce the strain on the client application
 Goverance the limitations

WHERE Clause

1SELECT column_name,column_name
2FROM table_name
3WHERE column_name operator value;

Some clause operators

FILTERING ON A SINGLE CONDITION

1SELECT ProductName
2,UnitPrice
3,SupplierID
4FROM Products
5WHERE ProductName = 'Tofu';

FILTERING ON A SINGLE VALUE

1SELECT ProductName
2,UnitPrice
3,SupplierID
4FROM Products
5WHERE UnitPrice >= 75;

CHECKING FOR NON-MATCHES

1SELECT ProductName
2,UnitPrice
3,SupplierID
4FROM Products
5WHERE ProductName <> 'Alice';

FILTERING WITH A RANGE OF VALUES (BETWEEN…AND…)

1SELECT ProductName
2,UnitPrice
 3,SupplierID
4FROM Products
WHERE UnitsInStock BETWEEN 15 AND
580;

FILTERING NO VALUE (NULL)

1SELECT ProductName
2,UnitPrice
3,SupplierID
4FROM Products
5WHERE ProductName IS NULL;

Advanced filtering: IN, OR, and NOT

IN operator

 Specifies a range of conditions

 Comma delimited list of values
 Enclosed in ()
Example

1SELECT ProductID
2,UnitPrice
3,SupplierID
4FROM Products
5WHERE SupplierID IN (9,10,11);

OR operator

DBMS will not evaluate the second conditions in WHERE clause if the first condition is met.

Use for any rows matching the specific conditions.

1SELECT ProductName
2,UnitPrice
3,SupplierID
4FROM Products
5WHERE ProductName = 'Tofu' OR 'Konbu';

IN vs. OR
 IN works the same as OR.

Benfits of IN:

1. Long list of options,

2. IN executes faster than OR,
3. Don’t have to think about the order with IN,
4. Can contain another SELECT5

OR with AND

1SELECT ProductName
2,UnitPrice
3,SupplierID
4FROM Products
5WHERE SupplierID = 9 OR SupplierID = 11 AND UnitPrice > 15;
1SELECT ProductName
2,UnitPrice
3,SupplierID
4FROM Products
5WHERE (SupplierID = 9 OR SupplierID = 11) AND UnitPrice > 15;

Don’t rely on the default order, it’s better to have the habit of using the ().

NOT operator

1SELECT *
FROM Employees
2WHERE NOT City = 'London' AND NOT City =
3'Seattle';

Using Wildcards in SQL

 Special character used to match parts of a value,

 Use LIKE Operator

What are wildcards?

 Can only be used with strings.

 Cannot be used for non-text datatype.
 Helpful for data scientists as they explore string variables.
 Using % wildcards

 % wildcards will not match NULLS, NULL represents no value in a column.

Using underscore(_) wildcards

 Matches a single character.

 Is not supported by DB2.

Downsides of wildcards

 Takes longer to run.

 Better to use another operator (=, <, >=, and etc.)
 Statements with wildcards will take longer to run if used at the end of search
pattern.
 Placement of wildcards is important.

Sorting with ORDER BY
1SELECT something
2FROM database
3ORDER BY characteristic;
 Can sort by a column not retrieved
 Must always be the last clause in a select statement

Sorting by column position

1ORDER BY 2,3;
 It means data sorted by 2nd and 3rd column.

Sort direction

DESC: descending order

ASC: ascending order
Only applies to the column names it directly precedes.

Math operators

Operator D

+ Addition

- Subtraction

* Multiplication

/ Division

Order of operators

 Parenthese
 Exponents
 Multiplication
 Division
 Addition
 Subtraction

Use parenthese if needed

1SELECT ProductID
2,Quantity
3,UnitPrice
 4,Discount
5,(UnitPrice - Discount) / Quantity AS Total_Cost
6FROM Orders;

Aggregate Functions
What?

 Used to summarize data

 Finding the highest and lowest values
 Finding the total number of rows
 Finding the average value

Functions

Average function

1SELECT AVG(UnitPrice) AS avg_price

2FROM products;

Count functions

1.COUNT(*) - Counts all rows (contain NULLs).

1SELECT COUNT(*) AS total_customers

2FROM Customers;

2.COUNT(Column_name) - Counts all rows in a specific column ignoring NULL value.

1SELECT COUNT(CustomerID) AS total_customers

2FROM Customers;

MIN and MAX functions

 Columns with NULL values are ignored by MIN and MAX functions.

1SELECT MAX(UnitPrice) AS max_prod_price

2,MIN(UnitPrice) AS min_prod_price
3FROM Products;

SUM aggregate functions

1SELECT SUM(UnitPrice) AS total_prod_price

2FROM Products;

Using DISTINCT on aggregate functions

 If DISTINCT is not specified, ALL is assumed.

 Cannot use DISTINCT on COUNT(*).
 No value to use with MIN and MAX function.
1SELECT COUNT(DISTINCT CustomerID)
2FROM Customers;

Grouping Data with SQL

Grouping examples

1SELECT
2Region
3,COUNT(CustomerID) AS total_customers
4FROM Customers
5GROUP BY Region;

Notices

 GROUP BY clause can contain multiple columns.

 Every column in your SELECT statement must be present in a GROUP BY clause
(except for aggregate functions).
 NULLs will be grouped together if your GROUP BY column contains NULLs.

HAVING clause filtering for groups

WHERE does not work for GROUP.

 WHERE filters on rows.

Instead using HAVING clause to filter for groups.

HAVING examples

1SELECT
2CustomerID
3,COUNT(*) AS orders
4FROM Orders
5GROUP BY CustomerID
6HAVING COUNT(*) >=2;

WHERE vs. HAVING

WHERE filters before data is grouped.

HAVING filters after data is grouped.

Rows elimated by WHERE clause will not be included in the group.

1SELECT
2SupplierID
3,COUNT(*) AS Num_prod
4FROM Products
5WHERE UnitPrice >= 4
6GROUP BY SupplierID
7HAVING COUNT(*) >=2;

Putting it all together

Filtering is useful

Narrow down your results.

Increasing query & application performance.

Understanding your data:

 Finding specific values

 Finding a range of values
 Finding blank values

Key SQL clauses

Week 3: Subqueries and joins in SQL

Using Subqueries
What?

 Queries embedded into other queries

 Relational databases store data in multiple tables
 Subqueries merge data from multiple source together
 Helps with adding other filtering criteria

Example

1SELECT CustomerID
2,CompanyName
3,Region
4FROM Customers
5WHERE customerID in (SELECT CustomerID
6 FROM Orders
7 WHERE Freight > 100);

Working with Subsequery statements

Always perform the innermost SELECT portion first.

DBMS is performing two operations

1. Getting the order numbers for the product selected,

2. Adding that to the WHERE clause and processing the overall SELECT statement.
 Subquery best practices and considerations

1. There is no limit to the number of subqueries you can have.

2. Performance slows when you nest too deeply.
3. Subquery selects can only retrieve a single column.

Subquery in subquery

Be sure to use indenting.

1SELECT customer_name
2,customer_state
3,(SELECT COUNT(*) AS orders
4 FROM Orders
5 WHERE Orders.customer_id = Customer.customer_id) AS orders
6FROM Customers
7ORDER BY Customer_name;

The power of subqueries

 Subqueries are powerful tools

 Not always the best option due to performance

Joining tables: an introduction

Benefits of breaking data into tables

 Efficient storage
 Easier manipulation
 Greater scalability
 Logically models a process
 Tables are related through common values (keys)

Joins

 Associate correct records from each table on the fly.

 Allows data retrieval from multiple tables in one query.
 Joins are not physical, they persist data for the duration of the query execution.
 Cartesian (Cross) Joins

 Not frequently used.

 Computationally taxing.
 Will return products with the incorrect vendor or no vendor at all.

Examples

1SELECT vendor_name
2,unit_price
3,company_name
4FROM suppliers
5CROSS JOIN products;
1SELECT vendor_name
2,product_name
3,product_price
4FROM Vendors,Products
5WHERE Vendors.vendor_id = Products.vendor_id;

Inner Joins
What?

The INNER JOIN keyword selects records that have matching values in both tables.

Example

1SELECT suppliers.CompanyName
2,ProductName
3,UnitPrice
4FROM Suppliers INNER JOIN Products
5ON Suppliers.supplierid = Products.supplierid;

Syntax

 Join type is specified (INNER JOIN)

 Join condition is in the FROM clause and uses ON clause
 Joining more tables together affects overall database performance
 You can join multiple tables, no limit
 List all the tables, then define conditions
 Inner Join with multiple tables

1SELECT o.OrderID,
c.CompanyName,
2e.LastName
3FROM ((Orders o INNER JOIN Customers c ON o.CustomerID =
4c.CustomerID)
5INNER JOIN Employee e ON o.EmployeeID = e.EmployeeID);

Best practices

 Make sure you are pre-qualifying names

 Do not make unnecessary joins
 Think about the type if join you are making

Aliases and self joins

What?

1SELECT column_name
2FROM table_name AS alias_name;

Query example using alias

1SELECT vendor_name
2,product_name
3,product_price
4FROM Vendors AS v, Products AS p
5WHERE v.vendor_id = p.vendor_id;

Self joins

 Match customers from the same city

 Take the table and treat it like two separate tables
 Join the original table to itself
1SELECT column_names
2FROM Table1 T1, Table2 T2
3WHERE conditions;

EXAMPLES
 Matching the customers from the same city:

1SELECT A.CustomerName AS CustomerName1

2,B.CustomerName AS CustomerName2
3,A.City
4FROM Customers A, Customers B
5WHERE A.CustomerID = B.CustomerID
6AND A.City = B.City
7ORDER BY A.City;

Advanced joins: Left, Right and Full Outer Joins.

 SQL Lite only does left joins,

 Other DBMS use all joins.

Left Join

Return all records from the left table, and the matched records from the right table.

The result is NULL from the right side, if there is no match.

1SELECT Persons.LastName, Persons.FirstName, Orders.OrderNo

2FROM Persons
3LEFT JOIN Orders
4ON Persons.Id_P=Orders.Id_P
5ORDER BY Persons.LastName

Right Join

Return all records from the right table, and the matched records from the left table.

The result is NULL from the left side, if there is no match.

1SELECT Persons.LastName, Persons.FirstName, Orders.OrderNo

2FROM Persons
3RIGHT JOIN Orders
4ON Persons.Id_P=Orders.Id_P
5ORDER BY Persons.LastName

Left Joins can be turned into right joins by reversing the order of tables.

Full Outer Join

 Return all records when there is a match in either left table or right table records.

Select all customers and orders:

1SELECT Persons.LastName, Persons.FirstName, Orders.OrderNo

2FROM Persons
3FULL OUTER JOIN Orders
4ON Persons.Id_P=Orders.Id_P
5ORDER BY Persons.LastName

Unions

 The UNION operator is used to combine the result-set of two or more SELECT
statements.

 Each SELECT statement with UNION must have the same number of columns.

 Columns must have similar data types.

 The columns in each SELECT statement must be in the same order.
1SELECT column_name(s) FROM table1
2UNION
3SELECT column_name(s) FROM table2;

Note

 UNION selects only distinct values.

 Use UNION ALL to also select duplicate values!

Avoiding duplicate in a Union

1SELECT * FROM table1

2WHERE a = X
3UNION ALL
4SELECT * FROM table2
5WHERE b = Y AND a!=X;

Summary
Best practices using joins
 Check the number of records.
 Check for duplicates.
 Check the number of records each time you make a new join.
 Start smart: one table at a time.

Join syntax

 Check the particular syntax for your DBMS.

 SQLite does not do RIGHT and FULL OUTER joins.

Further readings

1. Difference Between Union and Union All - Optimal Performance Comparison

2. Thinking in SQL vs Thinking in Python

Module 3 Coding Assignment

All of the questions in this quiz refer to the open source Chinook Database. Please familiarize
yourself with the ER diagram in order to familiarize yourself with the table and column names in
order to write accurate queries and get the appropriate answers.

1. Using a subquery, find the names of all the tracks for the album
“Californication”.

1SELECT Name
2FROM Tracks
3WHERE AlbumId IN (
4SELECT AlbumId
5FROM Albums
6WHERE Title = "Californication");
1+-------------------+
2| Name |
3+-------------------+
 4| Around The World |
5| Parallel Universe |
6| Scar Tissue |
7| Otherside |
8| Get On Top |
9| Californication |
10| Easily |
11| Porcelain |
12| Emit Remmus |
13| I Like Dirt |
14+-------------------+
15(Output limit exceeded, 10 of 15 total rows shown)

2. Find the total number of invoices for each customer along with the customer’s
full name, city and email.

1SELECT Email,City,FirstName,
2(SELECT COUNT(*)
3FROM Invoices
4WHERE Invoices.CustomerId = Customers.CustomerId) AS Numbers
5FROM Customers;
+--------------------------+---------------------+-----------
+---------+
| Email | City | FirstName |
Numbers |
+--------------------------+---------------------+-----------
+---------+
| [email protected] | São José dos Campos | Luís |
17 |
2| [email protected] | Stuttgart | Leonie |
37 |
4| [email protected] | Montréal | François |
57 |
6| [email protected] | Oslo | Bjørn |
77 |
8| [email protected] | Prague | František |
97 |
10| [email protected] | Prague | Helena |
117 |
12| [email protected] | Vienne | Astrid |
137 |
14| [email protected] | Brussels | Daan |
157 |
| [email protected] | Copenhagen | Kara |
7 |
| [email protected] | São Paulo | Eduardo |
7 |
+--------------------------+---------------------+-----------
+---------+
(Output limit exceeded, 10 of 59 total rows shown)

3. Retrieve the track name, album, artist, and trackID for all the albums.
1SELECT Tracks.Name, Tracks.trackID, Albums.Title, Albums.ArtistId
2FROM Albums
3LEFT JOIN Tracks
4ON Albums.AlbumId = Tracks.AlbumId;
+-----------------------------------------+---------
+---------------------------------------+----------+
| Name | TrackId | Title
| ArtistId |
+-----------------------------------------+---------
+---------------------------------------+----------+
| For Those About To Rock (We Salute You) | 1 | For Those
1About To Rock We Salute You | 1 |
2| Put The Finger On You | 6 | For Those
3About To Rock We Salute You | 1 |
4| Let's Get It Up | 7 | For Those
5About To Rock We Salute You | 1 |
6| Inject The Venom | 8 | For Those
7About To Rock We Salute You | 1 |
8| Snowballed | 9 | For Those
9About To Rock We Salute You | 1 |
10| Evil Walks | 10 | For Those
11About To Rock We Salute You | 1 |
12| C.O.D. | 11 | For Those
13About To Rock We Salute You | 1 |
14| Breaking The Rules | 12 | For Those
15About To Rock We Salute You | 1 |
| Night Of The Long Knives | 13 | For Those
About To Rock We Salute You | 1 |
| Spellbound | 14 | For Those
About To Rock We Salute You | 1 |
+-----------------------------------------+---------
+---------------------------------------+----------+
(Output limit exceeded, 10 of 3503 total rows shown)

4. Retrieve a list with the managers last name, and the last name of the
employees who report to him or her.

1SELECT e1.LastName AS manager

2, e2.LastName AS employee
3FROM Employees e1, Employees e2
4WHERE e1.EmployeeId = e2.ReportsTo;
1+----------+----------+
2| manager | employee |
3+----------+----------+
4| Adams | Edwards |
5| Adams | Mitchell |
6| Edwards | Peacock |
7| Edwards | Park |
8| Edwards | Johnson |
9| Mitchell | King |
10| Mitchell | Callahan |
11+----------+----------+

5. Find the name and ID of the artists who do not have albums.
1SELECT Artists.Name, Artists.ArtistId
2FROM Artists
3LEFT JOIN Albums
4ON Artists.ArtistId = Albums.ArtistId
5WHERE Albums.AlbumId IS NULL;
1+----------------------------+----------+
2| Name | ArtistId |
3+----------------------------+----------+
4| Milton Nascimento & Bebeto | 25 |
5| Azymuth | 26 |
6| João Gilberto | 28 |
7| Bebel Gilberto | 29 |
8| Jorge Vercilo | 30 |
9| Baby Consuelo | 31 |
10| Ney Matogrosso | 32 |
11| Luiz Melodia | 33 |
12| Nando Reis | 34 |
13| Pedro Luís & A Parede | 35 |
14+----------------------------+----------+
15(Output limit exceeded, 10 of 71 total rows shown)

6. Use a UNION to create a list of all the employee’s & customer’s first names
and last names ordered by the last name in descending order.

1SELECT FirstName,LastName
2FROM Employees
3UNION
4SELECT FirstName,LastName
5FROM Customers
6ORDER BY LastName DESC;
1+-----------+--------------+
2| FirstName | LastName |
3+-----------+--------------+
4| Fynn | Zimmermann |
5| Stanisław | Wójcik |
6| František | Wichterlová |
7| Johannes | Van der Berg |
8| François | Tremblay |
9| Mark | Taylor |
10| Ellie | Sullivan |
11| Victor | Stevens |
12| Puja | Srivastava |
13| Jack | Smith |
14+-----------+--------------+
15(Output limit exceeded, 10 of 67 total rows shown)

7. See if there are any customers who have a different city listed in their billing
city versus their customer city.

1SELECT COUNT(*)
2FROM Customers
3LEFT JOIN Invoices
4ON Customers.CustomerId = Invoices.CustomerId
 5WHERE Customers.City <> Invoices.BillingCity;
1+----------+
2| COUNT(*) |
3+----------+
4| 0 |
5+----------+

Week 4: Modifying and analyzing data with

SQL
Working with text strings
Retrieve the data in the format you need

Support Joins

String Functions:

 Concatenations
1SELECT
2CompanyName
3,ContactName
4,CompanyName || '(' || ContactName || ')'
5FROM customers;

SQL server supports + instead of ||

 Trimming strings
Trims the leading or trailing space from a string

TRIM

RTRIM

LTRIM

 Substring
Returns the specified number of characters from a particular position of a given strings.
 1SUBSTR(string name, string position, number of characters to be
returned)
1SELECT firstname
2,SUBSTR(firstname,2,3)
3FROM employees;
 Upper and Lower
1SELECT UPPER(column_name) FROM tablename;

1SELECT LOWER(column_name) FROM tablename;

1SELECT UCASE(column_name) FROM tablename;

Working with date and time strings

 Dates are stored as datetypes

 Each DBMS uses it’s own variety of datetypes
DATE: Format YYYY-MM-DD

DATETIME: Format YYYY-MM-DD HH:MI:SS

TIMESTAMP: Format YYYY-MM-DD HH:MI:SS

Date and time string