Introduction to ETL

in Python
ETL IN PYTHON

Stefano Francavilla
CEO - Geowox
What is ETL?
Extract, Transform, Load

ETL IN PYTHON
What is ETL?
Extract, Transform, Load

ETL IN PYTHON
What is ETL?
Extract, Transform, Load

ETL IN PYTHON
The scene
Private equity fund called "DataCamp
Capital Group" (DCG Capital)

Residential assets

Monthly sales insights

In charge of the ETL pipeline

Stakeholder is the business analyst

ETL IN PYTHON
The pipeline

ETL IN PYTHON
The pipeline

ETL IN PYTHON
The pipeline

ETL IN PYTHON
In this lesson

ETL IN PYTHON
In this lesson

ETL IN PYTHON
In this lesson

ETL IN PYTHON
Requests
GET requests.get('<url>')

request/fetch data from a resource get_url = 'https://example.com/file.zip'

response = requests.get(get_url)
Social network GETs contacts content

response : requests.Response object

POST requests.post('<url>', data={'key':

'value'})
create/update a resource
post_url = 'https://example.com/page'
Social network POSTs user generated post_data = {'company': 'DCG Capital'}
content response = requests(post_url,
data=post_data)
response : requests.Response object

ETL IN PYTHON
Common requests attributes
response = requests.get('https://example.com/ny-properties-onsale.csv')

city, address, price

New York, "441 W 37th St FLOOR 2, New York, NY 10018", "$1,700,000"
New York, "22 W 57th St #Q56, New York, NY 10019", "$3,895,000"
New York, "788 9th Ave APT 1B, New York, NY 10019", "$1,000,000"

ETL IN PYTHON
Common requests attributes

Name Output Example

b'city, address, price\nNew
response.content raw bytes response payload
York...'
character encoded (e.g. UTF-8)
response.text 'city, address, price\nNew York...'
string payload
dictionary-like object which { 'Date': 'Wed, 20 Oct 2021
response.headers contains header payload as key- 18:49:30 GMT', 'Content-Length':
value '218'... }
status code returned by the
response.status_code 200
external service

ETL IN PYTHON
Zipfile
from zipfile import ZipFile class

Built-in zipfile function

Commonly used with two arguments: ZipFile(filepath, mode)

Read mode

with ZipFile(filepath, mode='r') as f:

f.namelist()
f.extract()

f.namelist() returns the list of les inside the opened .zip le

f.extract(filename, path=None) extracts a speci c le to a speci ed directory

ETL IN PYTHON
Zipfile: an example
from zipfile import ZipFile

filepath = "/my/custom/path/example.zip"
with ZipFile(filepath, mode='r') as f:
name_list = f.namelist()
print("List of files:", name_list)
extract_path = f.extract(name_list[0], path="/my/custom/path/")
print("Extract Path:", extract_path)

List of files: ["example.csv"]

Extract path: "/my/custom/path/example.csv"

ETL IN PYTHON
Let's practice!
ETL IN PYTHON
 Ask the right
questions
ETL IN PYTHON

Stefano Francavilla
CEO - Geowox
Where we are in the pipeline

ETL IN PYTHON
Dataset example
Date of Sale Postal Description
Address County Price (€)
(dd/mm/yyyy) Code of Property
Second-
Hand
123 WALKINSTOWN PARK, Dublin
12/02/2021 Dublin €297,000.00 Dwelling
WALKINSTOWN, DUBLIN 12 12
house
/Apartment
New
12 Oileain Na
Dublin Dwelling
04/01/2021 Cranoige.Cranogue Isl, Dublin €192,951.00
11 house
Balbutcher Lane, BALLYMUN
/Apartment

ETL IN PYTHON
Open a file
Built-in open() function

Commonly used with 2 arguments: open(filepath, mode)

Most common mode :

Character Meaning
'r' open for reading (default)
'w' open for writing

encoding argument example: open("file.csv", mode="r", encoding="windows-1252")

ETL IN PYTHON
Open a file: example
Read mode

with open('file.csv', mode="r", encoding="windows-1252"):

# Code here

Write mode

with open('file.csv', mode="w", encoding="windows-1252"):

# Code here

ETL IN PYTHON
CSV module
csv implements classes to read and write tabular data in CSV format

Dictionary form with csv.DictReader() and csv.DictWriter() functions

csv.DictReader(file, fieldnames=None...)

csv.DictWriter(file, fieldnames, ...)

keys = column names

values = row values

ETL IN PYTHON
Read in action
Code

with open("file.csv", mode="r") as csv_file:

reader = csv.DictReader(csv_file)
row = next(reader)
print(row)

Output

OrderedDict([
('Date of Sale (dd/mm/yyyy)', '03/01/2021'),('Postal Code', 'Dublin 4'),
('Address', '16 BURLEIGH COURT, BURLINGTON ROAD, DUBLIN 4'),('County', 'Dublin'),
('Price (€)', '€450,000.00'), ...])

ETL IN PYTHON
Write in action
Code

with open("file.csv", mode="w") as csv_file:

new_column_names = {"Date of Sale (dd/mm/yyyy)": "date_of_sale",
"Address": "address", "Postal Code": "postal_code", "County": "county",
"Price (€)": "price", "Description of Property": "description"}
writer = csv.DictWriter(csv_file, fieldnames=new_column_names)
# Write headers as first line
writer.writeheader()
# Write all rows in file
for row in reader:
writer.writerow(row)

ETL IN PYTHON
Let's practice!
ETL IN PYTHON
 Extracting
ETL IN PYTHON

Stefano Francavilla
CEO - Geowox
End goal
Automated pipeline

cron
Command line utility used for scheduling

execute.py
1. extract.py

2. transform.py

3. load.py

Download and process property transactions

ETL IN PYTHON
E(xtract)TL

ETL IN PYTHON
E(xtract)TL

ETL IN PYTHON
E(xtract)TL

ETL IN PYTHON
E(xtract)TL

ETL IN PYTHON
E(xtract)TL

ETL IN PYTHON
E(xtract)TL

ETL IN PYTHON
E(xtract)TL

ETL IN PYTHON
In this lesson: E(xtract)

ETL IN PYTHON
Create a folder
Make sure the downloaded_at folder exists

import os
Allows Python to interact with the operating system

os.makedirs() creates a folder recursively

Missing intermediate-level directories are created as well

os.makedirs(path, exist_ok=[True|False])

ETL IN PYTHON
Create a folder: an example
January 1st, 2021
rst time we run the cron job

Save the .zip le in the current month directory:

<root>/source/downloaded_at=2021-02-01
create .../downloaded_at=2021-01-01 folder

but... .../source folder does not exist yet

ETL IN PYTHON
Create a folder: an example
# Create <root>/source/downloaded_at=2021-01-01
path = "root/source/downloaded_at=2021-01-01"
os.makedirs(path, exist_ok=True)
# 1. Create source
# 2. Create downloaded_at=2021-01-01

/source/downloaded_at=2021-01-01/<zipfile_name>.zip

ETL IN PYTHON
Save ZIP file locally
open()
Commonly used with two arguments: open(filepath, mode)

Text vs binary mode :

Character Meaning
'w' open for writing in text format
'wb' open for writing in binary format

Write binary mode

with open('source/downloaded_at.../ppr-all.zip', mode="wb") as f:

f.write(...)

ETL IN PYTHON
Let's practice!
ETL IN PYTHON
 Project folder
structure
ETL IN PYTHON

Stefano Francavilla
CEO Geowox
ETL IN PYTHON
ETL IN PYTHON
ETL IN PYTHON
ETL IN PYTHON
ETL IN PYTHON
ETL IN PYTHON
ETL IN PYTHON
ETL IN PYTHON
ETL IN PYTHON
ETL IN PYTHON
ETL IN PYTHON
ETL IN PYTHON
ETL IN PYTHON
ETL IN PYTHON
ETL IN PYTHON
ETL IN PYTHON
ETL IN PYTHON
Extract, Transform and Load
# Import libraries

def methodX():
# Code here
pass
def methodY():
# Code here
pass

def main():
methodX()
methodY()

ETL IN PYTHON
Execute
# Import extract, transform and load
import extract, transform, load

# Ensure execute.py can only be ran from bash

if __name__ == "__main__":
# 1. Run Extract
extract.main()
# 2. Run Transform
transform.main()
# 3. Run Load
load.main()

python execute.py

ETL IN PYTHON
Let's practice!
ETL IN PYTHON