09/05/2023 09:48 01-Python_02-Data-Sourcing

Data Sourcing

Finding Data
I can export it from a software (CSV)
I know it exists somewhere in a database (SQL)
It's on this website I visit daily (Scraping)
I have found a service (API) that gives access to it
...

Plan
1. Reading/Writing CSV (the hard way)
2. Consuming an API
3. Scraping a website

Next lectures
Databases & SQL

CSV
A comma-separated values file is a delimited text file that uses a comma to separate
values. A CSV file stores tabular data (numbers and text) in plain text. Each line of the file is
a data record. Each record consists of one or more fields, separated by commas.

Source: Wikipedia (https://en.wikipedia.org/wiki/Comma-separated_values)

Example
👉 On https://people.sc.fsu.edu/~jburkardt (https://people.sc.fsu.edu/~jburkardt/data/csv/csv.html), let's look
at the addresses.csv

https://kitt.lewagon.com/camps/1173/lectures/content/01-Python_02-Data-Sourcing.html 1/9
09/05/2023 09:48 01-Python_02-Data-Sourcing

In [ ]:

%%bash
mkdir -p data
curl -s https://people.sc.fsu.edu/~jburkardt/data/csv/addresses.csv > data/addresses.cs
v
cat data/addresses.csv

John,Doe,120 jefferson st.,Riverside, NJ, 08075

Jack,McGinnis,220 hobo Av.,Phila, PA,09119
"John ""Da Man""",Repici,120 Jefferson St.,Riverside, NJ,08075
Stephen,Tyler,"7452 Terrace ""At the Plaza"" road",SomeTown,SD, 91234
,Blankman,,SomeTown, SD, 00298
"Joan ""the bone"", Anne",Jet,"9th, at Terrace plc",Desert City,CO,00123

CSV Reading

In [ ]:

import csv

with open('data/addresses.csv') as csvfile:

reader = csv.reader(csvfile, skipinitialspace=True)
for row in reader:
# row is a `list`
print(row)

['John', 'Doe', '120 jefferson st.', 'Riverside', 'NJ', '08075']

['Jack', 'McGinnis', '220 hobo Av.', 'Phila', 'PA', '09119']
['John "Da Man"', 'Repici', '120 Jefferson St.', 'Riverside', 'NJ', '0807
5']
['Stephen', 'Tyler', '7452 Terrace "At the Plaza" road', 'SomeTown', 'SD',
'91234']
['', 'Blankman', '', 'SomeTown', 'SD', '00298']
['Joan "the bone", Anne', 'Jet', '9th, at Terrace plc', 'Desert City', 'C
O', '00123']

CSV with Headers

In [ ]:

%%bash
curl -s https://people.sc.fsu.edu/~jburkardt/data/csv/biostats.csv > data/biostats.csv
head -n 3 data/biostats.csv

"Name", "Sex", "Age", "Height (in)", "Weight (lbs)"

"Alex", "M", 41, 74, 170
"Bert", "M", 42, 68, 166

https://kitt.lewagon.com/camps/1173/lectures/content/01-Python_02-Data-Sourcing.html 2/9
09/05/2023 09:48 01-Python_02-Data-Sourcing

CSV with Headers

In [ ]:

import csv

with open('data/biostats.csv') as csvfile:

reader = csv.DictReader(csvfile, skipinitialspace=True)
for row in reader:
# row is a dict
print(row['Name'], row['Sex'], int(row['Age']))

Alex M 41
Bert M 42
Carl M 32
Dave M 39
Elly F 30
Fran F 33
Gwen F 26
Hank M 30
Ivan M 53
Jake M 32
Kate F 47
Luke M 34
Myra F 23
Neil M 36
Omar M 38
Page F 31
Quin M 29
Ruth F 28

Writing a CSV

In [ ]:

beatles = [
{ 'first_name': 'John', 'last_name': 'lennon', 'instrument': 'guitar'},
{ 'first_name': 'Ringo', 'last_name': 'Starr', 'instrument': 'drums'}
]

In [ ]:

import csv

with open('data/beatles.csv', 'w') as csvfile:

writer = csv.DictWriter(csvfile, fieldnames=beatles[0].keys())
writer.writeheader()
for beatle in beatles:
writer.writerow(beatle)

https://kitt.lewagon.com/camps/1173/lectures/content/01-Python_02-Data-Sourcing.html 3/9
09/05/2023 09:48 01-Python_02-Data-Sourcing

In [ ]:

%%bash
cat data/beatles.csv

first_name,last_name,instrument
John,lennon,guitar
Ringo,Starr,drums

API
An application programming interface (API) is an interface or communication protocol
between a client and a server intended to simplify the building of client-side software. It has
been described as a “contract” between the client and the server.

Source Wikipedia (https://en.wikipedia.org/wiki/Application_programming_interface)

HTTP
A client-server protocol based on a request/response cycle.

Modern Web API

RESTful ( GET , POST , etc.)
Returns JSON (https://en.wikipedia.org/wiki/JSON#JSON_sample)

👉 Examples (https://github.com/public-apis/public-apis)

Requests: HTTP for Humans™

👉 Documentation (https://pypi.org/project/requests/)

Basic request

In [ ]:

import requests

url = 'https://api.github.com/users/ssaunier'
response = requests.get(url).json()

print(response['name'])

Sébastien Saunier

https://kitt.lewagon.com/camps/1173/lectures/content/01-Python_02-Data-Sourcing.html 4/9
09/05/2023 09:48 01-Python_02-Data-Sourcing

Example
Let's use the Open Library Books API (https://openlibrary.org/dev/docs/api/books).

This API documentation is not that good - let's decipher it together!

Query Parameters:

Provide an ISBN ( bibkeys )

Options:
format=json
jscmd=data

Livecode: Let's find the book title behind ISBN 9780747532699

In [ ]:

import requests

isbn = '0-7475-3269-9'
key = f'ISBN:{isbn}'

response = requests.get(
'https://openlibrary.org/api/books',
params={'bibkeys': key, 'format':'json', 'jscmd':'data'},
).json()

print(response[key]['title'])

Harry Potter and the Philosopher's Stone

Web Scraping

HTTP (again)
This time, we'll have to deal with HTML (~unstructured data)

https://kitt.lewagon.com/camps/1173/lectures/content/01-Python_02-Data-Sourcing.html 5/9
09/05/2023 09:48 01-Python_02-Data-Sourcing

HTML
Right click -> Inspect Element on any website

<!DOCTYPE html>
<html>
<head>
<title>Title of the browser tab</title>
</head>
<body>
<h1>Main title</h1>
<p>Some content</p>
<ul id="results">
<li class="result">Result 1</li>
<li class="result">Result 2</li>
</ul>
</body>
</html>

HTML Vocabulary

BeautifulSoup
The Python package to browse HTML (and XML!)

👉 Documentation (https://www.crummy.com/software/BeautifulSoup/bs4/doc/)

https://kitt.lewagon.com/camps/1173/lectures/content/01-Python_02-Data-Sourcing.html 6/9
09/05/2023 09:48 01-Python_02-Data-Sourcing

Typical Web Scraper with BeautifulSoup

import requests
from bs4 import BeautifulSoup

response = requests.get(url)
soup = BeautifulSoup(response.content, "html.parser")

# You now can query the `soup` object!

soup.title.string
soup.find('h1')
soup.find_all('a')
# etc...

Searching by element name

<p>A paragraph</p>

<article>An article...</article>
<article>Another...</article>

paragraph = soup.find("p")
articles = soup.find_all("article")

Searching by id

<a href="https://www.lewagon.com" id="wagon">Le Wagon</a>

item = soup.find(id="wagon")

https://kitt.lewagon.com/camps/1173/lectures/content/01-Python_02-Data-Sourcing.html 7/9
09/05/2023 09:48 01-Python_02-Data-Sourcing

Searching by CSS Class

<ul>
<li class="pizza">Margharita</li>
<li class="pizza">Calzone</li>
<li class="pizza">Romana</li>
<li class="dessert">Tiramisu</li>
</ul>

items = soup.find_all("li", class_="pizza")

Live-code
Let's scrape IMDb Top 50 (https://www.imdb.com/list/ls055386972/) and extract the following information for
each movie:

Title
Duration

Let's build a list ( movies ) of dict ( { 'title': ?, 'duration': ? } )

Solution

In [ ]:

import requests
from bs4 import BeautifulSoup

response = requests.get("https://www.imdb.com/list/ls055386972/", headers={"Accept-Lang

uage":"en-US"})
soup = BeautifulSoup(response.content, "html.parser")

movies = []
for movie in soup.find_all("div", class_="lister-item-content"):
title = movie.find("h3").find("a").string
duration = int(movie.find("span", class_="runtime").string.strip(' min'))
movies.append({'title': title, 'duration': duration})

print(movies[0:2])

[{'title': 'The Godfather', 'duration': 175}, {'title': "Schindler's Lis

t", 'duration': 195}]

https://kitt.lewagon.com/camps/1173/lectures/content/01-Python_02-Data-Sourcing.html 8/9
09/05/2023 09:48 01-Python_02-Data-Sourcing

Bonus

Then convert cURL command to Python Requests (https://sqqihao.github.io/trillworks.html)

Your turn!
There are 3 challenges for this lecture:

1. Reading and writing CSVs (the hard way!)

2. Making API calls with Python
3. Scraping a website
4. (Optional) Scraping a JavaScript client-side rendered website

https://kitt.lewagon.com/camps/1173/lectures/content/01-Python_02-Data-Sourcing.html 9/9