Getting data

The problem!

Data exists in different sources and different formats

and we have to work with whatever format we get

or

We go to data analysis with data in the format we have,

not the format we want!
 Sources of data
local data
csv files
pdf files
xls files

web data
json
xml
html

database servers
mysql
postgres
mongoDB
 RESTful Web Services

REST: Representational State Transfer

“A network of web pages connected through

links and HTTP commands (GET, POST,
etc.)”

RESTful: A web service that conforms to the

REST standards
 RESTful Web Services

URLs: RESTful Web Services deliver

resources to the client. Each resources (html,
json, image, etc.) is associated with a URL
and an HTTP method

RESTful: A web service that conforms to the

REST standards
Example: Epicurious

http://www.epicurious.com

type “Tofu Chili” in search

box

http GET

new url:
http://www.epicurious.com/se
arch/Tofu%20Chili
Example: NYTIMES login

https://myaccount.nytimes.com/auth/login

type Email address and

Password in the form

http POST

new url:
http://www.nytimes.com/
 Example: Google GEOCODING API

HTTP GET request with

a JSON response

https://maps.googleapis.com/maps/api/geocode/json?address=Columbia_University,_New_York,_NY

All google API requests take the form:

<api_url>/<response_type>?<parameters>
json (or xml)
address=Columbia_University,_New_York,_NY

https://maps.googleapis.com/maps/api/geocode/
 What we need

The ability to

* create and send HTTP requests

* receive and process HTTP responses
* convert data residing in JSON/XML/HTML
format into python objects
 Python libraries for
getting web data
* Send an http request and get an http response
* requests
* urllib.requests (urllib2 on python2)

* parse the response and extract data

* json
* lxml
* BeautifulSoup, Selenium (for html data)
 http requests

requests: Python library for handling http requests and responses

http://docs.python-requests.org/en/master/
 using requests
* Import the library
import requests
* Construct the url
url = “http://www.epicurious.com/search/Tofu+Chili”
* Send the request and get a response
response = requests.get(url)
* Check if the request was successful
if response.status_code == 200:
“SUCCESS”!!!!
else:
“FAILURE”!!!
 response status codes
* 200 or 201
the request response cycle worked as planned
* Other 200s
the request response cycle worked but there is
additional information associated with the response
* 400s
there was an error (page not found/malformed
request/etc.)
* General rule of thumb
check if the status code was 200 for accessing data
through a GET or POST HTTP request
* response.content response content
returns the content of the HTTP response
* response.content.decode(‘utf-8’)
if the content is byte encoded (which it usually is!),
converts it into unicode - a python str
* What is unicode?
http://unicode.org/standard/WhatIsUnicode.html
* General rule of thumb
web pages are usually returned as byte strings and need
to be decoded. utf-8 is the usual decoding (but not always!)
* What is utf-8?
https://www.w3schools.com/charsets/ref_html_utf8.asp
 Try this

1. Open https://en.wikipedia.org/wiki/main_page using the

requests library
2. Check the status code. Did your request work?
3. Get the content. Decode it. Then search the page for the string
“Did you know” using the str find function
4. If your find function returned a positive number - Great!
5. If it returned -1 (that means it was not found), you’ve done
something wrong. Try figuring out what went wrong
 web data formats

* HTML
the common format when scraping web pages for data
* JSON or XML
usually when accessing data through an API or when
the server is explicitly sharing data with you
 JSON
JavaScript Object Notation

- Standard for "serializing" data objects for

storage or transmission
- Human-readable, useful for data
interchange
- Also useful for representing and storing
semistructured data
- Stored as plain (byte strings or utf-8 strings)
text
JSON constructs and
Python equivalents

JSON Python

number int,float

string str

Null None

true/false True/False

Object dict

Array list
 python json library

json.loads(<str>): converts a JSON string to

python objects

json.dumps(<python_object>): converts a
python object into a JSON formatted string
 python json library

Converts a json string to an equivalent

python type

str
import json
json_data = '[{"b": [2, 4], "c": 3.0, "a": "A"}]'
python_data = json.loads(json_data)

list
 python json library

Converts a json string to an equivalent

python type

str import json

data_string = json.dumps(python_data)

list
 requests and json

The response object handles json

the requests library handles
spaces in a url for you

address=“Columbia University, New York, NY"

url="https://maps.googleapis.com/maps/api/geocode/json?address=%s" % (address)
response = requests.get(url).json()

response now points to a

python data object
 requests and json

requests.json returns an exception if the

JSON object is ill-formed

note that some http errors are returned as

JSON

always check for exceptions!

 requests and json
address="Columbia University, New York, NY"
url="https://maps.googleapis.com/maps/api/geocode/json?address=%s" % (address)
try:
response = requests.get(url)
if not response.status_code == 200:
print("HTTP error",response.status_code)
else:
try:
response_data = response.json()
except:
print("Response not in valid JSON format")
except:
print("Something went wrong with requests.get")
print(type(response_data))
 requests and json: example

Let’s take a look at the JSON object returned by Google Geocoding API
 Working with json

Problem 1: Write a function that takes an

address as an argument and returns a
(latitude, longitude) tuple

def get_lat_lng(address_string):
#python code goes here
 Solution to problem 1

def get_lat_lng(address):
url="https://maps.googleapis.com/maps/api/geocode/json?address=%s"%(address)
import requests
response = requests.get(url)
if response.status_code == 200:
lat = response.json()['results'][0]['geometry']['location']['lat']
lng = response.json()['results'][0]['geometry']['location']['lng']
return lat,lng
else:
return None
 xml

Extensible Markup Language

- Tree structure
- Tagged elements (nested)
- Attributes
- Text (leaves of the tree)
 <Last_Name>Berenholtz</Last_Name>

</Author> xml: Example

</Authors>

</Book>

<Book ISBN="ISBN-13:978-1579128562" Price="15.80">

<Remark>

Five Hundred Buildings of New York and over one million other books are available for Amazon Kindle.

</Remark>

<Title>Five Hundred Buildings of New York</Title>

<Authors>

<Author Residence="Beijing">

<First_Name>Bill</First_Name>

<Last_Name>Harris</Last_Name>

</Author>
 XML Tree
ISBN="ISBN-13:978-1599620787" Price="15.23" Weight="1.5"
$15.23
1.5oz Bookstore
ISBN-13:978-1599620787

Book Book

Title Authors Authors Title

subtree subtree
Author Author
New York Deco

firstname lastname

Bill Harris
 lxml: Python xml library

xml tree object definition

from lxml import etree

root = etree.XML(data)
print(root.tag) prints the root tag
(Bookstore in our
example)

http://lxml.de/1.3/tutorial.html
 lxml: Python xml library
Examining the tree

print(etree.tostring(root, pretty_print=True).decode("utf-8"))
 lxml: Iterating over
children of a tag

root is a collection of
children so we can
iterate over it
for child in root:
print(child.tag)
lxml: Iterating over elements

for element in root.iter():

print(element.tag)

iter is an ‘iterator’. it
generates a sequence of
elements in the order
they appear in the xml
code
 lxml: Iterating over elements

iterates over the tree

matching only those
elements that have
“Author” as the tag
for element in root.iter("Author"):
print(element.find('First_Name').text,element.find('Last_Name').text)

returns the first element

that has “First_Name”
as the tag
 lxml: using XPath

XPath: expression for

navigating through an
xml tree

for element in root.findall('Book/Title'):

print(element.text)
 Try this

Find the last names of all authors in the tree “root” using path
 Solution

for element in root.findall('Book/Authors/Author/Last_Name'):

print(element.text)
 lxml: Finding by attribute value

root.find('Book[@Weight="1.5"]/Authors/Author/First_Name').text
 Problem 3

Print first and last names of all authors who live in

New York City
from lxml import etree
root = etree.XML(data_string)
for author_element in root.findall('Book[@ISBN="ISBN-13:978-1579128562"]/Authors/Author'):
f_name = author_element.find('First_Name').text
l_name = author_element.find('Last_Name').text
print(f_name,l_name)
 HTML

HyperText Markup Language

- Formats text
- Tagged elements (nested)
- Attributes
- Derived from SGML (but who
cares!)
- Closely related to XML
- Can contain runnable scripts
 HTML/CSS

Study this on your own!

Make sure you’ve reviewed the first two topics (“Intro to html” and
“Intro to CSS”) on Khan Academy
https://www.khanacademy.org/computing/computer-programming/html-css
getting data from the
web
 creeping, crawling, pouncing!

Web scraping: Automating the process of extracting information from web pages
* for data collection and analysis
* for incorporating in a web app

APIs (Application Programming Interface): Functions and libraries for communicating with
specific web servers
* for data collection and analysis
* for incorporating in a web app

Web crawling: Automating the process of traversing links on web pages

* for indexing the web
* for collecting data from multiple web sites
 Legal and ethical
Legal issues issues
➡ Often against the ‘Terms of Use’ of a web site
➡ but, regardless, murky and and not fully settled
➡ See: https://en.wikipedia.org/wiki/Web_scraping#Legal_issues

➡ probably depends upon three things:

๏ factual, non-proprietary data is generally ok
๏ proprietary data scraping depends
Ethical on what you do with it
issues
๏ potential or actual damage to the scrapee
➡ Public vs. private information. Public information is rarely unethical
➡ Purpose. Ethical or unethical depends on why you’re scraping the web site
➡ Always better to try and get the information openly using APIs or contacting the
owner of the server
➡ Is there a public interest involved? If yes, it’s probably ethical to scrape
 Libraries for web scraping

requests: Python library for connecting to a web page, managing

the connection and retrieving contents of the page

Beautiful Soup: A library that utilizes the ‘tag structure’ of

an html page to quickly parse the contents of a page and retrieve data

Selenium: A library that utilizes the ‘tag structure’ of

an html page to execute javascript scripts on the page and retrieve data.
Slower than Beautiful Soup but gets around the ‘javascript’ problem
 BeautifulSoup4
➡ HTML (and XML) parser
➡ Uses ‘tags’

➡ Creates a parse tree (using lxml/html5lib or other

python parser)
➡ Can handle incomplete tagging

➡ tags are organized in hierarchical dictionaries

https://www.crummy.com/software/BeautifulSoup/bs4/doc/
 bs4

initialize bs4 object: BeautifulSoup(document,parser)

parser: lxml (fast) or html5lib (slower but more robust)

import requests
from bs4 import BeautifulSoup
url = "http://www.epicurious.com/search/Tofu%20Chili"
response = requests.get(url)
page_soup = BeautifulSoup(response.content,'lxml')
print(page_soup.prettify())
page_soup is the object from
which we will extract the
data we need
 Unique data identifiers

➡ We want to create a list of recipes and links to the

recipes
➡ We need to figure out how to ‘programmatically’
extract each recipe name and recipe link

➡ Search for the tag with a unique attribute value that

identifies recipes and recipe links
➡ The easiest way is to examine the page source on a
browser
 Finding unique data identifiers
['article-content-card']

['gallery-content-card']

for tag in page_soup.find_all(‘article'): ['recipe-content-card']

print(tag.get(‘class’) ['article-content-card']

['recipe-content-card']
prints
This gets the innermost tags with the recipe name. ['recipe-content-card']

['recipe-content-card']

['article-content-card']

['recipe-content-card']

['recipe-content-card']
looks like class=‘recipe-content-card’ gives us the
['recipe-content-card']
recipes
['recipe-content-card']

['article-content-card']

['article-content-card']

['recipe-content-card']
 bs4 functions
<tag>.find(<tag_name>,attribute=value) finds the first matching child tag (recursively)

<tag>.find_all(<tag_name>,attribute=value) finds all matching child tags (recursively)

<tag>.get_text() returns the marked up text

<tag>.parent returns the (immediate) parent

<tag>.parents returns all parents (recursively)

<tag>.children returns the (direct) children

<tag>.descendants returns all children (recursively)

<tag>.get(attribute) returns the value of the specified attribute

<tag>.name returns the name of a tag

<tag>.attrs returns all the attributes of a tag

 Problem 4: Extract recipes and recipe links

Write a function epicurious_recipes(search_string) that returns

the list of recipes and links associated with search_string

Call the function with a search_string, open the link associated

with the first recipe, then return the ingredients and preparation
instructions associated with that link
 Problem 4: Step 1

given a recipe url, get the description and the ingredients

def get_recipe_detail(url):
import requests
from bs4 import BeautifulSoup
html_data = requests.get(url)
if not html_data.status_code == 200:
return '',[]
recipe_data = BeautifulSoup(html_data.content,'lxml')
description = get_description(recipe_data)
ing_list = get_ingredients(recipe_data)
return description,ing_list
need to write these two functions
 Problem 4: Step 2

write the get_description function

def get_description(recipe_page_data):
description_tag = recipe_page_data.find('div',itemprop = 'description')
if description_tag:
return description_tag.get_text()
return ''
 Problem 4: Step 3

write the get_ingredients function

def get_ingredients(ing_page_data):
ing_list = list()
for item in ing_page_data.find_all('li',class_='ingredient'):
ing_list.append(item.get_text())
return ing_list
 Problem 4: Step 4

write the function that:

takes key words as an argument
and returns a list of tuples
(name,description,ingredients)
def get_recipes(keywords):
recipe_list = list()
Problem 4: Step 4 code
import requests
from bs4 import BeautifulSoup
url = "http://www.epicurious.com/search/" + keywords
response = requests.get(url)
if not response.status_code == 200:
return None
try:
results_page = BeautifulSoup(response.content,'lxml')
recipes = results_page.find_all('article',class_="recipe-content-card")
for recipe in recipes:
recipe_link = "http://www.epicurious.com" + recipe.find('a').get('href')
recipe_name = recipe.find('a').get_text()
try:
recipe_description = recipe.find('p',class_='dek').get_text()
except:
recipe_description = ''
recipe_list.append((recipe_name,recipe_link,recipe_description))
return recipe_list
 logging in with requests and beautifulsoup
➡ Figure out the login url
➡ https://en.wikipedia.org/w/index.php?title=Special:UserLogin&returnto=Main+Page
➡ Look for the login form in the html source
➡ form_tag = page_soup.find('form')
➡ Look for ALL the inputs in the login form (some may be tricky!)
➡ input_tags = form_tag.find_all('input')
➡ Create a Python dict object with key,value pairs for each input
➡ Use requests.session to create an open session object
➡ Send the login request (POST)
➡ Send followup requests keeping the sessions object open
 Setting up the inputs

payload = {
'wpName': username,
'wpPassword': password,
'wploginattempt': 'Log in',
'wpEditToken': "+\\",
'title': "Special:UserLogin",
'authAction': "login",
'force': "",
'wpForceHttps': "1",
'wpFromhttp': "1",
#'wpLoginToken': ‘', #We need to read this from the page
}
 Extracting token information

wpLoginToken: the value of this attribute is provided by the page. we need

to extract it.

login_page_response =
s.get('https://en.wikipedia.org/w/index.php?title=Special:UserLogin&returnto
=Main+Page')
soup = BeautifulSoup(login_page_response.content,'lxml')
token = soup.find('input',{'name':"wpLoginToken"}).get('value')
 Finalizing session parameters
username=<your username>
password=<your password>

def get_login_token(response):
soup = BeautifulSoup(response.text, 'lxml')
token = soup.find('input',{'name':"wpLoginToken"}).get('value')
return token

payload = {
'wpName': username,
'wpPassword': password,
'wploginattempt': 'Log in',
'wpEditToken': "+\\",
'title': "Special:UserLogin",
'authAction': "login",
'force': "",
'wpForceHttps': "1",
'wpFromhttp': "1",
#'wpLoginToken': '',
}
 Activating session

with requests.session() as s:
response = s.get('https://en.wikipedia.org/w/index.php?title=Special:UserLogin&returnto=Main+Page')
payload['wpLoginToken'] = get_login_token(response)
#Send the login request
response_post = s.post('https://en.wikipedia.org/w/index.php?title=Special:UserLogin&action=submitlogin&type=login',
data=payload)
#Get another page and check if we’re still logged in
response = s.get('https://en.wikipedia.org/wiki/Special:Watchlist')