Scraping
without
programming
© Samantha Sunne
What's scraping?
How do journalists get data?
...from humans? ...from computers?
ay :) ay :(
Easy w Hard w
Ask
FOIA Download Scrape
nicely
ay :) ay :(
Easy w Hard w
Scraping is...
catching, collecting or coaxing data off the web.
Web scrapers are often called:
EXTRACTORS CRAWLERS
if they pull data from or if they pull data from
a single webpage multiple webpages
You'll need a basic
understanding of HTML
HTML is meant for computers, but some of it is understandable to humans.
Sometimes the code itself is interesting.
Jeb Bush’s
campaign site
Summary of the
movie “Die Hard”
Browser HTML
You can see it by right-clicking and clicking View Source.
This is called looking “under the hood."
Technique 1:
import HTML with Google Sheets
HTML content is wrapped in tags that look like this:
<>content content</>
There are a lot of tags, and a lot of them have different abbreviations, so you'll need to refer to an
HTML dictionary.
Some of them are intuitive, like tables:
<table>
Here is my table, everything between these table tags.
</table>
Webpages are actually made up of tons of "elements" - images, text chunks, headers, you name it.
You can grab any of them!
Video from Firefox
Elements can be inside each other. But any word you see immediately after a "<" is the name of the
element.
<table>
<tr>
A "table Inside a "table"
<td>Hi! I'm text inside a table.</td> Inside a "table
data" element
row" element
</tr> element
</table>
What we have here is an element inside an element inside an element.
But no worries, you can grab any of them! The larger elements will just include the smaller ones.
Version for humans Version for computers
<table>
<tr>
Hi! I'm text inside a table. <td>Hi! I'm text inside a table.
</td>
</tr>
</table>
How to scrape an HTML table with Google Sheets
You can use this Google Sheets formula to scrape a table or a list:
=ImportHTML(“url”, “element”, number of element)
The "url" is the link you are scraping.
The "element" is the HTML tag.
The number is the order the element is in.
tutorial
ed? Try this
Still confus
Example 1
=importHTML("sample.com",”table”,0)
This would import the first table on sample.com
(If it’s the first element, it’s number 0. I know, weird, it’s a computer thing.)
You can use View Source to figure out what element and number you want.
tutorial
ed? Try this
Still confus
This scrapes
<table>'s
For example, the IRE
Job Center
I used this formula
to scrape the IRE
jobs board:
https://www.ire.org
/jobs/
It also scrapes lists
(<ol>'s and <ul>'s)
Again, refer to the HTML
dictionary if you need to know the
HTML abbreviation for lists
This time, I scraped a list.
Remember, this isn’t just
any old list, but the first
“list” HTML element on the
page.
If you aren't sure if there is
a List element, check the
source code. (Right click >
View Source > Ctrl+F for
"<li>")
The importHTML formula can
only scrape lists and tables.
Technique 2:
import XML with Google Sheets
Browser XML
HTML
XML is also source code, but it's
in more of a "tree" format.
You can see
it by right
clicking and
clicking
"Inspect."
With XML, you can find the "XPath"
An XPath is like an address, or a map, to a very specific spot in the webpage’s code.
It looks like this:
This webpage, in the "wikitable" table, in the first row, in the fourth column
//table[@class='wikitable']/tr[1]/td[4]
On this the with the class in the first the fourth
page column
table “wikitable” row
It's easiest to highlight the element you want and Right Click > Inspect on it.
How to scrape XML data with Google Sheets
You can scrape everything at a particular XPath using this Google Sheets formula:
=importxml(“url”,”XPath”)
For example,
=importxml(“sample.com”,”//table[2]//tr”)
returns all the table rows ("tr") from the second table ("table[2]") in sample.com
ed? Try this tutorial
Still confus
XPaths open up a whole new world
all bold text: //b
all large text (known as headers): //h1
all headers with the word 'Parish': //h1[contains(.,'parish')]
all headers with the class 'Parish_name': //h1[@class='parish_name']
me tutorial!
r an aweso
See here fo
Here, I scraped:
the first column (<td>)
from all the table rows (<tr>)
in the table (<table>)
on ire.org/jobs
But what if your
data isn't in a
<table>?
Like the Talking Biz News
job board
Here, the page source
told me that all the job
titles are headers.
So I only scraped <h2>'s
For instance, the
dates on the
Talking Biz News
job board are
paragraph (<p>)
elements with
the class
'post-date'
Technique 3:
import RSS feeds with Google Sheets
This job board is
available as an RSS
feed
The Arab Reporters for
Investigative Journalism
How to scrape an RSS feed with Google Sheets
Google Sheets has a third import function you can use for scraping:
=importFeed(“source”,”items”)
It scrapes data from an RSS feed,
which is useful if you want to scrape something at regular intervals.
But we’re not going to do that in this session.
If you’re interested, check out this tutorial.
Here, I filled the first
column of my Google
Sheet with the item
titles - which in this
case, are the job
positions
Technique 4:
Writing code!
What a simple scrape would look like in Ruby
require 'Nokogiri'
require 'open-uri'
require 'csv'
url = "https://ire.org/jobs/"
html = Nokogiri::HTML(open(url))
csv = CSV.open("ire_jobs.csv", "w",{:col_sep => ",", :quote_char => '\'', :force_quotes => true})
html.xpath('//table//tr').each do |row|
tarray = []
row.xpath('td').each do |cell|
tarray << cell.text
end
csv << tarray
t here...
end
is scrip
csv.close
Copy th
Technique 5:
Point-and-click apps
OutWit Hub OutWit Hub is a desktop app that can
identify each HTML element on a
webpage and scrape it.
OutWit Hub is a desktop
app that can identify each
HTML element on a
webpage and scrape it.
(The free version limits how
many elements you can scrape.)
Some other good tools
● DownThemAll (Firefox add-on)
○ Highlights a whole list of links (or images/media) and downloads them
○ Not as robust as OutWit Hub
● Zapier and IFTTT (websites)
○ These automation tools can help keep track of new data, such as new Instagram posts
or tweets
○ Try: save RSS feed content to a Google Sheet or get emailed tweets from a certain
area
● Web Scraper (Chrome extension)
○ Has you build a sitemap for a site to be crawled
I wouldn’t recommend...
● Import.io (desktop app)
○ $300 a month! (But it does have a seven-day free trial.)
● Scraper (Chrome extensions)
○ Extremely basic; can only handle the simplest of tables. You’d be better off with
Google Sheets
● Helium Scraper (desktop app)
○ Works like other scraping tools, but costs a lot of money. (Although it does have a free
trial.)
● InfoExtractor (website)
○ Doesn't work.
If you want to delve further...
How to Feel Like You’re Hacking
My tutorial on getting data and metadata off the web
Using the web inspector for complex scrapes
Eric Sagara’s tutorial on using Python or Ruby for more advanced scraping
For more tools
I reviewed OutWit Hub and
DownThemAll for my newsletter, Tools
for Reporters. You can find new, useful
tools there - scraping and otherwise -
every week.
For questions, contact me at
[email protected] or
@samanthasunne. If I don’t know an
answer, I can point you to one.
xkcd
