Getting Data from the Web with R

Part 5: Handling JSON data

Gaston Sanchez
April-May 2014
Content licensed under CC BY-NC-SA 4.0

Readme

License:
Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License
http://creativecommons.org/licenses/by-nc-sa/4.0/

You are free to:

Share copy and redistribute the material
Adapt rebuild and transform the material

Under the following conditions:

Attribution You must give appropriate credit, provide a link to the license, and indicate if changes were
made.
NonCommercial You may not use this work for commercial purposes.
Share Alike If you remix, transform, or build upon this work, you must distribute your contributions under
the same license to this one.

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

3 / 39

Lectures Menu

Slide Decks
1. Introduction
2. Reading files from the Web
3. Basics of XML and HTML
4. Parsing XML / HTML content
5. Handling JSON data
6. HTTP Basics and the RCurl Package
7. Getting data via Web Forms
8. Getting data via Web APIs

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

4 / 39

JSON Data

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

5 / 39

Goal

JSON
The goal of these slides is to provide an introduction
for handling JSON data in R

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

6 / 39

Synopsis

In a nutshell
Well cover the following topics:
I

JSON Basics

R packages for JSON data

Reading JSON data from the Web

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

7 / 39

Some References
I

XML and Web Technlogies for Data Sciences with R

by Deb Nolan and Duncan Temple Lang

Introducing JSON
http://www.json.org/

R package RJSONIO
http://cran.r-project.org/web/packages/RJSONIO/index.html

R package jsonlite
http://cran.r-project.org/web/packages/jsonlite/vignettes/
json-mapping.pdf

R package rjson
http://cran.r-project.org/web/packages/rjson/index.html

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

8 / 39

JSON Basics

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

9 / 39

Basics First

Fundamentals
JSON stands for JavaScript Object Notation
and it is a format for representing data
I

general purpose format

lightweight format

widely popular

fairly simple

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

10 / 39

Basics First

Why should we care?

When working with data from the Web, well inevitably find some
JSON data
I

JSON can be used directly in JavaScript code for Web pages

many Web APIs provide data in JSON format

R has packages designed to handle JSON data

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

11 / 39

Understanding JSON

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

12 / 39

Understanding JSON

JSON Data Types

JSON Data Containers

null

square brackets [ ]

true

curly brackets { }

false
number
string

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

13 / 39

JSON Arrays

Unnamed Arrays
Square brackets [ ] are used for ordered unnamed arrays
I

[ 1, 2, 3, ...

[ true, true, false, ...

]
]

Named Arrays
Curly brackets { } are used for named arrays
I

{ "dollars" :

{ "city" :

gastonsanchez.com

5, "euros" :

20, ...

"Berkeley", "state" :

Getting data from the web with R

"CA", ...

CC BY-SA-NC 4.0

14 / 39

JSON Arrays
Containers can be nested

Example A

Example B

[
"name": ["X", "Y", "Z"],
"grams": [300, 200, 500],
"qty": [4, 5, null],
"new": [true, false, true],

{ "name": "X",
"grams": 300,
"qty": 4,
"new": true },
{ "name": "Y",
"grams": 200,
"qty": 5,
"new": false },
{ "name": "Z",
"grams": 500,
"qty": null,
"new": true}

]
gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

15 / 39

Data Table Toy Example

Imagine we have some data
Name
Anakin
Amidala
Luke
Leia
Obi-Wan
Han
Palpatine
R2-D2

Gender
male
female
male
female
male
male
male
unknown

Homeland
Tatooine
Naboo
Tatooine
Alderaan
Stewjon
Corellia
Naboo
Naboo

Born
41.9BBY
46BBY
19BBY
19BBY
57BBY
29BBY
82BBY
33BBY

Jedi
yes
no
yes
no
yes
no
no
no

There are several ways to represent this data in JSON format

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

16 / 39

One way to represent data

[
{
"Name": "Anakin",
"Gender": "male",
"Homeworld": "Tatooine",
"Born": "41.9BBY",
"Jedi": "yes"
},
...
{
"Name": "R2-D2",
"Gender": "unknown",
"Homeworld": "Naboo",
"Born": "33BBY",
"Jedi": "no"
},
]

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

17 / 39

Another way to represent data

{
"Name": [ "Anakin", "Amidala", "Luke", ... , "R2-D2" ],
"Gender": [ "male", "female", "male", ... , "unknown" ],
"Homeworld": [ "Tatooine", "Naboo", "Tatooine", ... , "Naboo" ],
"Born": [ "41.9BBY", "46BBY", "19BBY", ... , "33BBY" ],
"Jedi": [ "yes", "no", "yes", ... , "no" ]
}

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

18 / 39

JSON R packages

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

19 / 39

R packages

R packages for JSON

R has 3 packages for working with JSON data
I

"RJSONIO" by Duncan Temple Lang

"rjson" by Alex Couture-Beil

"jsonlite" by Jeroen Ooms, Duncan Temple Lang, Jonathan

Wallace

All packages provide 2 main functions toJSON() and

fromJSON() that allow conversion to and from data in JSON
format, respectively.
Well focus on the functions from "RJSONIO"

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

20 / 39

R package RJSONIO

R package "RJSONIO"
If you dont have "RJSONIO" youll have to install it:
# install RJSONIO
install.packages("RJSONIO", dependencies = TRUE)

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

21 / 39

R package RJSONIO

Main functions
There are 2 primary functions in "RJSONIO"
I

toJSON() converts an R object to a string in JSON

fromJSON() converts JSON content to R objects

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

22 / 39

toJSON()
Function toJSON()
toJSON(x, container = isContainer(x, asIs, .level),
collapse = "\n", ...)
I

x the R object to be converted to JSON format

container whether to treat the object as a vector/container or

a scalar

collapse string used as separator when combining the

individual lines of the generated JSON content

... additional arguments controlling the JSON formatting

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

23 / 39

fromJSON()
Function fromJSON()
fromJSON(content, handler = NULL, default.size = 100,
depth = 150L, allowComments = TRUE, ...)
I

content the JSON content: either a file name or a character

string

handler R object responsible for processing each individual

token/element

deafult.size size to use for arrays and objects in an effort to

avoid reallocating each time we add a new element.

depth maximum number of nested JSON levels

allowComments whether to allow C-style comments within the

JSON content

... additional parameters

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

24 / 39

Data Table Toy Example

Imagine we have some tabular data

Name
Anakin
Amidala
Luke
Leia
Obi-Wan
Han
Palpatine
R2-D2

gastonsanchez.com

Gender
male
female
male
female
male
male
male
unknown

Homeland
Tatooine
Naboo
Tatooine
Alderaan
Stewjon
Corellia
Naboo
Naboo

Getting data from the web with R

Born
41.9BBY
46BBY
19BBY
19BBY
57BBY
29BBY
82BBY
33BBY

Jedi
yes
no
yes
no
yes
no
no
no

CC BY-SA-NC 4.0

25 / 39

R Data Frame
# toy data
sw_data = rbind(
c("Anakin", "male", "Tatooine", "41.9BBY", "yes"),
c("Amidala", "female", "Naboo", "46BBY", "no"),
c("Luke", "male", "Tatooine", "19BBY", "yes"),
c("Leia", "female", "Alderaan", "19BBY", "no"),
c("Obi-Wan", "male", "Stewjon", "57BBY", "yes"),
c("Han", "male", "Corellia", "29BBY", "no"),
c("Palpatine", "male", "Naboo", "82BBY", "no"),
c("R2-D2", "unknown", "Naboo", "33BBY", "no"))
# convert to data.frame and add column names
swdf = data.frame(sw_data)
names(swdf) = c("Name", "Gender", "Homeworld", "Born", "Jedi")
swdf
##
##
##
##
##
##
##
##
##

Name Gender Homeworld

Born Jedi
1
Anakin
male Tatooine 41.9BBY yes
2
Amidala female
Naboo
46BBY
no
3
Luke
male Tatooine
19BBY yes
4
Leia female Alderaan
19BBY
no
5
Obi-Wan
male
Stewjon
57BBY yes
6
Han
male Corellia
29BBY
no
7 Palpatine
male
Naboo
82BBY
no
8
R2-D2 unknown
Naboo
33BBY
no

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

26 / 39

From R to JSON
# load RJSONIO
library(RJSONIO)
# convert R data.frame to JSON
sw_json = toJSON(swdf)
# what class?
class(sw_json)
## [1] "character"
# display JSON format
cat(sw_json)
##
##
##
##
##
##
##

{
"Name": [ "Anakin", "Amidala", "Luke", "Leia", "Obi-Wan", "Han", "Palpatine", "R2-D2" ],
"Gender": [ "male", "female", "male", "female", "male", "male", "male", "unknown" ],
"Homeworld": [ "Tatooine", "Naboo", "Tatooine", "Alderaan", "Stewjon", "Corellia", "Naboo", "Naboo" ],
"Born": [ "41.9BBY", "46BBY", "19BBY", "19BBY", "57BBY", "29BBY", "82BBY", "33BBY" ],
"Jedi": [ "yes", "no", "yes", "no", "yes", "no", "no", "no" ]
}

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

27 / 39

From JSON to R
# convert JSON string to R list
sw_R = fromJSON(sw_json)
# what class?
class(sw_R)
## [1] "list"
# display JSON format
sw_R
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##

$Name
[1] "Anakin"
"Amidala"
[7] "Palpatine" "R2-D2"
$Gender
[1] "male"
"female"
[8] "unknown"

"male"

$Homeworld
[1] "Tatooine" "Naboo"
[7] "Naboo"
"Naboo"
$Born
[1] "41.9BBY" "46BBY"
[8] "33BBY"
$Jedi
[1] "yes" "no"

"Leia"

"female"

"male"

"Obi-Wan"

"Han"

"male"

"male"

"Tatooine" "Alderaan" "Stewjon"

"19BBY"

"yes" "no"

gastonsanchez.com

"Luke"

"19BBY"

"yes" "no"

"57BBY"

"no"

"Corellia"

"29BBY"

"82BBY"

"no"

Getting data from the web with R

CC BY-SA-NC 4.0

28 / 39

Reading JSON Data

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

29 / 39

JSON Data from the Web

How do we read JSON data from the Web?

We read JSON data in several ways. One way is to pass the url
directly to fromJSON(). Another way is by passing fromJSON() the
name of the file with the JSON content as a single string.

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

30 / 39

File: miserables.js
Well read the miserables dataset from:
http://mbostock.github.io/protovis/ex/miserables.js

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

31 / 39

Reading Issues

Houston we have a problem ...

The data is in a file that contains several javascript comments and
some other javascript notation.
Unfortunately, we cannot use any of the fromJSON() functions
directly on this type of content.
Instead, we need to read the content as text, get rid of the
comments, and change some characters before using fromJSON()

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

32 / 39

Reading miserables.js
# load RJSONIO and jsonlite
library(RJSONIO)
library(jsonlite)
# url with JSON content
miser = "http://mbostock.github.io/protovis/ex/miserables.js"
# import content as text (character vector)
miserables = readLines(miser)
# eliminate first 11 lines (containing comments)
miserables = miserables[-c(1:11)]

Now check the first and the last lines:

# first line
miserables[1]
## [1] "var miserables = {"
# last line
miserables[length(miserables)]
## [1] "};"

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

33 / 39

Preparing JSON content

We need to modify the first and last lines so they dont contain
non-JSON javascript notation
# open curly bracket in first line
miserables[1] = "{"
# closing curly bracket in last line
miserables[length(miserables)] = "}"

Now we must concatenate all the content into a single string:

# JSON content in one single string
miserables_str = paste(miserables, collapse = "")

Once we have the JSON content in the proper shape, we can parse
it with fromJSON().

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

34 / 39

Parsing JSON content

fromJSON() from package "RJSONIO":
# fromJSON() in package RJSONIO
mis1 = RJSONIO::fromJSON(miserables_str)

# class of each element

lapply(mi1, class)

# class
class(mis1)

## Error:

## [1] "list"

# how many elements in each list component

lapply(mis1, length)

# how many elements

length(mis1)
## [1] 2

##
##
##
##
##

object mi1 not found

$ode
[1] 77
$ink
[1] 254

# names
names(mis1)
## [1] "ode" "ink"

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

35 / 39

Parsing JSON content

# take a peek at nodes
head(mis1[[1]], n = 3)

# take a peek at links

head(mis1[[2]], n = 3)

##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##

##
##
##
##
##
##
##
##
##
##
##

[[1]]
[[1]]$odeNam
[1] "Myriel"
[[1]]$rou
[1] 1

[[2]]
[[2]]$odeNam
[1] "Napoleon"

[[1]]
ourc arge
1
0

alu
1

[[2]]
ourc arge
2
0

alu
8

[[3]]
ourc arge
3
0

alu
10

[[2]]$rou
[1] 1

[[3]]
[[3]]$odeNam
[1] "Mlle. Baptistine"
[[3]]$rou
[1] 1

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

36 / 39

Parsing JSON content

fromJSON() from package "jsonlite":
# fromJSON() in package jsonlite
mis2 = jsonlite::fromJSON(miserables_str)
# class
class(mis2)
## [1] "list"
# how many elements
length(mis2)

# class of each element

lapply(mis2, class)
##
##
##
##
##

$ode
[1] "data.frame"
$ink
[1] "data.frame"

# dimensions of each element

lapply(mis2, dim)

## [1] 2
# names
names(mis2)

##
##
##
##
##

$ode
[1] 77

$ink
[1] 254

## [1] "ode" "ink"

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

37 / 39

Parsing JSON content

# take a peek at nodes

head(mis2[[1]], n = 5)

# take a peek at links

head(mis2[[2]], n = 5)

##
##
##
##
##
##

##
##
##
##
##
##

odeNam rou
1
Myriel
1
2
Napoleon
1
3 Mlle. Baptistine
1
4
Mme. Magloire
1
5
Countess de Lo
1

gastonsanchez.com

1
2
3
4
5

ourc arge alu

1
0
1
2
0
8
3
0 10
3
2
6
4
0
1

Getting data from the web with R

CC BY-SA-NC 4.0

38 / 39

Parsing Differences

"RJSONIO" -vs- "jsonlite"

The package "jsonlite" is a fork of "RJSONIO". However,
"jsonlite" implements a smarter mapping between JSON data
and R classes.
From the previous example, we saw that "jsonlite" returns a list
of data frames instead of the list of lists returned by "RJSONIO"

gastonsanchez.com

Getting data from the web with R

CC BY-SA-NC 4.0

39 / 39