Understanding JSON

data in Postgres
INTRODUCTION TO NOSQL

Jake Roach
Data Engineer
JSON and JSONB in Postgres
{ JSON
'guardian': 'mother',
'status': 'A',
Store data in JSON format
'educations': [4, 4], Key-values pairs, arrays, nested objects
'jobs': {
'P1': 'teacher',
'P2': 'at_home'
JSONB
}
}
Data is stored in binary format

CREATE TABLE students Similar to BSON in MongoDB

INTRODUCTION TO NOSQL
Why semi-structured data in Postgres?

INTRODUCTION TO NOSQL
Querying JSON data with Postgres
SELECT We'll also be able:
address,
famsize, Insert JSON-formatted records to a
... Postgres table
FROM students
[WHERE | GROUP BY | ORDER BY];
Tabular to JSON, JSON to tabular

Extract individual records from JSON

row_to_json , json_to_record

-> , ->> , #> , #>> operators

json_extract_path ,
json_extract_path_text

INTRODUCTION TO NOSQL
Executing queries with sqlalchemy and pandas
import sqlalchemy
import pandas as pd

# Create a connection
db_engine = sqlalchemy.create_engine(
"postgresql+psycopg2://<user>:<password>@<host>:5432/<database>"
)

# Write a query
query = "SELECT * FROM table_name;"

# Execute the query, show results

INTRODUCTION TO NOSQL
Let's practice!
INTRODUCTION TO NOSQL
 Storing JSON data in
Postgres
INTRODUCTION TO NOSQL

Jake Roach
Data Engineer
INSERT INTO and COPY JSON records to Postgres
INSERT INTO students (school, age, address, parent_meta) VALUES (
'GP',
18,
'U',
'{\"guardian\": \"mother\", ... \"P2\": \"at_home\"}}'
);

Populate a table with contents of a file using COPY ... FROM

COPY students FROM 'students.csv' DELIMITER ',' CSV, HEADER;

INTRODUCTION TO NOSQL
Turning tabular data into JSON format
row_to_json function

Converts a row to JSON

Use with the row() function, and pass

SELECT
row_to_json(row(
school,
age,
address
))
FROM students;

INTRODUCTION TO NOSQL
Extracting keys from JSON
json_object_keys function

Extracts keys from a column of type JSON

SELECT
json_object_keys(parent_meta)
FROM students;

Pair with DISTINCT to find all unique keys

SELECT
DISTINCT json_object_keys(parent_meta)
FROM students;

INTRODUCTION TO NOSQL
Review
SELECT
row_to_json(row(
<column-1>,
<column-2>,
...
))
FROM <table-name>;

SELECT
DISTINCT json_object_keys(parent_meta)
FROM <table-name>;

INTRODUCTION TO NOSQL
Let's practice!
INTRODUCTION TO NOSQL
 Querying JSON data
using Postgres
INTRODUCTION TO NOSQL

Jake Roach
Data Engineer
Querying JSON data with Postgres
-> operator

Takes a key, returns field as JSON

->> operator

Takes a key, returns field as text

SELECT
parent_meta -> 'guardian' AS guardian
parent_meta ->> 'status' AS status
FROM student;

INTRODUCTION TO NOSQL
Querying nested JSON objects
To query nested JSON objects:

Use -> and ->> in tandem

First, return nested object

SELECT
parent_meta -> 'jobs' ->> 'P1' AS jobs_P1,
parent_meta -> 'jobs' ->> 'P2' AS jobs_P2
FROM student;

INTRODUCTION TO NOSQL
Querying JSON arrays
Accessing JSON array elements:

Pass an INT to -> , returns field as JSON

Pass an INT to ->> , returns field as text

SELECT
parent_meta -> 'educations' ->> 0
parent_meta -> 'educations' ->> 1
FROM student;

INTRODUCTION TO NOSQL
Finding the type of data store in JSON objects
json_typeof function

Returns the type of the outermost object

Use with ->

Useful for debugging, building queries

Typically not used with the ->> operator

SELECT
json_typeof(parent_meta -> 'jobs')
FROM students;

INTRODUCTION TO NOSQL
Review
SELECT
-- Top-level fields
<column-name> -> '<field-name>' AS <alias>,
<column-name> ->> '<field-name>' AS <alias>,

-- Nested fields
<column-name> -> '<parent-field-name>' ->> '<nested-field-name>' AS <alias>,

-- Arrays
<column-name> -> '<parent-field-name>' -> 0 AS <alias>,
<column-name> -> '<parent-field-name>' ->> 1 AS <alias>,

-- Type of
json_typeof(<column-name> -> <field-name>) AS <alias>
FROM <table-name>;

INTRODUCTION TO NOSQL
Let's practice!
INTRODUCTION TO NOSQL
 Advanced Postgres
JSON query
techniques
INTRODUCTION TO NOSQL

Jake Roach
Data Engineer
Querying nested JSON data
SELECT
parent_meta -> 'jobs' ->> 'P1' AS jobs_P1,
parent_meta -> 'jobs' ->> 'P2' AS jobs_P2
FROM student;

To make querying nested data easier:

#> and #>> operators

json_extract_path and json_extract_path_text functions

INTRODUCTION TO NOSQL
#> and #>> operators
#> operator

Called on column, takes a string array

Returns null if path does not exist

SELECT
parent_meta #> '{jobs}' AS jobs,
parent_meta #> '{jobs, P1}' AS jobs_P1,
parent_meta #> '{jobs, income}' AS income,
parent_meta #>> '{jobs, P2}' AS jobs_P2
FROM student;

INTRODUCTION TO NOSQL
json_extract_path and json_extract_path_text
json_extract_path

Provide column and arbitrary list of fields

Returns null if path does not exist

SELECT
json_extract_path(parent_meta, 'jobs') AS jobs,
json_extract_path(parent_meta, 'jobs', 'P1') AS jobs_P1,
json_extract_path(parent_meta, 'jobs', 'income') AS income,
json_extract_path_text(parent_meta, 'jobs', 'P2') AS jobs_P2,
FROM student;

INTRODUCTION TO NOSQL
Review
SELECT
<column-name> #> '{<field-name>}' AS <alias>,
<column-name> #> '{<field-name>, <field-name>}' AS <alias>,
<column-name> #>> '{<field-name>, <field-name>}' AS <alias>
FROM <table-name>;

SELECT
json_extract_path(<column-name>, '<field-name>') AS <alias>,
json_extract_path(<column-name>, '<field-name>', '<field-name>') AS <alias>,
json_extract_path_text(<column-name>, '<field-name>', '<field-name>') AS <alias>
FROM <table-name>;

INTRODUCTION TO NOSQL
Let's practice!
INTRODUCTION TO NOSQL