Load, Query And Process JSON

Data In Snowflake

Summary

1. Introduction - JSON & Snowflake

2. What is JSON (JavaScript Object Notation)?
3. Basic JSON Syntax
4. JSON values can be:
5. Load JSON Data Using Insert Queries
6. Query JSON Data In Snowflake
 Colon Notation To Access JSON Elements
 Alias of JSON Elements
7. Date & Timestamp Data Types in Snowflake JSON
8. Flatting JSON Data & Loading Into Snowflake Tables
 Creating Sequence Objects
 Creating Master Tables
 Insert into employee master table
 Insert into employee phone table
 Insert into employee phone table

How to Load, Query And Proces JSON Data Into Snowflake

Introduction - JSON & Snowflake

Snowflake can load semi-structured data into database tables, such as JSON.
Snowflake has excellent functionality for directly querying semi-structured
data as well as flattening it into a columnar structure once the data is loaded
from stage into a database table.

One of Snowflake’s key strengths is its ability to rapidly ingest both

structured and unstructured data. This blog will show you how to work with
JSON files in Snowflake, including how to directly query Snowflake JSON data
and copy it into a Snowflake table.

Once you finish this video, you will be able to answer following questions:

1. How to insert JSON data into snowflake using queries?

2. How to insert multiple JSON data in the snowflake using queries?
3. What data type should be used to store JSON data into snowflake
table?
4. How to parse JSON data in the snowflake and validate them?
5. How to parse JSON data and get individual elements?
6. How to parse nested JSON in snowflake and access all elements?
7. How Snowflake handle Date & Timestamp data type?
8. How to flatten JSON data & load elements in individual tables in
snowflake?
9. What is the role of colon & dot-notation and explicit casting?
10. How to join JSON data with other standard table in snowflake?
What is JSON (JavaScript Object Notation)?
JSON, which is pronounced “Jay-Sawn” or “Jason,” is a well-liked data
serialization format that is simple for both humans and machines to read,
write, and produce. The third edition of the JavaScript Programming
Language, Standard ECMA-262, is where the JSON file format had its start. It
was initially created as an alternative to XML and is mostly used to convey
data between a server and a web application.

Basic JSON Syntax

Rules:

1 - <KEY>:<VALUE> Data is in key/value pairs

2 - Data is separated by commas
3 - Objects are enclosed in curly braces ({})
4 - Arrays are enclosed in square brackets ([])

1 {
2 "employee": {
3 "name": "John",
4 "age": 30,
5 "height_in_ft": 5.11,
6 "married": true,
7 "has_kids": false,
8 "stock_options": null,
9 "phone": "+1 123-456-7890",
10 "date_of_birth":"2022-12-08",
11 "dob_timestamp":"2022-12-08T19:27:40.606-08:00",
12 "Address": {
13 "street": "4132 Grant Street",
14 "city": "Rockville",
15 "State": "Minnesota"
16 }
17 }
18 }

JSON values can be:

JSON Value Can Be From Following Data Type.

 A number
 A string
 A Boolean
 An array (The Phone Number in Above example is of an arrya type)
 A JSON object (The address in above example is an object type)
 Null (The stock option is an example of null value)
Load JSON Data Using Insert Queries
You can directly load JSON data into relational tables using Snowflake JSON.
Then, without having to perform any modifications, you may query this data
with SQL and join it to other structured data. This enables businesses to
streamline their data pipelines and quicken the rate at which this data is
made available for analysis.

SQL Script Used In Video

Following are the SQL script used in this video.

1 create or replace transient table json_tbl(

2 json_col variant
3 );
4
5 desc table json_tbl;
6
7 -- lets insert a record
8 insert into json_tbl(json_col) values ('{"firstName":"John",
9 "empid":1001}');
10
11 -- it is not possible to insert a row into variant using direct insert
12 statement
13
14 insert into json_tbl(json_col)
15 select '{"firstName":"John", "empid":1001}';
16
17 -- the above also does not work.. so we have to use
18 -- a function called parse_json
19
20 insert into json_tbl(json_col)
21 select parse_json('{"firstName":"John", "empid":1001}');
22
23 select * from json_tbl;
24
25 delete from json_tbl;
26 -- since it is a single value.. I don't need to specify
27 insert into json_tbl (json_col)
28 select parse_json(Column1) from values
29 ('{"firstName":"Name-1", "empid":1001}'),
30 ('{"firstName":"Name-2", "empid":1002}'),
31 ('{"firstName":"Name-3", "empid":1003}'),
32 ('{"firstName":"Name-4", "empid":1004}'),
33 ('{"firstName":"Name-5", "empid":1005}')
34 ;
35
36 -- if you observer, the order of field changed..
37
38 -- so it is not easy to insert the json data via insert statement.
39 delete from json_tbl;
 insert into json_tbl(json_col)
40 select parse_json(
41 '{
42 "employee": {
43 "name": "John",
44 "age": 30,
45 "height_in_ft": 5.11,
46 "married": true,
47 "has_kids": false,
48 "stock_options": null,
49 "phone": [
50 "+1 123-456-7890",
51 "+1 098-765-4321"
52 ],
53 "Address" {
54 "street": "4132 Grant Street",
55 "city": "Rockville",
56 "State": "Minnesota"
57 }
58 }
59 }');
60
61
62 -- how to insert data using internal & external stage are part of future
63 chapters
-- in this playlist.

Query JSON Data In Snowflake

Lets Query JSON Data

1
-- create an employee table with one variant column
2
3
create or replace transient table employee_tbl(
4
emp_json variant
5
);
6
7
-- let me insert one single record
8
insert into employee_tbl(emp_json)
9
select parse_json(
10
'{
11
"name": "John",
12
"age": 30,
13
"height_in_ft": 5.11,
14
"married": true,
15
"has_kids": false,
16
"stock_options": null
17
}');
18
19
-- lets run the select sql
20
select * from employee_tbl;
21
22
Colon Notation To Access JSON Elements

1
select
2
emp_json:name,
3
emp_json:"age",
4
emp_json:height_in_ft,
5
emp_json:married,
6
emp_json:has_kids,
7
emp_json:stock_options
8
from employee_tbl;
9
10

Alias of JSON Elements

1 select
2
3 emp_json:age::integer as age,
4 emp_json:name::string as "Name",
5 emp_json:height_in_ft::decimal as height,
6 emp_json:married as is_married,
7 emp_json:has_kids as has_kids,
8 emp_json:stock_options as stock_options
9 from employee_tbl;
10
11 -- add typeof() funciton to check how snowflake treat these name/value pair
12 select
13 typeof(emp_json:name) as name,
14 typeof(emp_json:age) as age,
15 typeof(emp_json:height_in_ft) as height,
16 typeof(emp_json:married) as is_married,
17 typeof(emp_json:has_kids) as has_kids,
18 typeof(emp_json:stock_options) as stock_options
19 from employee_tbl;
20
21
22 insert into employee_tbl(emp_json)
23 select parse_json(
24 '{
25 "employee": {
26 "name": "John",
27 "age": 30,
28 "height_in_ft": 5.11,
29 "married": true,
30 "has_kids": false,
31 "stock_options": null,
32 "phone": [
33 "+1 123-456-7890",
34 "+1 098-765-4321"
35 ],
36 "Address": {
37 "street": "3621 McDonald Avenue",
38 "city": "Orlando",
39 "State": "Florida"
40 }
41 }
42 }');
43
44 select * from employee_tbl;
45
46 select
47 emp_json:employee.name::string as name,
48 emp_json:employee.age as age,
49 emp_json:employee.height_in_ft as height,
50 emp_json:employee.married as is_married,
51 emp_json:employee.has_kids as has_kids,
52 emp_json:employee.stock_options as stock_options,
53 typeof(emp_json:employee.phone) as all_phone_type,
54 ARRAY_SIZE(emp_json:employee.phone) as how_many_phone,
55 emp_json:employee.phone[0] as work_phone,
56 emp_json:employee.phone[1] as office_phone,
57 typeof(emp_json:employee:Address) as address_type,
58 emp_json:employee:Address:street as street,
59 emp_json:employee.Address.city as city,
60 emp_json:employee.Address.State as state
61 from employee_tbl;
62
63
64 -- apply other function and mathematical operation without casting it.
65 select
66 emp_json:employee.age as age,
67 (emp_json:employee.height_in_ft) * (12*2.54) as height_in_cm,
68 typeof(emp_json:employee.Phone) as all_phone_type,
69 ARRAY_SIZE(emp_json:employee.Phone) as how_many_phone
70 from employee_tbl;
71

Date & Timestamp Data Types in Snowflake

JSON
1 insert into employee_tbl(emp_json)
2 select parse_json(
3 '{
4 "name": "John",
5 "age": 30,
6 "height_in_ft": 5.11,
7 "dob":"2022-12-11",
8 "dob_timestemp":"2022-12-11T00:19:06.043-08:00",
9 "married": true,
10 "has_kids": false,
11 "stock_options": null
12 }');
13
14
15 select
16 emp_json:dob::date,
17 emp_json:dob_timestemp::timestamp
18 from employee_tbl order by 1 desc;

Flatting JSON Data & Loading Into Snowflake

Tables
-- create the table

create transient table employee_tbl(

1
emp_json variant
2
);
3
4
--1st record
5
insert into employee_tbl(emp_json)
6
select parse_json(
7
'{
8
"employee": {"name": "John-1","age": 30,"height_in_ft": 5.11,"married":
9
true,"has_kids": false,
10
"stock_options": null,"email":"[email protected]","phone": ["+1 123-
11
456-7890","+1 098-765-4321"],
12
"Address": {"street": "3621 McDonald Avenue","city":
13
"Orlando","State": "Florida"}
14
}
15
}');
16
--2nd record
17
insert into employee_tbl(emp_json)
18
select parse_json(
19
'{
20
"employee": {"name": "John-2","age": 33,"height_in_ft": 5.09,"married":
21
false,"has_kids": false,
22
"stock_options": 10,"email":"[email protected]","phone": ["+1 222-456-
23
0987"],
24
"Address": {"street": "532 Locust View Drive","city": "San
25
Jose","State": "California"}
26
}
}');

Creating Sequence Objects

1 -- create sequencer
2 create or replace sequence emp_seq
3 start 1
4 increment 1
5 comment = 'employee sequence';
6
7 create or replace sequence phone_seq
8 start 1
9 increment 1
10 comment = 'phone sequence';
11
12 create or replace sequence address_seq
13 start 1
14 increment 1
15 comment = 'address sequence';
16

Creating Master Tables

1 -- employee master table

2 create or replace table employee_master(
3 emp_pk integer default emp_seq.nextval,
4 name string,
5 age number(3),
6 height_in_cm decimal(6,3),
7 is_married boolean,
8 has_kids boolean,
9 stock_options integer,
10 email varchar(100)
11 );
12
13 -- child table holding all the phones
14 create or replace table emp_phones(
15 phone_pk integer default phone_seq.nextval,
16 emp_fk number,
17 phone_type varchar(20),
18 phone_number varchar(30)
19 );
20
21 -- child table holding all the phones
22 create or replace table emp_address(
23 address_pk integer default address_seq.nextval,
24 emp_fk number,
25 street_address varchar(200),
26 city varchar(50),
27 state varchar(50)
28 );
29

Insert into employee master table

insert into employee_master (name, age,

1
height_in_cm,is_married,has_kids,stock_options,email)
2
select
3
emp_json:employee.name::string as name,
4
emp_json:employee.age as age,
5
(emp_json:employee.height_in_ft)*(12*2.54) as height_in_cm,
6
emp_json:employee.married as is_married,
7
emp_json:employee.has_kids as has_kids,
8
emp_json:employee.stock_options as stock_options,
9
emp_json:employee.email::string as email
10
from employee_tbl;
11
12
select * from employee_master;
13

Insert into employee phone table

1
insert into emp_phones (emp_fk,phone_type,phone_number)
2
select
3
b.emp_pk,
4
'home_phone' as home_phone,
5
a.emp_json:employee.phone[0]::string as home_phone
6
from
7
employee_tbl a
8
join
9
employee_master b
10
on
11
a.emp_json:employee.email = b.email
12
union all
13
select
14
b.emp_pk,
15
'work_phone' as work_phone,
16
a.emp_json:employee.phone[1]::string as work_phone
17
from
18
employee_tbl a
19
join
20
employee_master b
21
on
22
a.emp_json:employee.email = b.email;
23
24
select * from emp_phones;
25
26

Insert into employee phone table

1 insert into emp_address (emp_fk,street_address,city,state)

2 select
3 b.emp_pk,
4 a.emp_json:employee.Address.street::string as street,
5 a.emp_json:employee.Address.city::string as city,
6 a.emp_json:employee.Address.State::string as state
7 from
8 employee_tbl a
9 join
10 employee_master b
11 on
12 a.emp_json:employee.email = b.email;
13
14 select * from emp_address;
15
16 select e.*, a.*
17 from employee_master e join emp_address a on e.emp_pk = a.emp_fk;
18