Processing JSON with SQL

Speaker: Paul Tuohy

23 Simplifying IBM i Application

20
ch

Management with X-Analysis

ar
M

Speaker: Ray Everhart

n
ar
Le

Paul’s handout at Sponsored by

d
an
h
nc
Lu

[email protected]

See more Summit Lunch & Learn webinars at

SystemiDeveloper.com/LunchLearn
© ComCon & System i Developer, LLC 2005-2023 Page 1 of 30
 23
20
ch

Processing JSON with SQL

ar
M
n
ar
Le
d
an
h
nc
Lu

Paul Tuohy
ComCon Phone: +353 1 282 6230
e-Mail: [email protected]
System i Developer Web: www.systemideveloper.com
5, Oakton Court, www.ComConAdvisor.com
Ballybrack
Co. Dublin
Ireland

© ComCon & System i Developer, LLC 2005-2023 Page 2 of 30

ComCon
 Paul Tuohy ComCon

Paul Tuohy, author of "Re-engineering RPG Legacy Applications" and "The

Programmer's Guide to iSeries Navigator", is one of the most prominent consultants and
trainer/educators for application modernization and development technologies on the
IBM Midrange. He currently holds positions as CEO of ComCon, a consultancy firm
based in Dublin, Ireland, and founding partner of System i Developer, the consortium of
23
top educators who produce the acclaimed RPG & DB2 Summit conference. Previously,
20
he worked as IT Manager for Kodak Ireland Ltd. and Technical Director of Precision
ch

Software Ltd.
ar
M

In addition to hosting and speaking at the RPG & DB2 Summit, Paul is an award-winning
n

speaker at COMMON, COMMON Europe Congress and other conferences throughout

ar
Le

the world. His articles frequently appear in iProDeveloper, The Four Hundred Guru, RPG
Developer and other leading publications. Paul also hosts the popular iTalk with Tuohy
d
an

podcast interviews.
h
nc
Lu

This presentation may contain small code examples that are furnished as simple
examples to provide an illustration. These examples have not been thoroughly tested
under all conditions. We therefore, cannot guarantee or imply reliability, serviceability,
or function of these programs.
All code examples contained herein are provided to you "as is". THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE ARE EXPRESSLY DISCLAIMED.
© ComCon & System i Developer, LLC 2005-2023 Page 3 of 30
 Agenda ComCon

Sample JSON
Consuming JSON
The JSON_TABLE Table Function
JSON_TABLE Examples

23
Generate Column List
20
Get an IFS File with SQL
ch

JSON Processing with Embedded SQL

ar

Generating JSON
M
n

The JSON_OBJECT and JSON_ARRAY Functions

ar

JSON_OBJECT and JSON_ARRAY Examples

Le

Generate Column List

d
an

Get an IFS File with SQL

h

JSON Processing with Embedded SQL

nc
Lu

© ComCon & System i Developer, LLC 2005-2023 Page 4 of 30

 Sample JSON - Simple ComCon

23
20
{ "order_id":10248,
ch

"customer_id":"VINET",
ar

"employee_id":5,
"order_date":"2021-07-04",
M

"required_date":"2021-08-01",
n

"order_details":[
ar

{ "product_id":11,
Le

"unit_price":14.0000,
d

"quantity":12,
an

"discount":0
h

},
nc

{ "product_id":42,
Lu

"unit_price":9.8000,
"quantity":10,
"discount":0
},
{ "product_id":72,
"unit_price":34.8000,
"quantity":5,
"discount":0
}
]
}
© ComCon & System i Developer, LLC 2005-2023 Page 5 of 30
 Sample JSON - Slightly More Complex ComCon

{ "agent_Id": "AMAZON",
"batch_Id": "AMZ0001",
"order_count": 2,

23
"orders": [
20
{ "order_id": 10248,
"customer_id": "VINET",
ch

"employee_id": 5,
ar

"order_date": "2021-07-04",
M

"required_date": "2021-08-01",
"order_details": [
n
ar

{"product_id": 11, "unit_price": 14.0000, "quantity": 12, "discount": 0},

Le

{"product_id": 42, "unit_price": 9.8000, "quantity": 10, "discount": 0},

{"product_id": 72, "unit_price": 34.8000, "quantity": 5, "discount": 0}
d

]
an

},
h

{ "order_id": 10249,
nc

"customer_id": "TOMSP",
Lu

"employee_id": 6,
"order_date": "2021-07-05",
"required_date": "2021-08-16",
"order_details": [
{"product_id": 14, "unit_price": 18.6000, "quantity": 9, "discount": 0},
{"product_id": 51, "unit_price": 42.4000, "quantity": 40, "discount": 0}
]
}
]
}
© ComCon & System i Developer, LLC 2005-2023 Page 6 of 30
 23
20
Consume JSON with SQL
ch
ar
M
n
ar
Le
d
an
h
nc
Lu

© ComCon & System i Developer, LLC 2005-2023 Page 7 of 30

 The JSON_TABLE Table Function ComCon

Returns a result table from the evaluation of SQL/JSON path expressions

Each item in the result sequence of the row SQL/JSON path expression represents one or many
rows in the result table

23
20
ch
ar
M
n
ar
Le
d
an
h
nc
Lu

© ComCon & System i Developer, LLC 2005-2023 Page 8 of 30

 JSON_TABLE ComCon

JSON_TABLE function consists of three parts.

The JSON object to be deconstructed
A path expression that generates zero or more rows from the JSON object
The definition of the result columns to be returned. This includes

23
- The column name
20
- The result data type
- A path expression to use to locate the column information
ch
ar

JSON_TABLE is a table function

M

Used in place of a table name in a SELECT statement

n
ar
Le
d
an
h
nc
Lu

JSON_TABLE(json_object, path_expression, result_columns)

© ComCon & System i Developer, LLC 2005-2023 Page 9 of 30
 JSON_TABLE With Simple Sample ComCon

23
20
ch
ar
M
n
ar

select *
Le

from json_table(singleorder, '$'

COLUMNS (
d
an

order_id DECIMAL(11, 0) PATH '$.order_id',

customer_id CHAR(5) PATH '$.customer_id',
h

employee_id DECIMAL(11, 0) PATH '$.employee_id',

nc

order_date DATE PATH '$.order_date',

Lu

required_date DATE PATH '$.required_date',

NESTED PATH '$.order_details'
COLUMNS (
product_id DECIMAL(11, 0) PATH '$.product_id',
unit_price DECIMAL(19) PATH '$.unit_price',
quantity DECIMAL(7) PATH '$.quantity',
discount DECIMAL(3) PATH '$.discount'
)
)
) AS myJSON ;
© ComCon & System i Developer, LLC 2005-2023 Page 10 of 30
 JSON_TABLE With More Complex Sample ComCon

23
20
ch
ar
M

select * from json_table(batchorder, '$'

n

COLUMNS ( agent_Id char(6) PATH '$.agent_Id',

ar

batch_Id char(7) PATH '$.batch_Id',

Le

order_count integer PATH '$.order_count',

d

NESTED PATH '$.orders'

an

COLUMNS (
order_id DECIMAL(11, 0) PATH '$.order_id',
h
nc

customer_id CHAR(5) PATH '$.customer_id',

Lu

employee_id DECIMAL(11, 0) PATH '$.employee_id',

order_date DATE PATH '$.order_date',
required_date DATE PATH '$.required_date',
NESTED PATH '$.order_details'
COLUMNS ( product_id DECIMAL(11, 0) PATH '$.product_id',
unit_price DECIMAL(19, 4) PATH '$.unit_price',
quantity DECIMAL(7, 0) PATH '$.quantity',
discount DECIMAL(3, 1) PATH '$.discount')
)
)
) AS myJSON ;
© ComCon & System i Developer, LLC 2005-2023 Page 11 of 30
 Generate Column List ComCon

Generating a column list from the system catalog

23
20
ch
ar
M

-- Get the longest name length - this example returns 16

n

select max(length(column_name))
ar

from qsys2.syscolumns
Le

where (table_schema, table_name) = ('JONSAMPLE', 'ORDERS');

d
an

select rpad(lower(column_name), 17) concat

rpad( case when data_type in ('DATE', 'TIME', 'TIMESTAMP') then data_type
h

when data_type in ('NUMERIC', 'DECIMAL')

nc

then 'DECIMAL(' concat length

Lu

concat ', '

concat numeric_scale
concat ')'
else data_type concat '(' concat length concat ')'
end
, 17)
concat ' PATH ''$.' concat lower(column_name) concat ''','
from qsys2.syscolumns
where (table_schema, table_name) = ('JONSAMPLE', 'ORDERS')
order by ordinal_position;
© ComCon & System i Developer, LLC 2005-2023 Page 12 of 30
 Getting an IFS File in SQL ComCon

Use the GET_CLOB_FROM_FILE() function

When using an SQL Client (like Run SQL Scripts)
Global variables are your friend
Or use the function directly in the SELECT statement

23
- Less legible
20
ch
ar
M
n
ar
Le
d
an
h
nc
Lu

create variable singleorder varchar(32000);

create variable batchorder varchar(32000);

set singleorder = qsys2.GET_CLOB_FROM_FILE('/home/paris/jsonstuff/singleorder.json');

set batchorder = qsys2.GET_CLOB_FROM_FILE('/home/paris/jsonstuff/batchorder.json');
© ComCon & System i Developer, LLC 2005-2023 Page 13 of 30
 Getting an IFS File with Embedded SQL ComCon

23
20
dcl-s gv_ifs_File SQLType(CLOB_FILE) ccsid(*utf8);
ch
ar
M
n
ar
Le

dcl-proc du_get_ifsFile export;

dcl-pi *n varChar(32000) ccsid(*utf8);
d

fileName varChar(250) const;

an

end-Pi;
h
nc

dcl-s fileContent varChar(32000) ccsid(*utf8);

Lu

gv_ifs_File_FO = SQFRD;
gv_ifs_File_NAME = fileName;
gv_ifs_File_NL = %len(fileName);

exec SQL
values :gv_ifs_File into :fileContent;

return fileContent;
end-Proc;
© ComCon & System i Developer, LLC 2005-2023 Page 14 of 30
 Writing an IFS File with Embedded SQL ComCon

23
20
dcl-s gv_ifs_File SQLType(CLOB_FILE) ccsid(*utf8);
ch
ar
M
n
ar
Le

dcl-proc du_write_ifsFile export;

d

dcl-pi *n varChar(32000) ccsid(*utf8);

an

fileName varChar(250) const;

fileContent varChar(32000) ccsid(*utf8) const;
h
nc

end-Pi;
Lu

gv_ifs_File_FO = SQFOVR;
gv_ifs_File_NAME = fileName;
gv_ifs_File_NL = %len(fileName);

exec SQL
values :fileContent into :gv_ifs_File;

return fileContent;
end-Proc;

© ComCon & System i Developer, LLC 2005-2023 Page 15 of 30

 With MERGE ComCon

Most powerful when used to directly manipulate database from JSON

Example updates or inserts rows in the ORDER_DETAILS table

23
20
ch

merge into order_details as OLD

ar

using (select *
M

from json_table(singleorder, '$'

n

COLUMNS (
ar

order_id DECIMAL(11, 0) PATH '$.order_id',

Le

NESTED PATH '$.order_details'

d

COLUMNS (
an

product_id DECIMAL(11, 0) PATH '$.product_id',

unit_price DECIMAL(19) PATH '$.unit_price',
h
nc

quantity DECIMAL(7) PATH '$.quantity',

discount DECIMAL(3) PATH '$.discount'
Lu

)
)
) AS myJSON) as NEW
on (old.order_id, old.product_id) = (new.order_id, new.product_id)
when NOT MATCHED then
insert (order_id, product_id, unit_price, quantity, discount)
values (new.order_id, new.product_id, new.unit_price, new.quantity, new.discount)
when MATCHED then
update set (unit_price, quantity, discount)
= (new.unit_price, new.quantity, new.discount);
© ComCon & System i Developer, LLC 2005-2023 Page 16 of 30
 In RPG - Declare Host Variable and Get JSON ComCon

dcl-ds order_T extName('JONSAMPLE/ORDERS') alias template qualfied

end-ds;
dcl-ds order_details_T extName('JONSAMPLE/ORDDETAIL') alias template qualfied
end-ds;

23
20
dcl-ds order qualified;
order_id like(order_T.order_id);
ch

customer_id like(order_T.customer_id);
ar

employee_id like(order_T.employee_id);
M

order_date like(order_T.order_date);
n

required_date like(order_T.required_date);
ar

num_order_details int(5);
Le

end-ds;
d
an

dcl-ds order_details dim(99) qualified;

product_id like(order_details_T.product_id);
h

unit_price like(order_details_T.unit_price);
nc

quantity like(order_details_T.quantity);
Lu

discount like(order_details_T.discount);
end-ds;

dcl-s myJSON varChar(32000) ccsid(*utf8);

myJSON = du_get_ifsFile('/home/paris/jsonstuff/singleorder.json');
© ComCon & System i Developer, LLC 2005-2023 Page 17 of 30
 Get Order and Order Details Seperately ComCon

exec SQL
select order_id, customer_id, employee_id, order_date, required_date, 0
into :order
from json_table(:myJSON, '$'
COLUMNS ( order_id DECIMAL(11, 0) PATH '$.order_id',

23
20 customer_id CHAR(5) PATH '$.customer_id',
employee_id DECIMAL(11, 0) PATH '$.employee_id',
order_date DATE PATH '$.order_date',
ch

required_date DATE PATH '$.required_date'

ar

)) AS myJSON ;
M

exec SQL
declare get_details scroll cursor for
n
ar

select product_id, unit_price, quantity, discount

Le

from json_table(:myJSON, '$.order_details'

COLUMNS ( product_id DECIMAL(11, 0) PATH '$.product_id',
d

unit_price DECIMAL(19, 4) PATH '$.unit_price',

an

quantity DECIMAL(7, 0) PATH '$.quantity',

h

discount DECIMAL(3, 1) PATH '$.discount'

nc

)) AS myJSON ;
Lu

exec SQL
open get_details;

exec SQL
fetch first from get_details for 99 rows into :order_details;

order.num_order_details = SQLERRD(3);

exec SQL
close get_details;
© ComCon & System i Developer, LLC 2005-2023 Page 18 of 30
 Processing JSON with SQL
Speaker: Paul Tuohy

23 Simplifying IBM i Application

20
ch

Management with X-Analysis

ar
M

Speaker: Ray Everhart

n
ar
Le

Paul’s handout at Sponsored by

d
an
h
nc
Lu

[email protected]

See more Summit Lunch & Learn webinars at

SystemiDeveloper.com/LunchLearn
© ComCon & System i Developer, LLC 2005-2023 Page 19 of 30
 23
20
Generate JSON with SQL
ch
ar
M
n
ar
Le
d
an
h
nc
Lu

© ComCon & System i Developer, LLC 2005-2023 Page 20 of 30

 JSON Functions ComCon

Main functions (scalar)

JSON_OBJECT
JSON_ARRAY

Can also make use of (scalar)

23
JSON_QUERY
20
ch

JSON_TO_BSON
ar

JSON_VALUE
M
n
ar

Aggregate Functions
Le

JSON_OBJECTAGG
d

JSON_ARRAYAGG
an
h
nc
Lu

© ComCon & System i Developer, LLC 2005-2023 Page 21 of 30

 The JSON_OBJECT Scalar Function ComCon

Generates a JSON object using the specified key:value pairs

If no key:value pairs are provided, an empty object is returned

23
20
ch
ar
M
n
ar
Le
d
an
h
nc
Lu

© ComCon & System i Developer, LLC 2005-2023 Page 22 of 30

 The JSON_ARRAY Scalar Function ComCon

Generates a JSON array

By either explicitly listing the array elements or by using a query
If no JSON-expression is provided, the fullselect returns no values
If all values are null and ABSENT ON NULL is specified, an empty array is returned

23
20
ch
ar
M
n
ar
Le
d
an
h
nc
Lu

© ComCon & System i Developer, LLC 2005-2023 Page 23 of 30

 Generating the Simple Sample ComCon

23
20
ch
ar
M

select json_object(
n

'order_id' : order_id
ar

,'customer_id' : customer_id
Le

,'employee_id' : employee_id
d

,'order_date' : order_date
an

,'required_date' : required_date
,'order_details' :
h
nc

json_array( (select json_object(

'product_id' : product_id
Lu

,'unit_price' : unit_price
,'quantity' : quantity
,'discount' : discount
)
from order_details
where order_id = 10248 )
format json)
)
from orders
where order_id = 10248;
 Generating the More Complex Sample ComCon

values json_object(

23
'agent_Id': 'AMAZON',
20
'batch_Id': 'AMZ0001',
'order_count': (select count(*) from orders where order_id in (10248, 10249)),
ch

'orders':
ar

(json_array( (select json_object(

M

'order_id' : order_id
n

,'customer_id' : customer_id
ar

,'employee_id' : employee_id
Le

,'order_date' : order_date
,'required_date' : required_date
d
an

,'order_details' :
json_array( (select json_object(
h

'product_id' : product_id
nc

,'unit_price' : unit_price
Lu

,'quantity' : quantity
,'discount' : discount
)
from order_details
where order_id = 10248 )
format json)
)
from orders
where order_id in (10248, 10249)) format json))
);
 Things to Watch Out For in the SQL ComCon

FORMAT JSON
Required when nesting functions

ABSENT ON NULL or NULL ON NULL

23
NULL ON NULL is the default
20
ch

WITHOUT UNIQUE KEYS or WITH UNIQUE KEYS

ar
M

WITHOUT UNIQUE KEYS is the default

n
ar
Le
d
an
h
nc
Lu
 Things to Watch Out For in RPG Programs ComCon

The host variable (to receive the JSON) should be defined as UTF8
The JSON scalar functions return a UTF8 value
If the host variable is greater than 32K, it must be defined as a CLOB
This is an SQL limitation, not an RPG limitation

23
20
ch

dcl-s gen_JSON SQLType(CLOB: 10000000) CCSID(*utf8);

ar
M

// Results in
n
ar

DCL-DS GEN_JSON;
Le

GEN_JSON_LEN UNS(10);
GEN_JSON_DATA CHAR(10000000) CCSID(1208);
d
an

END-DS GEN_JSON;
h
nc
Lu

dcl-s gen_JSON SQLType(CLOB: 10000000) CCSID(*utf8);

dcl-s gen_JSON_var varchar(10000000) CCSID(*utf8) based(gen_JSON_var_p);

dcl-s gen_JSON_var_p pointer inz(%addr(gen_JSON));

// Above definition is more efficent than

// gen_JSON_var = '';
// if (gen_JSON_len > 0);
// gen_JSON_var = %subst(gen_JSON_data: 1: gen_JSON_len);
// endIf;
 Getting an IFS File with Embedded SQL ComCon

23
20
dcl-s gv_ifs_File SQLType(CLOB_FILE) ccsid(*utf8);
ch
ar
M
n
ar
Le

dcl-proc du_get_ifsFile export;

dcl-pi *n varChar(32000) ccsid(*utf8);
d

fileName varChar(250) const;

an

end-Pi;
h
nc

dcl-s fileContent varChar(32000) ccsid(*utf8);

Lu

gv_ifs_File_FO = SQFRD;
gv_ifs_File_NAME = fileName;
gv_ifs_File_NL = %len(fileName);

exec SQL
values :gv_ifs_File into :fileContent;

return fileContent;
end-Proc;
 Writing an IFS File with Embedded SQL ComCon

23
20
dcl-s gv_ifs_File SQLType(CLOB_FILE) ccsid(*utf8);
ch
ar
M
n
ar
Le

dcl-proc du_write_ifsFile export;

d

dcl-pi *n varChar(32000) ccsid(*utf8);

an

fileName varChar(250) const;

fileContent varChar(32000) ccsid(*utf8) const;
h
nc

end-Pi;
Lu

gv_ifs_File_FO = SQFOVR;
gv_ifs_File_NAME = fileName;
gv_ifs_File_NL = %len(fileName);

exec SQL
values :fileContent into :gv_ifs_File;

return fileContent;
end-Proc;
 Processing JSON with SQL
Speaker: Paul Tuohy

23 Simplifying IBM i Application

20
ch

Management with X-Analysis

ar
M

Speaker: Ray Everhart

n
ar
Le

Paul’s handout at Sponsored by

d
an
h
nc
Lu

[email protected]

See more Summit Lunch & Learn webinars at

SystemiDeveloper.com/LunchLearn