SQL/JSON: functions

Attached patches implementing all SQL/JSON functions excluding JSON_TABLE:

JSON_OBJECT()
JSON_OBJECTAGG()
JSON_ARRAY()
JSON_ARRAYAGG()

JSON_EXISTS()
JSON_VALUE()
JSON_QUERY()

IS JSON predicate

This patchset depends on 8th version of jsonpath patchset that was
posted in
https://www.postgresql.org/message-id/48f13b75-0be7-c356-ff26-1db743add56e%40postgrespro.ru

--
Nikita Glukhov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Attached 9th version of SQL/JSON patches rebased onto the latest master.

Displayed column name for SQL/JSON functions was fixed.

Documentation drafts written by Oleg Bartunov:
https://github.com/obartunov/sqljsondoc

--
Nikita Glukhov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Attached 10th version of SQL/JSON patches rebased onto the latest master.

--
Nikita Glukhov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Attached 11th version of SQL/JSON patches rebased onto the latest master.

Fixed uninitialized FORMAT JSON location in gram.y and column names for
JSON constructors.

--
Nikita Glukhov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Attached 12th version of SQL/JSON patches rebased onto the latest master.

--
Nikita Glukhov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

On 7 March 2018 at 14:34, Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru> wrote:

> Attached 12th version of SQL/JSON patches rebased onto the latest master.

Please write some docs or notes to go with this.

If you drop a big pile of code with no explanation it will just be ignored.

I think many people want SQL/JSON, but the purpose of a patch
submission is to allow a committer to review and commit without
needing to edit anything. It shouldn't be like assembling flat pack
furniture while wearing a blindfold.

--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Tue, Mar 13, 2018 at 2:04 PM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> On 7 March 2018 at 14:34, Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru> wrote:
>
>> Attached 12th version of SQL/JSON patches rebased onto the latest master.
>
> Please write some docs or notes to go with this.
>
> If you drop a big pile of code with no explanation it will just be ignored.
>
> I think many people want SQL/JSON, but the purpose of a patch
> submission is to allow a committer to review and commit without
> needing to edit anything. It shouldn't be like assembling flat pack
> furniture while wearing a blindfold.

The docs are here
https://github.com/obartunov/sqljsondoc/blob/master/README.jsonpath.md

It's not easy to write docs for SQL/JSON in xml, so I decided to write in more
friendly way. We'll have time to convert it to postgres format.

>
> --
> Simon Riggs http://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Tue, Mar 13, 2018 at 04:08:01PM +0300, Oleg Bartunov wrote:
> The docs are here
> https://github.com/obartunov/sqljsondoc/blob/master/README.jsonpath.md
>
> It's not easy to write docs for SQL/JSON in xml, so I decided to write in more
> friendly way. We'll have time to convert it to postgres format.

If you aim at getting a feature committed first without its
documentation, and getting the docs written after the feature freeze
using a dedicated open item or such, this is much acceptable in my
opinion and the CF is running short in time.
--
Michael

On 2018-03-14 07:54:35 +0900, Michael Paquier wrote:
> On Tue, Mar 13, 2018 at 04:08:01PM +0300, Oleg Bartunov wrote:
> > The docs are here
> > https://github.com/obartunov/sqljsondoc/blob/master/README.jsonpath.md
> >
> > It's not easy to write docs for SQL/JSON in xml, so I decided to write in more
> > friendly way. We'll have time to convert it to postgres format.
>
> If you aim at getting a feature committed first without its
> documentation, and getting the docs written after the feature freeze
> using a dedicated open item or such, this is much acceptable in my
> opinion and the CF is running short in time.

Given that this patch still uses PG_TRY/CATCH around as wide paths of
code as a whole ExecEvalExpr() invocation, basically has gotten no
review, I don't see this going anywhere for v11.

Greetings,

Andres Freund

On 14 Mar 2018 01:54, "Michael Paquier" <michael(at)paquier(dot)xyz> wrote:

On Tue, Mar 13, 2018 at 04:08:01PM +0300, Oleg Bartunov wrote:
> The docs are here
> https://github.com/obartunov/sqljsondoc/blob/master/README.jsonpath.md
>
> It's not easy to write docs for SQL/JSON in xml, so I decided to write in
more
> friendly way. We'll have time to convert it to postgres format.

If you aim at getting a feature committed first without its
documentation, and getting the docs written after the feature freeze
using a dedicated open item or such,

Exactly. SQL/JSON is rather complex thing and "converting" the standard to
the user level understanding is a separate challenge and I'd like to
continue to work on it. It's mostly written, we need to understand how to
organize it.

this

is much acceptable in my
opinion and the CF is running short in time.
--
Michael

On Wed, Mar 14, 2018 at 2:10 AM, Andres Freund <andres(at)anarazel(dot)de> wrote:

> On 2018-03-14 07:54:35 +0900, Michael Paquier wrote:
> > On Tue, Mar 13, 2018 at 04:08:01PM +0300, Oleg Bartunov wrote:
> > > The docs are here
> > > https://github.com/obartunov/sqljsondoc/blob/master/README.jsonpath.md
> > >
> > > It's not easy to write docs for SQL/JSON in xml, so I decided to write
> in more
> > > friendly way. We'll have time to convert it to postgres format.
> >
> > If you aim at getting a feature committed first without its
> > documentation, and getting the docs written after the feature freeze
> > using a dedicated open item or such, this is much acceptable in my
> > opinion and the CF is running short in time.
>
> Given that this patch still uses PG_TRY/CATCH around as wide paths of
> code as a whole ExecEvalExpr() invocation,

I agree that we should either use PG_TRY/CATCH over some small and safe
codepaths or surround PG_TRY/CATCH with subtransactions. PG_TRY/CATCH over
ExecEvalExpr() looks really unacceptable.

basically has gotten no
> review, I don't see this going anywhere for v11.
>

I wouldn't be co categorical at this point. Patchset is there for about
year.
Some parts of code received more of review while some parts receives less.
We can surround all dangerous PG_TRY/CATCH pairs with subtransactions,
tolerate performance penalty and leave further optimizations for future
releases.
In worst case, we can remove codepaths which use PG_TRY/CATCH and
leave only ERROR ON ERROR behavior of SQL/JSON.

------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

2018-03-14 15:11 GMT+01:00 Alexander Korotkov <a(dot)korotkov(at)postgrespro(dot)ru>:

> On Wed, Mar 14, 2018 at 2:10 AM, Andres Freund <andres(at)anarazel(dot)de> wrote:
>
>> On 2018-03-14 07:54:35 +0900, Michael Paquier wrote:
>> > On Tue, Mar 13, 2018 at 04:08:01PM +0300, Oleg Bartunov wrote:
>> > > The docs are here
>> > > https://github.com/obartunov/sqljsondoc/blob/master/README.j
>> sonpath.md
>> > >
>> > > It's not easy to write docs for SQL/JSON in xml, so I decided to
>> write in more
>> > > friendly way. We'll have time to convert it to postgres format.
>> >
>> > If you aim at getting a feature committed first without its
>> > documentation, and getting the docs written after the feature freeze
>> > using a dedicated open item or such, this is much acceptable in my
>> > opinion and the CF is running short in time.
>>
>> Given that this patch still uses PG_TRY/CATCH around as wide paths of
>> code as a whole ExecEvalExpr() invocation,
>
>
> I agree that we should either use PG_TRY/CATCH over some small and safe
> codepaths or surround PG_TRY/CATCH with subtransactions. PG_TRY/CATCH
> over
> ExecEvalExpr() looks really unacceptable.
>
> basically has gotten no
>> review, I don't see this going anywhere for v11.
>>
>
> I wouldn't be co categorical at this point. Patchset is there for about
> year.
> Some parts of code received more of review while some parts receives less.
> We can surround all dangerous PG_TRY/CATCH pairs with subtransactions,
> tolerate performance penalty and leave further optimizations for future
> releases.
> In worst case, we can remove codepaths which use PG_TRY/CATCH and
> leave only ERROR ON ERROR behavior of SQL/JSON.
>

I am thinking so using subtransactions on few places are acceptable.
PLpgSQL uses it years, and it is working good enough.

Regards

Pavel

>
>

> ------
> Alexander Korotkov
> Postgres Professional: http://www.postgrespro.com
> The Russian Postgres Company
>

On 14 Mar 2018 17:11, "Alexander Korotkov" <a(dot)korotkov(at)postgrespro(dot)ru>
wrote:

On Wed, Mar 14, 2018 at 2:10 AM, Andres Freund <andres(at)anarazel(dot)de> wrote:

> On 2018-03-14 07:54:35 +0900, Michael Paquier wrote:
> > On Tue, Mar 13, 2018 at 04:08:01PM +0300, Oleg Bartunov wrote:
> > > The docs are here
> > > https://github.com/obartunov/sqljsondoc/blob/master/README.jsonpath.md
> > >
> > > It's not easy to write docs for SQL/JSON in xml, so I decided to write
> in more
> > > friendly way. We'll have time to convert it to postgres format.
> >
> > If you aim at getting a feature committed first without its
> > documentation, and getting the docs written after the feature freeze
> > using a dedicated open item or such, this is much acceptable in my
> > opinion and the CF is running short in time.
>
> Given that this patch still uses PG_TRY/CATCH around as wide paths of
> code as a whole ExecEvalExpr() invocation,

I agree that we should either use PG_TRY/CATCH over some small and safe
codepaths or surround PG_TRY/CATCH with subtransactions. PG_TRY/CATCH over
ExecEvalExpr() looks really unacceptable.

basically has gotten no
> review, I don't see this going anywhere for v11.
>

I wouldn't be co categorical at this point. Patchset is there for about
year.
Some parts of code received more of review while some parts receives less.
We can surround all dangerous PG_TRY/CATCH pairs with subtransactions,
tolerate performance penalty and leave further optimizations for future
releases.

Agree it's not difficult.

In worst case, we can remove codepaths which use PG_TRY/CATCH and
leave only ERROR ON ERROR behavior of SQL/JSON.

No-no, json user will be really upset on this. Our goal is to be the first
relational database with strong standard compliance.

------
Alexander Korotkov

Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Attached 13th version of the patches:

* Subtransactions in PG_TRY/CATCH in ExecEvalJsonExpr() were made unconditional,
regardless of the volatility of expressions.

* PG_TRY/CATCH in ExecEvalExprPassingCaseValue() was removed along with the
entire function.

On 15.03.2018 11:08, Oleg Bartunov wrote:

> On 14 Mar 2018 17:11, "Alexander Korotkov" <a(dot)korotkov(at)postgrespro(dot)ru
> <mailto:a(dot)korotkov(at)postgrespro(dot)ru>> wrote:
>
> On Wed, Mar 14, 2018 at 2:10 AM, Andres Freund <andres(at)anarazel(dot)de
> <mailto:andres(at)anarazel(dot)de>> wrote:
>
> On 2018-03-14 07:54:35 +0900, Michael Paquier wrote:
> > On Tue, Mar 13, 2018 at 04:08:01PM +0300, Oleg Bartunov wrote:
> > > The docs are here
> > >
> https://github.com/obartunov/sqljsondoc/blob/master/README.jsonpath.md
> <https://github.com/obartunov/sqljsondoc/blob/master/README.jsonpath.md>
> > >
> > > It's not easy to write docs for SQL/JSON in xml, so I
> decided to write in more
> > > friendly way. We'll have time to convert it to postgres
> format.
> >
> > If you aim at getting a feature committed first without its
> > documentation, and getting the docs written after the
> feature freeze
> > using a dedicated open item or such, this is much acceptable
> in my
> > opinion and the CF is running short in time.
>
> Given that this patch still uses PG_TRY/CATCH around as wide
> paths of
> code as a whole ExecEvalExpr() invocation,
>
> I agree that we should either use PG_TRY/CATCH over some small and
> safe
> codepaths or surround PG_TRY/CATCH with subtransactions.
> PG_TRY/CATCH over
> ExecEvalExpr() looks really unacceptable.
>
> basically has gotten no
> review, I don't see this going anywhere for v11.
>
>
> I wouldn't be co categorical at this point. Patchset is there for
> about year.
> Some parts of code received more of review while some parts
> receives less.
> We can surround all dangerous PG_TRY/CATCH pairs with subtransactions,
> tolerate performance penalty and leave further optimizations for
> future releases.
>
>
> Agree it's not difficult.
>
> In worst case, we can remove codepaths which use PG_TRY/CATCH and
> leave only ERROR ON ERROR behavior of SQL/JSON.
>
>
> No-no, json user will be really upset on this. Our goal is to be the
> first relational database with strong standard compliance.

--
Nikita Glukhov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

On 15.03.2018 20:04, Nikita Glukhov wrote:
> Attached 13th version of the patches:
>
> * Subtransactions in PG_TRY/CATCH in ExecEvalJsonExpr() were made unconditional,
> regardless of the volatility of expressions.
>
> * PG_TRY/CATCH in ExecEvalExprPassingCaseValue() was removed along with the
> entire function.

Attached 15th version of the patches:
* disabled parallel execution of SQL/JSON query functions when internal
subtransactions are used (if ERROR ON ERROR is not specified)
* added experimental optimization of internal subtransactions (see below)

The new patch #14 is an experimental attempt to reduce overhead of
subtransaction start/commit which can result in 2x-slowdown in the simplest
cases. By the idea of Alexander Korotkov, subtransaction is not really
committed if it has not touched the database and its XID has not been assigned
(DB modification is not expected in type casts functions) and then can be reused
when the next subtransaction is started. So, all rows in JsonExpr can be
executed in the single cached subtransaction. This optimization really helps
to reduce overhead from 100% to 5-10%:

-- without subtransactions
=# EXPLAIN ANALYZE
SELECT JSON_VALUE('true'::jsonb, '$' RETURNING boolean ERROR ON ERROR)
FROM generate_series(1, 10000000) i;
...
Execution Time: 2785.410 ms

-- cached subtransactions
=# EXPLAIN ANALYZE
SELECT JSON_VALUE('true'::jsonb, '$' RETURNING boolean)
FROM generate_series(1, 10000000) i;
...
Execution Time: 2939.363 ms

-- ordinary subtransactions
=# EXPLAIN ANALYZE
SELECT JSON_VALUE('true'::jsonb, '$' RETURNING boolean)
FROM generate_series(1, 10000000) i;
...
Execution Time: 5417.268 ms

But, unfortunately, I don't believe that this patch is completely correct,
mainly because the behavior of subtransaction callbacks (and their expectations
about subtransaction's lifecycle too) seems unpredictable to me.

Even with this optimization, internal subtransactions still have one major
drawback -- they disallow parallel query execution, because background
workers do not support subtransactions now. Example:

=# CREATE TABLE test_parallel_json_value AS
SELECT i::text::jsonb AS js FROM generate_series(1, 5000000) i;
CREATE TABLE

=# EXPLAIN ANALYZE
SELECT sum(JSON_VALUE(js, '$' RETURNING numeric ERROR ON ERROR))
FROM test_parallel_json_value;
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------
Finalize Aggregate (cost=79723.15..79723.16 rows=1 width=32) (actual time=455.062..455.062 rows=1 loops=1)
-> Gather (cost=79722.93..79723.14 rows=2 width=32) (actual time=455.052..455.055 rows=3 loops=1)
Workers Planned: 2
Workers Launched: 2
-> Partial Aggregate (cost=78722.93..78722.94 rows=1 width=32) (actual time=446.000..446.000 rows=1 loops=3)
-> Parallel Seq Scan on t (cost=0.00..52681.30 rows=2083330 width=18) (actual time=0.023..104.779 rows=1666667 loops=3)
Planning Time: 0.044 ms
Execution Time: 456.460 ms
(8 rows)

=# EXPLAIN ANALYZE
SELECT sum(JSON_VALUE(js, '$' RETURNING numeric))
FROM test_parallel_json_value;
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------
Aggregate (cost=144347.82..144347.83 rows=1 width=32) (actual time=1381.938..1381.938 rows=1 loops=1)
-> Seq Scan on t (cost=0.00..81847.92 rows=4999992 width=18) (actual time=0.076..309.676 rows=5000000 loops=1)
Planning Time: 0.082 ms
Execution Time: 1384.133 ms
(4 rows)

--
Nikita Glukhov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

2018-06-28 2:18 GMT+02:00 Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru>:

> On 15.03.2018 20:04, Nikita Glukhov wrote:
>
>> Attached 13th version of the patches:
>>
>> * Subtransactions in PG_TRY/CATCH in ExecEvalJsonExpr() were made
>> unconditional,
>> regardless of the volatility of expressions.
>>
>> * PG_TRY/CATCH in ExecEvalExprPassingCaseValue() was removed along with
>> the
>> entire function.
>>
>
>
> Attached 15th version of the patches:
> * disabled parallel execution of SQL/JSON query functions when internal
> subtransactions are used (if ERROR ON ERROR is not specified)
> * added experimental optimization of internal subtransactions (see below)
>
>
> The new patch #14 is an experimental attempt to reduce overhead of
> subtransaction start/commit which can result in 2x-slowdown in the simplest
> cases. By the idea of Alexander Korotkov, subtransaction is not really
> committed if it has not touched the database and its XID has not been
> assigned
> (DB modification is not expected in type casts functions) and then can be
> reused
> when the next subtransaction is started. So, all rows in JsonExpr can be
> executed in the single cached subtransaction. This optimization really
> helps
> to reduce overhead from 100% to 5-10%:
>

I read a technical report for SQL/JSON. If I understand it well, then ON
ERROR clause is primary related to structural errors, not to all errors.

So your implementation is maybe too tolerant, what has this issue. There
was not any example, so this clause should to handle cast errors or any
other errors than JSON structural.

The playing with other implementation of subtransactions doesn't look like
safe way, more if it is not necessary

The other possible error are casts errors. We can introduce new exception
safe input functions. These functions can be interesting for fault tolerant
COPY for example.

Regards

Pavel

> -- without subtransactions
> =# EXPLAIN ANALYZE
> SELECT JSON_VALUE('true'::jsonb, '$' RETURNING boolean ERROR ON ERROR)
> FROM generate_series(1, 10000000) i;
> ...
> Execution Time: 2785.410 ms
>
> -- cached subtransactions
> =# EXPLAIN ANALYZE
> SELECT JSON_VALUE('true'::jsonb, '$' RETURNING boolean)
> FROM generate_series(1, 10000000) i;
> ...
> Execution Time: 2939.363 ms
>
> -- ordinary subtransactions
> =# EXPLAIN ANALYZE
> SELECT JSON_VALUE('true'::jsonb, '$' RETURNING boolean)
> FROM generate_series(1, 10000000) i;
> ...
> Execution Time: 5417.268 ms
>
>
> But, unfortunately, I don't believe that this patch is completely correct,
> mainly because the behavior of subtransaction callbacks (and their
> expectations
> about subtransaction's lifecycle too) seems unpredictable to me.
>
>
>
> Even with this optimization, internal subtransactions still have one major
> drawback -- they disallow parallel query execution, because background
> workers do not support subtransactions now. Example:
>
> =# CREATE TABLE test_parallel_json_value AS
> SELECT i::text::jsonb AS js FROM generate_series(1, 5000000) i;
> CREATE TABLE
>
> =# EXPLAIN ANALYZE
> SELECT sum(JSON_VALUE(js, '$' RETURNING numeric ERROR ON ERROR))
> FROM test_parallel_json_value;
> QUERY PLAN
> ------------------------------------------------------------
> ------------------------------------------------------------
> -----------------
> Finalize Aggregate (cost=79723.15..79723.16 rows=1 width=32) (actual
> time=455.062..455.062 rows=1 loops=1)
> -> Gather (cost=79722.93..79723.14 rows=2 width=32) (actual
> time=455.052..455.055 rows=3 loops=1)
> Workers Planned: 2
> Workers Launched: 2
> -> Partial Aggregate (cost=78722.93..78722.94 rows=1 width=32)
> (actual time=446.000..446.000 rows=1 loops=3)
> -> Parallel Seq Scan on t (cost=0.00..52681.30
> rows=2083330 width=18) (actual time=0.023..104.779 rows=1666667 loops=3)
> Planning Time: 0.044 ms
> Execution Time: 456.460 ms
> (8 rows)
>
> =# EXPLAIN ANALYZE
> SELECT sum(JSON_VALUE(js, '$' RETURNING numeric))
> FROM test_parallel_json_value;
> QUERY PLAN
> ------------------------------------------------------------
> --------------------------------------------------------
> Aggregate (cost=144347.82..144347.83 rows=1 width=32) (actual
> time=1381.938..1381.938 rows=1 loops=1)
> -> Seq Scan on t (cost=0.00..81847.92 rows=4999992 width=18) (actual
> time=0.076..309.676 rows=5000000 loops=1)
> Planning Time: 0.082 ms
> Execution Time: 1384.133 ms
> (4 rows)
>
>
>
> --
> Nikita Glukhov
> Postgres Professional: http://www.postgrespro.com
> The Russian Postgres Company
>
>

Attached 16th version of the patches:
* changed type of new SQL keyword STRING
(STRING is used as a function parameter name in Pl/Tcl tests)
* removed implicit coercion via I/O from JSON_VALUE (see below)

On 28.06.2018 07:25, Pavel Stehule wrote:
> 2018-06-28 2:18 GMT+02:00 Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru
> <mailto:n(dot)gluhov(at)postgrespro(dot)ru>>:
>
> On 15.03.2018 20:04, Nikita Glukhov wrote:
>
> Attached 13th version of the patches:
>
> * Subtransactions in PG_TRY/CATCH in ExecEvalJsonExpr() were
> made unconditional,
>   regardless of the volatility of expressions.
>
> * PG_TRY/CATCH in ExecEvalExprPassingCaseValue() was removed
> along with the
>   entire function.
>
>
>
> Attached 15th version of the patches:
>  * disabled parallel execution of SQL/JSON query functions when
> internal
>    subtransactions are used (if ERROR ON ERROR is not specified)
>  * added experimental optimization of internal subtransactions
> (see below)
>
>
> The new patch #14 is an experimental attempt to reduce overhead of
> subtransaction start/commit which can result in 2x-slowdown in the
> simplest
> cases.  By the idea of Alexander Korotkov, subtransaction is not
> really
> committed if it has not touched the database and its XID has not
> been assigned
> (DB modification is not expected in type casts functions) and then
> can be reused
> when the next subtransaction is started.  So, all rows in JsonExpr
> can be
> executed in the single cached subtransaction.  This optimization
> really helps
> to reduce overhead from 100% to 5-10%:
>
>
> I read a technical report for SQL/JSON. If I understand it well, then
> ON ERROR clause is primary related to structural errors, not to all
> errors.
>
> So your implementation is maybe too tolerant, what has this issue.
> There was not any example, so this clause should to handle cast errors
> or any other errors than JSON structural.
>
> The playing with other implementation of subtransactions doesn't look
> like safe way, more if it is not necessary
>
> The other possible error are casts errors. We can introduce new
> exception safe input functions. These functions can be interesting for
> fault tolerant COPY for example.
>
SQL/JSON standard requires handling of cast errors too.

9.40 Casting an SQL/JSON sequence to an SQL type (pages 724-725):

4) If TEMPST is successful completion, then:
b) If the length of SEQ is 1 (one), then let I be the SQL/JSON item in SEQ.
Case:
...
iii) Otherwise, let IDT be the data type of I.
Case:
1) If IDT cannot be cast to target type DT according to the Syntax Rules
of Subclause 6.13, "<cast specification>", then let TEMPST be data
exception — SQL/JSON item cannot be cast to target type.
2) Otherwise, let X be an SQL variable whose value is I. Let V be the
value of CAST (X AS DT). If an exception condition is raised by this
<cast specification>, then let TEMPST be that exception condition.
...
5) Case:
a) If TEMPST is successful completion, then let OUTST be successful
completion.
b) If ONERROR is ERROR, then let OUTST be TEMPST.
c) If ONERROR is NULL, then let V be the SQL null value and let OUTST be
successful completion.
d) If ONERROR immediately contains DEFAULT, then let VE be the
<value expression> immediately contained in ONERROR. Let V be the value of
CAST (VE AS DT)
Case:
i) If an exception condition is raised by this <cast specification>, then
let OUTST be that exception condition.
ii) Otherwise, let OUTST be successful completion.

In 4.b.iii.1 said that there should be an error if the desired cast does not exist.
In the previous versions of the patches there was implicit coercion via I/O here
instead of error, so I decided to fix it the last version (fix is combined with a
minor refactoring of ExecEvalJsonExpr()).

--
Nikita Glukhov
Postgres Professional:http://www.postgrespro.com
The Russian Postgres Company

2018-07-03 14:30 GMT+02:00 Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru>:

> Attached 16th version of the patches:
> * changed type of new SQL keyword STRING
> (STRING is used as a function parameter name in Pl/Tcl tests)
> * removed implicit coercion via I/O from JSON_VALUE (see below)
>
>
> On 28.06.2018 07:25, Pavel Stehule wrote:
>
> 2018-06-28 2:18 GMT+02:00 Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru>:
>
>> On 15.03.2018 20:04, Nikita Glukhov wrote:
>>
>>> Attached 13th version of the patches:
>>>
>>> * Subtransactions in PG_TRY/CATCH in ExecEvalJsonExpr() were made
>>> unconditional,
>>> regardless of the volatility of expressions.
>>>
>>> * PG_TRY/CATCH in ExecEvalExprPassingCaseValue() was removed along with
>>> the
>>> entire function.
>>>
>>
>>
>> Attached 15th version of the patches:
>> * disabled parallel execution of SQL/JSON query functions when internal
>> subtransactions are used (if ERROR ON ERROR is not specified)
>> * added experimental optimization of internal subtransactions (see below)
>>
>>
>> The new patch #14 is an experimental attempt to reduce overhead of
>> subtransaction start/commit which can result in 2x-slowdown in the
>> simplest
>> cases. By the idea of Alexander Korotkov, subtransaction is not really
>> committed if it has not touched the database and its XID has not been
>> assigned
>> (DB modification is not expected in type casts functions) and then can be
>> reused
>> when the next subtransaction is started. So, all rows in JsonExpr can be
>> executed in the single cached subtransaction. This optimization really
>> helps
>> to reduce overhead from 100% to 5-10%:
>>
>
> I read a technical report for SQL/JSON. If I understand it well, then ON
> ERROR clause is primary related to structural errors, not to all errors.
>
> So your implementation is maybe too tolerant, what has this issue. There
> was not any example, so this clause should to handle cast errors or any
> other errors than JSON structural.
>
> The playing with other implementation of subtransactions doesn't look like
> safe way, more if it is not necessary
>
> The other possible error are casts errors. We can introduce new exception
> safe input functions. These functions can be interesting for fault tolerant
> COPY for example.
>
> SQL/JSON standard requires handling of cast errors too.
>
> I didn't speak something else. But cast (and in this case it is from JSON
to some else) can be exception safe.

Regards

Pavel

>
> 9.40 Casting an SQL/JSON sequence to an SQL type (pages 724-725):
>
> 4) If TEMPST is successful completion, then:
> b) If the length of SEQ is 1 (one), then let I be the SQL/JSON item in SEQ.
> Case:
> ...
> iii) Otherwise, let IDT be the data type of I.
> Case:
> 1) If IDT cannot be cast to target type DT according to the Syntax Rules
> of Subclause 6.13, "<cast specification>", then let TEMPST be data
> exception — SQL/JSON item cannot be cast to target type.
> 2) Otherwise, let X be an SQL variable whose value is I. Let V be the
> value of CAST (X AS DT). If an exception condition is raised by this
> <cast specification>, then let TEMPST be that exception condition.
> ...
> 5) Case:
> a) If TEMPST is successful completion, then let OUTST be successful
> completion.
> b) If ONERROR is ERROR, then let OUTST be TEMPST.
> c) If ONERROR is NULL, then let V be the SQL null value and let OUTST be
> successful completion.
> d) If ONERROR immediately contains DEFAULT, then let VE be the
> <value expression> immediately contained in ONERROR. Let V be the value of
> CAST (VE AS DT)
> Case:
> i) If an exception condition is raised by this <cast specification>, then
> let OUTST be that exception condition.
> ii) Otherwise, let OUTST be successful completion.
>
>
> In 4.b.iii.1 said that there should be an error if the desired cast does not exist.
> In the previous versions of the patches there was implicit coercion via I/O here
> instead of error, so I decided to fix it the last version (fix is combined with a
> minor refactoring of ExecEvalJsonExpr()).
>
> --
> Nikita Glukhov
> Postgres Professional: http://www.postgrespro.com
> The Russian Postgres Company
>
>

> On Tue, Jul 3, 2018 at 2:57 PM Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com> wrote:
>
> 2018-07-03 14:30 GMT+02:00 Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru>:
>>
>> Attached 16th version of the patches:
>> * changed type of new SQL keyword STRING
>> (STRING is used as a function parameter name in Pl/Tcl tests)
>> * removed implicit coercion via I/O from JSON_VALUE (see below)

Unfortunately, the current version of patch 0010-add-invisible-coercion-form
doesn't not apply anymore without conflicts, could you please rebase it?

On 26.11.2018 15:57, Dmitry Dolgov wrote:

>> On Tue, Jul 3, 2018 at 2:57 PM Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com> wrote:
>>
>> 2018-07-03 14:30 GMT+02:00 Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru>:
>>> Attached 16th version of the patches:
>>> * changed type of new SQL keyword STRING
>>> (STRING is used as a function parameter name in Pl/Tcl tests)
>>> * removed implicit coercion via I/O from JSON_VALUE (see below)
> Unfortunately, the current version of patch 0010-add-invisible-coercion-form
> doesn't not apply anymore without conflicts, could you please rebase it?

Attached 20th version of the patches rebased onto the current master.

--
Nikita Glukhov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

> On Mon, Nov 26, 2018 at 5:11 PM Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru> wrote:
>
> On 26.11.2018 15:57, Dmitry Dolgov wrote:
>
> >> On Tue, Jul 3, 2018 at 2:57 PM Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com> wrote:
> >>
> >> 2018-07-03 14:30 GMT+02:00 Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru>:
> >>> Attached 16th version of the patches:
> >>> * changed type of new SQL keyword STRING
> >>> (STRING is used as a function parameter name in Pl/Tcl tests)
> >>> * removed implicit coercion via I/O from JSON_VALUE (see below)
> > Unfortunately, the current version of patch 0010-add-invisible-coercion-form
> > doesn't not apply anymore without conflicts, could you please rebase it?
>
> Attached 20th version of the patches rebased onto the current master.

Thank you,

Unfortunately, looks like we're in the loop "Rebase. Wait. Repeat", since the
patch again has some conflicts (this time I hope just minor conflicts). Which
is probably understandable taking into account the size of it. Maybe we need to
think about how to proceed with this, do you have any ideas?

On 01.12.2018 15:36, Dmitry Dolgov wrote:
>> On Mon, Nov 26, 2018 at 5:11 PM Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru> wrote:
>>
>> On 26.11.2018 15:57, Dmitry Dolgov wrote:
>>
>>>> On Tue, Jul 3, 2018 at 2:57 PM Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com> wrote:
>>>>
>>>> 2018-07-03 14:30 GMT+02:00 Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru>:
>>>>> Attached 16th version of the patches:
>>>>> * changed type of new SQL keyword STRING
>>>>> (STRING is used as a function parameter name in Pl/Tcl tests)
>>>>> * removed implicit coercion via I/O from JSON_VALUE (see below)
>>> Unfortunately, the current version of patch 0010-add-invisible-coercion-form
>>> doesn't not apply anymore without conflicts, could you please rebase it?
>> Attached 20th version of the patches rebased onto the current master.
> Thank you,
>
> Unfortunately, looks like we're in the loop "Rebase. Wait. Repeat", since the
> patch again has some conflicts (this time I hope just minor conflicts). Which
> is probably understandable taking into account the size of it. Maybe we need to
> think about how to proceed with this, do you have any ideas?

Attached 21st version of the patches.

I decided to include here patch 0000 with complete jsonpath implementation (it
is a squash of all 6 jsonpath-v21 patches). I hope this will simplify reviewing
and testing in cfbot.cputube.org.

If it will conflict again, then you can see all SQL/JSON v21 patches
successfully applied in our GitHub repository on the following branches:
https://github.com/postgrespro/sqljson/tree/sqljson_v21 (one commit per patch)
https://github.com/postgrespro/sqljson/tree/sqljson (original commit history)

--
Nikita Glukhov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Hi,

On 2018-12-05 02:01:19 +0300, Nikita Glukhov wrote:

> + if (!PG_ARGISNULL(1) &&
> + strncmp("any", VARDATA(type), VARSIZE_ANY_EXHDR(type)))
> + {
> + JsonLexContext *lex;
> + JsonTokenType tok;
> +
> + lex = makeJsonLexContext(json, false);
> +
> + /* Lex exactly one token from the input and check its type. */
> + PG_TRY();
> + {
> + json_lex(lex);
> + }
> + PG_CATCH();
> + {
> + if (ERRCODE_TO_CATEGORY(geterrcode()) == ERRCODE_DATA_EXCEPTION)
> + {
> + FlushErrorState();
> + MemoryContextSwitchTo(mcxt);
> + PG_RETURN_BOOL(false); /* invalid json */
> + }
> + PG_RE_THROW();
> + }
> + PG_END_TRY();

It baffles me that a year after I raised this as a serious issue, in
this thread, this patch still contains code like this.

Greetings,

Andres Freund

Attached 34th version of the patches.

On 16.02.2019 8:12, Andres Freund wrote:

> On 2018-12-05 02:01:19 +0300, Nikita Glukhov wrote:
>> + JsonLexContext *lex;
>> + JsonTokenType tok;
>> +
>> + lex = makeJsonLexContext(json, false);
>> +
>> + /* Lex exactly one token from the input and check its type. */
>> + PG_TRY();
>> + {
>> + json_lex(lex);
>> + }
>> + PG_CATCH();
>> + {
>> + if (ERRCODE_TO_CATEGORY(geterrcode()) == ERRCODE_DATA_EXCEPTION)
>> + {
>> + FlushErrorState();
>> + MemoryContextSwitchTo(mcxt);
>> + PG_RETURN_BOOL(false); /* invalid json */
>> + }
>> + PG_RE_THROW();
>> + }
>> + PG_END_TRY();
>
> It baffles me that a year after I raised this as a serious issue, in
> this thread, this patch still contains code like this.

PG_TRY/PG_CATCH was removed here: 'throwErrors' flag was added to JsonLexContext
instead.

Also usage of subtransactions is SQL/JSON functions (JsonExpr node) was
optimized: they can be not only omitted in ERROR ON ERROR case but also when
the resulting conversion from the SQL/JSON item type to the target SQL type is
no-op.

Below are the results of simple performance test
(operator #>> uses optimization which I recently posted in the separate patch):

query | time, ms
-------------------------------------------------------------------
JSON_VALUE(js, '$.x.y.z' RETURNING text) = '123' | 1923,360 (subtrans!)
JSON_VALUE(js, '$.x.y.z' RETURNING text
ERROR ON ERROR) = '123' | 970,604
JSON_VALUE(js, '$.x.y.z' RETURNING numeric) = '123' | 792,412
JSON_VALUE(js, '$.x.y.z' RETURNING numeric
ERROR ON ERROR) = '123' | 786,647

(js->'x'->'y'->'z') = '123' | 1104,470
(js->'x'->'y'->'z')::numeric = '123' | 940,037
(js->'x'->'y'->>'z') = '123' | 688,484

(js #> '{x,y,z}') = '123' | 1127,661
(js #> '{x,y,z}')::numeric = '123' | 971,931
(js #>> '{x,y,z}') = '123' | 718,173

Table with jsonb rows like '{"x": {"y": {"z": 123}}}':
CREATE TABLE t AS
SELECT JSON_OBJECT('x' : JSON_OBJECT('y' : JSON_OBJECT('z' : i))) js
FROM generate_series(1, 3000000) i;

Example query:
SELECT * FROM t WHERE JSON_VALUE(js, '$.x.y.z' RETURNING numeric) = '123';

--
Nikita Glukhov
Postgres Professional:http://www.postgrespro.com
The Russian Postgres Company

Attached 36th version of the patches rebased onto jsonpath v36.

--
Nikita Glukhov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

On 3/5/19 5:35 PM, Nikita Glukhov wrote:
> Attached 36th version of the patches rebased onto jsonpath v36.
While testing this patch a found a few issues:

[1] I was not able to apply the patch to the current HEAD. However, it
applies cleanly to commit: e988878f85 (NOTE: I did not investigate which
commit between e988878f85 and HEAD caused problems).

[2] JsonPath array slicing does not work. I'm not aware of a
comprehensive list of JsonPath features/syntax that is targeted for
support; however, I did try various forms of array slicing, which don't
currently work.

Here are a few examples:

The input document is the same in each example.

{
  "a1": 123,
  "b1": "xxx",
  "c1": {
    "a2": 456,
    "b2": "yyy",
    "c2": [
      {"a3": 777, "b3": "7z"},
      {"a3": 888, "b3": "8z"}
    ]
  }
}

array wildcard selector [*] works: $.c1.c2[*].b3

# select json_path_query('{"a1": 123, "b1": "xxx", "c1": {"a2": 456,
"b2": "yyy", "c2": [{"a3": 777, "b3": "7z"},{"a3": 888, "b3":
"8z"}]}}'::json, '$.c1.c2[*].b3'::jsonpath);
 json_path_query
-----------------
 "7z"
 "8z"
(2 rows)

array index selector [0] works: $.c1.c2[0].b3

jsonpatch=# select json_path_query('{"a1": 123, "b1": "xxx", "c1":
{"a2": 456, "b2": "yyy", "c2": [{"a3": 777, "b3": "7z"},{"a3": 888,
"b3": "8z"}]}}'::json, '$.c1.c2[0].b3'::jsonpath);
 json_path_query
-----------------
 "7z"
(1 row)

array slicing [0:], [:1], and [0:1] do not work:$.c1.c2[0:].b3

# select json_path_query('{"a1": 123, "b1": "xxx", "c1": {"a2": 456,
"b2": "yyy", "c2": [{"a3": 777, "b3": "7z"},{"a3": 888, "b3":
"8z"}]}}'::json, '$.c1.c2[0:].b3'::jsonpath);
2019-05-13 20:47:48.740 EDT [21856] ERROR:  bad jsonpath representation
at character 147
2019-05-13 20:47:48.740 EDT [21856] DETAIL:  syntax error, unexpected
':', expecting ',' or ']' at or near ":"
2019-05-13 20:47:48.740 EDT [21856] STATEMENT:  select
json_path_query('{"a1": 123, "b1": "xxx", "c1": {"a2": 456, "b2": "yyy",
"c2": [{"a3": 777, "b3": "7z"},{"a3": 888, "b3": "8z"}]}}'::json,
'$.c1.c2[0:].b3'::jsonpath);
ERROR:  bad jsonpath representation
LINE 1: ...7, "b3": "7z"},{"a3": 888, "b3": "8z"}]}}'::json, '$.c1.c2[0...
                                                             ^
DETAIL:  syntax error, unexpected ':', expecting ',' or ']' at or near ":"

Hi!

On Tue, May 14, 2019 at 3:54 AM Andrew Alsup <bluesbreaker(at)gmail(dot)com> wrote:
> array slicing [0:], [:1], and [0:1] do not work:$.c1.c2[0:].b3
>
> # select json_path_query('{"a1": 123, "b1": "xxx", "c1": {"a2": 456,
> "b2": "yyy", "c2": [{"a3": 777, "b3": "7z"},{"a3": 888, "b3":
> "8z"}]}}'::json, '$.c1.c2[0:].b3'::jsonpath);
> 2019-05-13 20:47:48.740 EDT [21856] ERROR: bad jsonpath representation
> at character 147
> 2019-05-13 20:47:48.740 EDT [21856] DETAIL: syntax error, unexpected
> ':', expecting ',' or ']' at or near ":"
> 2019-05-13 20:47:48.740 EDT [21856] STATEMENT: select
> json_path_query('{"a1": 123, "b1": "xxx", "c1": {"a2": 456, "b2": "yyy",
> "c2": [{"a3": 777, "b3": "7z"},{"a3": 888, "b3": "8z"}]}}'::json,
> '$.c1.c2[0:].b3'::jsonpath);
> ERROR: bad jsonpath representation
> LINE 1: ...7, "b3": "7z"},{"a3": 888, "b3": "8z"}]}}'::json, '$.c1.c2[0...
> ^
> DETAIL: syntax error, unexpected ':', expecting ',' or ']' at or near ":"

There is no color syntax from array slicing in jsonpath. Instead of
"[0:]" one should write "[0 to last]". See documentation
https://www.postgresql.org/docs/devel/datatype-json.html#TYPE-JSONPATH-ACCESSORS

------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

On Tue, May 14, 2019 at 12:54 PM Andrew Alsup <bluesbreaker(at)gmail(dot)com> wrote:
> On 3/5/19 5:35 PM, Nikita Glukhov wrote:
> > Attached 36th version of the patches rebased onto jsonpath v36.
> While testing this patch a found a few issues:
>
> [1] I was not able to apply the patch to the current HEAD. However, it
> applies cleanly to commit: e988878f85 (NOTE: I did not investigate which
> commit between e988878f85 and HEAD caused problems).

Thanks for that note, which should help other reviewers/testers
looking a this patch set in CF1. I hope we can eventually get a
rebased patch set, though.

--
Thomas Munro
https://enterprisedb.com

On 15.07.2019 5:35, Thomas Munro wrote:

> On Tue, May 14, 2019 at 12:54 PM Andrew Alsup <bluesbreaker(at)gmail(dot)com> wrote:
>> On 3/5/19 5:35 PM, Nikita Glukhov wrote:
>>> Attached 36th version of the patches rebased onto jsonpath v36.
>> While testing this patch a found a few issues:
>>
>> [1] I was not able to apply the patch to the current HEAD. However, it
>> applies cleanly to commit: e988878f85 (NOTE: I did not investigate which
>> commit between e988878f85 and HEAD caused problems).
> Thanks for that note, which should help other reviewers/testers
> looking a this patch set in CF1. I hope we can eventually get a
> rebased patch set, though.
>
Attached 37th version of the patches rebased onto current master.

Preliminary patch #1 contains:
 - implementation of jsonpath .datetime() method (see [1])
 - jsonpath support for json type (should be posted later in a separate
thread)

[1]
https://www.postgresql.org/message-id/CAPpHfduLcTtOx5dqs6ehDVy2cbLDci9JTkKdwFf1DzSy5dMjBA%40mail.gmail.com

--
Nikita Glukhov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

On 2019-Jul-16, Nikita Glukhov wrote:

> Attached 37th version of the patches rebased onto current master.
>
> Preliminary patch #1 contains:
>  - implementation of jsonpath .datetime() method (see [1])
>  - jsonpath support for json type (should be posted later in a separate
> thread)

0001 doesn't apply anymore. Can you please rebase?

(Only now I realize that this SQL/JSON stuff has its own commitfest
entry, separate from the JSON_TABLE one.)

Thanks,

--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 04.09.2019 1:27, Alvaro Herrera wrote:

> On 2019-Jul-16, Nikita Glukhov wrote:
>
>> Attached 37th version of the patches rebased onto current master.
>>
>> Preliminary patch #1 contains:
>>  - implementation of jsonpath .datetime() method (see [1])
>>  - jsonpath support for json type (should be posted later in a separate
>> thread)
> 0001 doesn't apply anymore. Can you please rebase?
>
> (Only now I realize that this SQL/JSON stuff has its own commitfest
> entry, separate from the JSON_TABLE one.)
>
> Thanks,

Attached 38th version of the patches rebased onto current master.

--
Nikita Glukhov
Postgres Professional:http://www.postgrespro.com
The Russian Postgres Company

On 17.09.2019 0:28, Nikita Glukhov wrote:

> On 04.09.2019 1:27, Alvaro Herrera wrote:
>
>> On 2019-Jul-16, Nikita Glukhov wrote:
>>
>>> Attached 37th version of the patches rebased onto current master.
>>>
>>> Preliminary patch #1 contains:
>>>   - implementation of jsonpath .datetime() method (see [1])
>>>   - jsonpath support for json type (should be posted later in a
>>> separate
>>> thread)
>> 0001 doesn't apply anymore.  Can you please rebase?
>>
>> (Only now I realize that this SQL/JSON stuff has its own commitfest
>> entry, separate from the JSON_TABLE one.)
>>
>> Thanks,
>
> Attached 38th version of the patches rebased onto current master.
>
Sorry, attached new edition of 38th version with minor corrections
described in "SQL/JSON: JSON_TABLE" thread.

--
Nikita Glukhov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

This has recently been broken -- please rebase. (JSON_TABLE likewise).

--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 25.09.2019 23:55, Alvaro Herrera wrote:

> This has recently been broken -- please rebase. (JSON_TABLE likewise).

Attached 39th version of the patches rebased onto current master.

--
Nikita Glukhov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

On 9/27/19 9:42 PM, Nikita Glukhov wrote:
> Attached 39th version of the patches rebased onto current master.
>
> --
> Nikita Glukhov
> Postgres Professional: http://www.postgrespro.com
> The Russian Postgres Company
I am unable to cleanly apply this patch.
I've tried applying to the current master (4f4061b2dd)
I've also tried apply to commit b81a9c2fc5 back through 709d003fbd, with
no success.

Can you please tell me to which commit I can apply the patch? Is the
following command correct for applying the patch files?

`patch -p1 < ~/Downloads/0001-Jsonpath-support-for-json-v39.patch`

Thanks for your help,
Andrew Alsup

On 21.10.2019 19:00, Andrew Alsup wrote:
> On 9/27/19 9:42 PM, Nikita Glukhov wrote:
>> Attached 39th version of the patches rebased onto current master.
> I am unable to cleanly apply this patch.
> I've tried applying to the current master (4f4061b2dd)
> I've also tried apply to commit b81a9c2fc5 back through 709d003fbd,
> with no success.
>
> Can you please tell me to which commit I can apply the patch? Is the
> following command correct for applying the patch files?
>
> `patch -p1 < ~/Downloads/0001-Jsonpath-support-for-json-v39.patch`
>
> Thanks for your help,
> Andrew Alsup
>
v39 patch is based on 5ee96b3e2221d154ffcb719bd2dee1179c53f821

Use the following git command to apply patches:

git am ~/Downloads/0001-Jsonpath-support-for-json-v39.patch

--
Nikita Glukhov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

On 10/21/19 12:44 PM, Nikita Glukhov wrote:
>
> v39 patch is based on 5ee96b3e2221d154ffcb719bd2dee1179c53f821
>
> Use the following git command to apply patches:
>
> git am ~/Downloads/0001-Jsonpath-support-for-json-v39.patch
>

Thank you. The patch applied fine, with no errors.

Is this the type of testing that would be helpful? I plan to construct a
number of separate queries to test more nuanced edge cases. This test
simply compares the output of 4 separate sub-queries that should provide
the same results.

SELECT
  CASE WHEN jop = jp_expr AND jop = jval AND jop = jval_path
    THEN 'pass'
    ELSE 'fail'
  END
FROM
  (
    -- jsonb operator
    SELECT count(*)
    FROM testjsonb
    WHERE j->>'abstract' LIKE 'A%'
  ) as jop,
  (
    -- jsonpath expression
    SELECT count(*)
    FROM testjsonb
    WHERE j @? '$.abstract ? (@ starts with "A")'
  ) as jp_expr,
  (
    -- json_value()
    SELECT count(*)
    FROM testjsonb
    WHERE JSON_VAlUE(j, 'lax $.abstract') LIKE 'A%'
  ) as jval,
  (
    -- json_value(jsonpath)
    SELECT count(*)
    FROM testjsonb
    WHERE JSON_VALUE(j, 'lax $.abstract ? (@ starts with "A")') IS NOT NULL
  ) as jval_path;

If I'm completely off base for how testing is normally conducted, please
let me know.

Thanks,
Andrew Alsup

Attached 40th version of the patches.

I have added some documentation which has been simply copied from [1].

Also, I have split patches more correctly, and extracted patch #2 with common
SQL/JSON clauses, that are used by both constructors and query functions.
Patches #3 and #4 are necessary only for constructors (patch #5).
Patches #5 and #6 are almost independent on patch #1, which is needed only for
query functions.

So, patches #5, #6, #7 are independent now, but the code changes in them are
still conflicting. I can rebase #6 and #7 on top of #2, if it is necessary
for separate review/commit.

[1] https://www.postgresql.org/message-id/732208d3-56c3-25a4-8f08-3be1d54ad51b%40postgrespro.ru

--
Nikita Glukhov
Postgres Professional:http://www.postgrespro.com
The Russian Postgres Company

Hi

čt 14. 11. 2019 v 17:46 odesílatel Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru>
napsal:

> Attached 40th version of the patches.
>
>
> I have added some documentation which has been simply copied from [1].
>
> Also, I have split patches more correctly, and extracted patch #2 with common
> SQL/JSON clauses, that are used by both constructors and query functions.
> Patches #3 and #4 are necessary only for constructors (patch #5).
> Patches #5 and #6 are almost independent on patch #1, which is needed only for
> query functions.
>
> So, patches #5, #6, #7 are independent now, but the code changes in them are
> still conflicting. I can rebase #6 and #7 on top of #2, if it is necessary
> for separate review/commit.
>
>
> [1] https://www.postgresql.org/message-id/732208d3-56c3-25a4-8f08-3be1d54ad51b%40postgrespro.ru
>
> I tested cumulative patch - sent in json_table patch.

I almost satisfied by quality of this patch. There is very good conformance
with standard and with Oracle. Unfortunately MySQL in this part of JSON
support is not compatible.

I found one issue, when I tested some examples from Oracle.

SELECT JSON_VALUE('{a:100}', '$.a' RETURNING int) AS value;

then the result was null.

But it is wrong, because it should to raise a exception, because this json
is broken on Postgres (Postgres requires quoted attribute names)

json_query has same problem

postgres=# SELECT JSON_QUERY('{a:100, b:200, c:300}', '$') AS value;
┌───────┐
│ value │
╞═══════╡
│ ∅ │
└───────┘
(1 row)

It should to check if input is correct json

All others was good.

I'll mark this patch as waiting for author

Regards

Pavel

> --
> Nikita Glukhov
> Postgres Professional: http://www.postgrespro.com
> The Russian Postgres Company
>
>

Attached 41th version of the patches.

Changes since previous version:
* Enabled DEFAULT clause for ON ERROR/ON EMPTY behaviors in JSON_QUERY()
* Added RETURNING clause to JSON_EXISTS() ("side effect" of implementation
EXISTS PATH columns in JSON_TABLE)
* ARRAY in EMPTY ARRAY ON ERROR clause is optional now for better Oracle
compatibility

On 17.01.2020 9:54, Pavel Stehule wrote:

> On 14. 11. 2019 v 17:46 Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru
> <mailto:n(dot)gluhov(at)postgrespro(dot)ru>> wrote:
>
> Attached 40th version of the patches.
>
> I have added some documentation which has been simply copied from [1].
>
> Also, I have split patches more correctly, and extracted patch #2 with common
> SQL/JSON clauses, that are used by both constructors and query functions.
> Patches #3 and #4 are necessary only for constructors (patch #5).
> Patches #5 and #6 are almost independent on patch #1, which is needed only for
> query functions.
>
> So, patches #5, #6, #7 are independent now, but the code changes in them are
> still conflicting. I can rebase #6 and #7 on top of #2, if it is necessary
> for separate review/commit.
>
>
> [1]https://www.postgresql.org/message-id/732208d3-56c3-25a4-8f08-3be1d54ad51b%40postgrespro.ru
>
> I tested cumulative patch - sent in json_table patch.
>
> I almost satisfied by quality of this patch. There is very good
> conformance with standard and with Oracle. Unfortunately MySQL in this
> part of JSON support is not compatible.
>
> I found one issue, when I tested some examples from Oracle.
>
> SELECT JSON_VALUE('{a:100}', '$.a' RETURNING int) AS value;
>
> then the result was null.
>
> But it is wrong, because it should to raise a exception, because this
> json is broken on Postgres (Postgres requires quoted attribute names)
>
> json_query has same problem
>
> postgres=# SELECT JSON_QUERY('{a:100, b:200, c:300}', '$') AS value;
> ┌───────┐
> │ value │
> ╞═══════╡
> │ ∅     │
> └───────┘
> (1 row)
>
> It should to check if input is correct json

By the standard, it is implementation-defined whether JSON parsing errors
should be caught by ON ERROR clause.

SQL/JSON query functions use "JSON API common syntax" which is a combination
of JSON context item and JSON path. It passes context item to JSON path
engine with ALREADY PARSED flag set to False. ALREADY PARSED flag can enable
special parsing rules.

Corresponding quotes from the standard:

10.14 <JSON API common syntax>
<JSON API common syntax> (
Parameter: "JSON API COMMON SYNTAX"
) Returns: "STATUS" and "SQL/JSON SEQUENCE"

General Rules:
...
3) General Rules of Subclause 9.39, "SQL/JSON path language: syntax and
semantics", are applied with P as PATH SPECIFICATION, C as CONTEXT ITEM,
False as ALREADY PARSED, and PC as PASSING CLAUSE; let ST be the STATUS
and let SEQ be the SQL/JSON SEQUENCE returned from the application of
those General Rules.

9.39 SQL/JSON path language: syntax and semantics

"SQL/JSON path language: syntax and semantics" [General Rules] (
Parameter: "PATH SPECIFICATION",
Parameter: "CONTEXT ITEM",
Parameter: "ALREADY PARSED",
Parameter: "PASSING CLAUSE"
) Returns: "STATUS" and "SQL/JSON SEQUENCE"

General Rules:
...

4) If ALREADY PARSED is False, then it is implementation-defined whether the
following rules are applied:
a) The General Rules of Subclause 9.36, "Parsing JSON text", are applied with
JT as JSON TEXT, an implementation-defined <JSON key uniqueness constraint>
as UNIQUENESS CONSTRAINT, and FO as FORMAT OPTION; let ST be the STATUS and
let CISJI be the SQL/JSON ITEM returned from the application of those
General Rules.
b) If ST is not successful completion, then ST is returned as the STATUS of
this application of these General Rules, and no further General Rules of
this Subclause are applied.

I decided to apply this rules, so the parsing errors are caught now by ON ERROR
(NULL ON ERROR is by default).

postgres=# SELECT JSON_VALUE('error', '$' ERROR ON ERROR);
ERROR: invalid input syntax for type json
DETAIL: Token "error" is invalid.
CONTEXT: JSON data, line 1: error

I'm not sure if it would be better to add an implicit cast to json type that
will be executed before so that parsing errors can no longer be caught.
But implicit casting can simplify a bit execution of SQL/JSON query functions.

I have checked error handling in JSON parsing in Oracle 18c/19c, and it behaves
like our current implementation. But Oracle seems to do JSON parsing on demand:

Oracle19c> SELECT JSON_VALUE('{a:1 error, b:2}', '$.a' ERROR ON ERROR) FROM dual;
1

Oracle19c> SELECT JSON_VALUE('{a:1 error, b:2}', '$.b' ERROR ON ERROR) FROM dual;
ORA-40441: JSON syntax error

Oracle19c> SELECT JSON_VALUE('{a:1 error, b:2}', '$.b') FROM dual;
NULL

--
Nikita Glukhov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

so 18. 1. 2020 v 18:46 odesílatel Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru>
napsal:

> Attached 41th version of the patches.
>
> Changes since previous version:
> * Enabled DEFAULT clause for ON ERROR/ON EMPTY behaviors in JSON_QUERY()
> * Added RETURNING clause to JSON_EXISTS() ("side effect" of implementation
> EXISTS PATH columns in JSON_TABLE)
> * ARRAY in EMPTY ARRAY ON ERROR clause is optional now for better Oracle
> compatibility
>
>
> On 17.01.2020 9:54, Pavel Stehule wrote:
>
> On 14. 11. 2019 v 17:46 Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru> wrote:
>
>> Attached 40th version of the patches.
>>
>> I have added some documentation which has been simply copied from [1].
>>
>> Also, I have split patches more correctly, and extracted patch #2 with common
>> SQL/JSON clauses, that are used by both constructors and query functions.
>> Patches #3 and #4 are necessary only for constructors (patch #5).
>> Patches #5 and #6 are almost independent on patch #1, which is needed only for
>> query functions.
>>
>> So, patches #5, #6, #7 are independent now, but the code changes in them are
>> still conflicting. I can rebase #6 and #7 on top of #2, if it is necessary
>> for separate review/commit.
>>
>>
>> [1] https://www.postgresql.org/message-id/732208d3-56c3-25a4-8f08-3be1d54ad51b%40postgrespro.ru
>>
>> I tested cumulative patch - sent in json_table patch.
>
> I almost satisfied by quality of this patch. There is very good
> conformance with standard and with Oracle. Unfortunately MySQL in this part
> of JSON support is not compatible.
>
> I found one issue, when I tested some examples from Oracle.
>
> SELECT JSON_VALUE('{a:100}', '$.a' RETURNING int) AS value;
>
> then the result was null.
>
> But it is wrong, because it should to raise a exception, because this json
> is broken on Postgres (Postgres requires quoted attribute names)
>
> json_query has same problem
>
> postgres=# SELECT JSON_QUERY('{a:100, b:200, c:300}', '$') AS value;
> ┌───────┐
> │ value │
> ╞═══════╡
> │ ∅ │
> └───────┘
> (1 row)
>
> It should to check if input is correct json
>
> By the standard, it is implementation-defined whether JSON parsing errors
> should be caught by ON ERROR clause.
>
> SQL/JSON query functions use "JSON API common syntax" which is a combination
> of JSON context item and JSON path. It passes context item to JSON path
> engine with ALREADY PARSED flag set to False. ALREADY PARSED flag can enable
> special parsing rules.
>
> Corresponding quotes from the standard:
>
> 10.14 <JSON API common syntax>
> <JSON API common syntax> (
> Parameter: "JSON API COMMON SYNTAX"
> ) Returns: "STATUS" and "SQL/JSON SEQUENCE"
>
> General Rules:
> ...
> 3) General Rules of Subclause 9.39, "SQL/JSON path language: syntax and
> semantics", are applied with P as PATH SPECIFICATION, C as CONTEXT ITEM,
> False as ALREADY PARSED, and PC as PASSING CLAUSE; let ST be the STATUS
> and let SEQ be the SQL/JSON SEQUENCE returned from the application of
> those General Rules.
>
> 9.39 SQL/JSON path language: syntax and semantics
>
> "SQL/JSON path language: syntax and semantics" [General Rules] (
> Parameter: "PATH SPECIFICATION",
> Parameter: "CONTEXT ITEM",
> Parameter: "ALREADY PARSED",
> Parameter: "PASSING CLAUSE"
> ) Returns: "STATUS" and "SQL/JSON SEQUENCE"
>
> General Rules:
> ...
>
> 4) If ALREADY PARSED is False, then it is implementation-defined whether the
> following rules are applied:
> a) The General Rules of Subclause 9.36, "Parsing JSON text", are applied with
> JT as JSON TEXT, an implementation-defined <JSON key uniqueness constraint>
> as UNIQUENESS CONSTRAINT, and FO as FORMAT OPTION; let ST be the STATUS and
> let CISJI be the SQL/JSON ITEM returned from the application of those
> General Rules.
> b) If ST is not successful completion, then ST is returned as the STATUS of
> this application of these General Rules, and no further General Rules of
> this Subclause are applied.
>
>
> I decided to apply this rules, so the parsing errors are caught now by ON ERROR
> (NULL ON ERROR is by default).
>
> postgres=# SELECT JSON_VALUE('error', '$' ERROR ON ERROR);
> ERROR: invalid input syntax for type json
> DETAIL: Token "error" is invalid.
> CONTEXT: JSON data, line 1: error
>
> I'm not sure if it would be better to add an implicit cast to json type that
> will be executed before so that parsing errors can no longer be caught.
> But implicit casting can simplify a bit execution of SQL/JSON query functions.
>
>
> I have checked error handling in JSON parsing in Oracle 18c/19c, and it behaves
> like our current implementation. But Oracle seems to do JSON parsing on demand:
>
> Oracle19c> SELECT JSON_VALUE('{a:1 error, b:2}', '$.a' ERROR ON ERROR) FROM dual;
> 1
>
> Oracle19c> SELECT JSON_VALUE('{a:1 error, b:2}', '$.b' ERROR ON ERROR) FROM dual;
> ORA-40441: JSON syntax error
>
> Oracle19c> SELECT JSON_VALUE('{a:1 error, b:2}', '$.b') FROM dual;
> NULL
>
>
Everywhere I don't like default masking error. I think so can be very
confusing to get NULL (by default) instead a error of broken format.

I vote for check of input is correct JSON, and if it, then start
processing. Else to raise a error.

More - our JSON Parser is different than Oracle's JSON parser. And if
somebody will run Oracle's JSONs, then he get some result on Oracle. But on
Postgres, same JSON can be invalid, and he get NULL.

The raising some errors looks like only one safe variant.

Regards

Pavel

> --
> Nikita Glukhov
> Postgres Professional: http://www.postgrespro.com
> The Russian Postgres Company
>

Attached 42th version of the patches.

On 18.01.2020 21:21, Pavel Stehule wrote:
> On 18. 1. 2020 v 18:46 Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru
> <mailto:n(dot)gluhov(at)postgrespro(dot)ru>> wrote:
>
> Attached 41th version of the patches.
>
> Changes since previous version:
> * Enabled DEFAULT clause for ON ERROR/ON EMPTY behaviors in JSON_QUERY()
> * Added RETURNING clause to JSON_EXISTS() ("side effect" of implementation
> EXISTS PATH columns in JSON_TABLE)
> * ARRAY in EMPTY ARRAY ON ERROR clause is optional now for better Oracle
> compatibility
>
>
> On 17.01.2020 9:54, Pavel Stehule wrote:
>
>> I tested cumulative patch - sent in json_table patch.
>>
>> I almost satisfied by quality of this patch. There is very good
>> conformance with standard and with Oracle. Unfortunately MySQL in
>> this part of JSON support is not compatible.
>>
>> I found one issue, when I tested some examples from Oracle.
>>
>> SELECT JSON_VALUE('{a:100}', '$.a' RETURNING int) AS value;
>>
>> then the result was null.
>>
>> But it is wrong, because it should to raise a exception, because
>> this json is broken on Postgres (Postgres requires quoted
>> attribute names)
>>
>> json_query has same problem
>>
>> postgres=# SELECT JSON_QUERY('{a:100, b:200, c:300}', '$') AS value;
>> ┌───────┐
>> │ value │
>> ╞═══════╡
>> │ ∅     │
>> └───────┘
>> (1 row)
>>
>> It should to check if input is correct json
>
> By the standard, it is implementation-defined whether JSON parsing errors
> should be caught by ON ERROR clause.
>
> SQL/JSON query functions use "JSON API common syntax" which is a combination
> of JSON context item and JSON path. It passes context item to JSON path
> engine with ALREADY PARSED flag set to False. ALREADY PARSED flag can enable
> special parsing rules.
>
> Corresponding quotes from the standard:
>
> 10.14 <JSON API common syntax>
> <JSON API common syntax> (
> Parameter: "JSON API COMMON SYNTAX"
> ) Returns: "STATUS" and "SQL/JSON SEQUENCE"
>
> General Rules:
> ...
> 3) General Rules of Subclause 9.39, "SQL/JSON path language: syntax and
> semantics", are applied with P as PATH SPECIFICATION, C as CONTEXT ITEM,
> False as ALREADY PARSED, and PC as PASSING CLAUSE; let ST be the STATUS
> and let SEQ be the SQL/JSON SEQUENCE returned from the application of
> those General Rules.
>
> 9.39 SQL/JSON path language: syntax and semantics
>
> "SQL/JSON path language: syntax and semantics" [General Rules] (
> Parameter: "PATH SPECIFICATION",
> Parameter: "CONTEXT ITEM",
> Parameter: "ALREADY PARSED",
> Parameter: "PASSING CLAUSE"
> ) Returns: "STATUS" and "SQL/JSON SEQUENCE"
>
> General Rules:
> ...
>
> 4) If ALREADY PARSED is False, then it is implementation-defined whether the
> following rules are applied:
> a) The General Rules of Subclause 9.36, "Parsing JSON text", are applied with
> JT as JSON TEXT, an implementation-defined <JSON key uniqueness constraint>
> as UNIQUENESS CONSTRAINT, and FO as FORMAT OPTION; let ST be the STATUS and
> let CISJI be the SQL/JSON ITEM returned from the application of those
> General Rules.
> b) If ST is not successful completion, then ST is returned as the STATUS of
> this application of these General Rules, and no further General Rules of
> this Subclause are applied.
>
>
> I decided to apply this rules, so the parsing errors are caught now by ON ERROR
> (NULL ON ERROR is by default).
>
> postgres=# SELECT JSON_VALUE('error', '$' ERROR ON ERROR);
> ERROR: invalid input syntax for type json
> DETAIL: Token "error" is invalid.
> CONTEXT: JSON data, line 1: error
>
> I'm not sure if it would be better to add an implicit cast to json type that
> will be executed before so that parsing errors can no longer be caught.
> But implicit casting can simplify a bit execution of SQL/JSON query functions.
>
>
> I have checked error handling in JSON parsing in Oracle 18c/19c, and it behaves
> like our current implementation. But Oracle seems to do JSON parsing on demand:
>
> Oracle19c> SELECT JSON_VALUE('{a:1 error, b:2}', '$.a' ERROR ON ERROR) FROM dual;
> 1
>
> Oracle19c> SELECT JSON_VALUE('{a:1 error, b:2}', '$.b' ERROR ON ERROR) FROM dual;
> ORA-40441: JSON syntax error
>
> Oracle19c> SELECT JSON_VALUE('{a:1 error, b:2}', '$.b') FROM dual;
> NULL
>
>
> Everywhere I don't like default masking error. I think so can be very
> confusing to get NULL
> (by default) instead a error of broken format.
>
> I vote for check of input is correct JSON, and if it, then start
> processing. Else to raise a error.
>
> More - our JSON Parser is different than Oracle's JSON parser. And if
> somebody will run Oracle's JSONs,
> then he get some result on Oracle. But on Postgres, same JSON can be
> invalid, and he get NULL.
>
> The raising some errors looks like only one safe variant.

I have removed handling of parsing errors in SQL/JSON functions and JSON_TABLE.
Now, FORMAT JSON expressions (implicit or explicit) are simply transformed into
ordinary casts to json type, and these casts are executed before execution of
SQL/JSON functions. Previously, separate expression was created for such casts,
and it was executed in the separate subtransaction in ExecEvalJsonExpr(). So,
this change also simplifies the code a bit.

--
Nikita Glukhov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

On 2020-03-02 23:33, Nikita Glukhov wrote:
> Attached 42th version of the patches.

> v1-0001-Add-jsonpath-pg-modifier-for-enabling-extensions.patch
> v1-0002-Add-raw-jbvArray-and-jbvObject-support-to-jsonpat.patch
> v1-0003-Add-jsonpath-sequence-constructors.patch
> v1-0004-Add-jsonpath-array-constructors.patch
> v1-0005-Add-jsonpath-object-constructors.patch
> v1-0006-Add-jsonpath-object-subscripting.patch

I can't get these to apply against master.

$ patch --dry-run -b -l -F 15 -p 1 <
0001-Jsonpath-support-for-json-v42.patch
can't find file to patch at input line 38

(tried -p o and -p 1 -- am I doing it wrong?)

Maybe it needs something else applied first?

Erik Rijkers

On 03.03.2020 2:12, Erik Rijkers wrote:

> On 2020-03-02 23:33, Nikita Glukhov wrote:
>> Attached 42th version of the patches.
>
>> v1-0001-Add-jsonpath-pg-modifier-for-enabling-extensions.patch
>> v1-0002-Add-raw-jbvArray-and-jbvObject-support-to-jsonpat.patch
>> v1-0003-Add-jsonpath-sequence-constructors.patch
>> v1-0004-Add-jsonpath-array-constructors.patch
>> v1-0005-Add-jsonpath-object-constructors.patch
>> v1-0006-Add-jsonpath-object-subscripting.patch
>
> I can't get these to apply against master.
>
>
> $ patch --dry-run -b -l -F 15 -p 1 <
> 0001-Jsonpath-support-for-json-v42.patch
> can't find file to patch at input line 38
>
> (tried  -p o and -p 1 -- am I doing it wrong?)
>
>
> Maybe it needs something else applied first?
>
>
> Erik Rijkers

Sorry, the wrong patch set was attached.

Patches can be applied the following command (which also creates a commit):
$ git am 0001-Jsonpath-support-for-json-v42.patch

--
Nikita Glukhov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

On 2020-03-03 00:24, Nikita Glukhov wrote:
> On 03.03.2020 2:12, Erik Rijkers wrote:
>
>> On 2020-03-02 23:33, Nikita Glukhov wrote:
>>> Attached 42th version of the patches.

20200302/0001-Jsonpath-support-for-json-v42.patch +
20200302/0002-Add-common-SQL_JSON-clauses-v42.patch+
20200302/0003-Add-invisible-coercion-form-v42.patch+
20200302/0004-Add-function-formats-v42.patch +
20200302/0005-SQLJSON-constructors-v42.patch +
20200302/0006-IS-JSON-predicate-v42.patch +
20200302/0007-SQLJSON-query-functions-v42.patch +

Thanks -- those applied fine.

Compiling, I get this error from the pg_stat_statements module (in
contrib):

-- [2020.03.03 02:22:20 json_functions/0] make contrib
pg_stat_statements.c: In function ‘JumbleExpr’:
pg_stat_statements.c:2933:29: error: ‘JsonExpr’ {aka ‘struct JsonExpr’}
has no member named ‘raw_expr’
2933 | JumbleExpr(jstate, jexpr->raw_expr);
| ^~
make[1]: *** [pg_stat_statements.o] Error 1
make: *** [all-pg_stat_statements-recurse] Error 2
-- contrib make returned 2 - abort
../../src/Makefile.global:919: recipe for target 'pg_stat_statements.o'
failed
Makefile:93: recipe for target 'all-pg_stat_statements-recurse' failed
pg_stat_statements.c: In function ‘JumbleExpr’:
pg_stat_statements.c:2933:29: error: ‘JsonExpr’ {aka ‘struct JsonExpr’}
has no member named ‘raw_expr’
2933 | JumbleExpr(jstate, jexpr->raw_expr);
| ^~
make[1]: *** [pg_stat_statements.o] Error 1
make: *** [install-pg_stat_statements-recurse] Error 2
-- contrib make install returned 2 - abort

(so I've edited out pg_stat_statements from the contrib/Makefile and
compiled a working server for further testing.)

Thanks,

Erik Rijkers

út 3. 3. 2020 v 0:24 odesílatel Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru>
napsal:

> On 03.03.2020 2:12, Erik Rijkers wrote:
>
> > On 2020-03-02 23:33, Nikita Glukhov wrote:
> >> Attached 42th version of the patches.
> >
> >> v1-0001-Add-jsonpath-pg-modifier-for-enabling-extensions.patch
> >> v1-0002-Add-raw-jbvArray-and-jbvObject-support-to-jsonpat.patch
> >> v1-0003-Add-jsonpath-sequence-constructors.patch
> >> v1-0004-Add-jsonpath-array-constructors.patch
> >> v1-0005-Add-jsonpath-object-constructors.patch
> >> v1-0006-Add-jsonpath-object-subscripting.patch
> >
> > I can't get these to apply against master.
> >
> >
> > $ patch --dry-run -b -l -F 15 -p 1 <
> > 0001-Jsonpath-support-for-json-v42.patch
> > can't find file to patch at input line 38
> >
> > (tried -p o and -p 1 -- am I doing it wrong?)
> >
> >
> > Maybe it needs something else applied first?
> >
> >
> > Erik Rijkers
>
> Sorry, the wrong patch set was attached.
>
> Patches can be applied the following command (which also creates a commit):
> $ git am 0001-Jsonpath-support-for-json-v42.patch
>

make check fails

but probably it is only forgotten actualization

Regards

Pavel

> --
> Nikita Glukhov
> Postgres Professional: http://www.postgrespro.com
> The Russian Postgres Company
>

Attached 43rd version of the patches.

The previous patch #4 ("Add function formats") was removed.
Instead, introduced new executor node JsonCtorExpr which is used to wrap
SQL/JSON constructor function calls (FuncExpr, Aggref, WindowFunc).

Also, JsonIsPredicate node began to be used as executor node.
This helped to remove unnecessary json[b]_is_valid() user functions.

On 06.03.2020 11:16, Pavel Stehule wrote:

> make check fails
> but probably it is only forgotten actualization

Fixed.

--
Nikita Glukhov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

čt 12. 3. 2020 v 0:09 odesílatel Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru>
napsal:

> Attached 43rd version of the patches.
>
> The previous patch #4 ("Add function formats") was removed.
> Instead, introduced new executor node JsonCtorExpr which is used to wrap
> SQL/JSON constructor function calls (FuncExpr, Aggref, WindowFunc).
>
> Also, JsonIsPredicate node began to be used as executor node.
> This helped to remove unnecessary json[b]_is_valid() user functions.
>
>
It looks very well - all tests passed, code looks well.

Now, when there are special nodes, then the introduction of
COERCE_INTERNAL_CAST is not necessary, and it is only my one and last
objection again this patch's set.

Regards

Pavel

> On 06.03.2020 11:16, Pavel Stehule wrote:
>
> make check fails
>
> but probably it is only forgotten actualization
>
> Fixed.
>
>
> --
> Nikita Glukhov
> Postgres Professional: http://www.postgrespro.com
> The Russian Postgres Company
>

Attached 44th version of the patches.

On 12.03.2020 16:41, Pavel Stehule wrote:
>
> On 12.03.2020 00:09 Nikita Glukhov wrote:
>
> Attached 43rd version of the patches.
>
> The previous patch #4 ("Add function formats") was removed.
> Instead, introduced new executor node JsonCtorExpr which is used to wrap
> SQL/JSON constructor function calls (FuncExpr, Aggref, WindowFunc).
>
> Also, JsonIsPredicate node began to be used as executor node.
> This helped to remove unnecessary json[b]_is_valid() user functions.
>
>
> It looks very well - all tests passed, code looks well.
>
> Now, when there are special nodes, then the introduction of
> COERCE_INTERNAL_CAST is not necessary, and it is only my one and last
> objection again this patch's set.
>
I have removed patch #3 with COERCE_INTERNAL_CAST too.

Coercions in JsonValueExpr in JsonCtorExpr, which were previously hidden with
COERCE_INTERNAL_CAST and which were outputted as RETURNING or FORMAT JSON
clauses, now moved into separate expressions.

User functions json[b]_build_object_ext() and json[b]_build_array_ext() also
can be easily removed. But it seems harder to remove new aggregate functions
json[b]_objectagg() and json[b]_agg_strict(), because they can't be called
directly from JsonCtorExpr node.

The support for json type in jsonpath also seems to be immature, so I will try
to remove it in the next iteration.

--
Nikita Glukhov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

út 17. 3. 2020 v 1:55 odesílatel Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru>
napsal:

> Attached 44th version of the patches.
>
>
> On 12.03.2020 16:41, Pavel Stehule wrote:
>
>
> On 12.03.2020 00:09 Nikita Glukhov wrote:
>
>> Attached 43rd version of the patches.
>>
>> The previous patch #4 ("Add function formats") was removed.
>> Instead, introduced new executor node JsonCtorExpr which is used to wrap
>> SQL/JSON constructor function calls (FuncExpr, Aggref, WindowFunc).
>>
>> Also, JsonIsPredicate node began to be used as executor node.
>> This helped to remove unnecessary json[b]_is_valid() user functions.
>>
>>
> It looks very well - all tests passed, code looks well.
>
> Now, when there are special nodes, then the introduction of
> COERCE_INTERNAL_CAST is not necessary, and it is only my one and last
> objection again this patch's set.
>
> I have removed patch #3 with COERCE_INTERNAL_CAST too.
>
> Coercions in JsonValueExpr in JsonCtorExpr, which were previously hidden with
> COERCE_INTERNAL_CAST and which were outputted as RETURNING or FORMAT JSON
> clauses, now moved into separate expressions.
>
> I am looking on the code, and although the code is correct it doesn't look
well (consistently with other node types).

I think so JsonFormat and JsonReturning should be node types, not just
structs. If these types will be nodes, then you can simplify _readJsonExpr
and all node operations on this node.

>
>
> User functions json[b]_build_object_ext() and json[b]_build_array_ext() also
> can be easily removed. But it seems harder to remove new aggregate functions
> json[b]_objectagg() and json[b]_agg_strict(), because they can't be called
> directly from JsonCtorExpr node.
>
>
I don't see reasons for another reduction now. Can be great if you can
finalize work what you plan for pg13.

+<->READ_ENUM_FIELD(on_error.btype, JsonBehaviorType);
+<->READ_NODE_FIELD(on_error.default_expr);
+<->READ_ENUM_FIELD(on_empty.btype, JsonBehaviorType);
+<->READ_NODE_FIELD(on_empty.default_expr);

JsonBehavior is node type. Then why you don't write just

READ_NODE_FIELD(on_error);
READ_NODE_FIELD(on_empty)

??

And maybe the code can be better if you don't use JsonPassing struct (or
use it only inside gram.y) and pass just List *names, List *values.

Nodes should to contains another nodes or scalar types. Using structs (that
are not nodes) inside doesn't look consistently.

I found some not finished code in 0003 patch
+
+json_name_and_value:
+/* TODO
+<-><--><-->KEY c_expr VALUE_P json_value_expr %prec POSTFIXOP
+<-><--><--><-->{ $$ = makeJsonKeyValue($2, $4); }
+<-><--><-->|
+*/

> The support for json type in jsonpath also seems to be immature, so I will try
> to remove it in the next iteration.
>
> What do you think? This patch is little bit off topic, so if you don't
need it, then can be removed. Is there some dependency for "jsontable" ?

Regards

Pavel

> --
> Nikita Glukhov
> Postgres Professional: http://www.postgrespro.com
> The Russian Postgres Company
>
>

Attached 45th version of the patches.

Nodes JsonFormat, JsonReturning, JsonPassing, JsonBehavior were fixed.

On 17.03.2020 21:35, Pavel Stehule wrote:
>
> út 17. 3. 2020 v 1:55 odesílatel Nikita Glukhov
> <n(dot)gluhov(at)postgrespro(dot)ru <mailto:n(dot)gluhov(at)postgrespro(dot)ru>> napsal:
>
> Attached 44th version of the patches.
>
>
> On 12.03.2020 16:41, Pavel Stehule wrote:
>>
>> On 12.03.2020 00:09 Nikita Glukhov wrote:
>>
>> Attached 43rd version of the patches.
>>
>> The previous patch #4 ("Add function formats") was removed.
>> Instead, introduced new executor node JsonCtorExpr which is used to wrap
>> SQL/JSON constructor function calls (FuncExpr, Aggref, WindowFunc).
>>
>> Also, JsonIsPredicate node began to be used as executor node.
>> This helped to remove unnecessary json[b]_is_valid() user functions.
>>
>>
>> It looks very well - all tests passed, code looks well.
>>
>> Now, when there are special nodes, then the introduction of
>> COERCE_INTERNAL_CAST is not necessary, and it is only my one and
>> last objection again this patch's set.
>>
> I have removed patch #3 with COERCE_INTERNAL_CAST too.
>
> Coercions in JsonValueExpr in JsonCtorExpr, which were previously hidden with
> COERCE_INTERNAL_CAST and which were outputted as RETURNING or FORMAT JSON
> clauses, now moved into separate expressions.
>
> I am looking on the code, and although the code is correct it doesn't
> look well (consistently with other node types).
>
> I think so JsonFormat and JsonReturning should be node types, not just
> structs. If these types will be nodes, then you can simplify
> _readJsonExpr and all node operations on this node.

JsonFormat and JsonReturning was transformed into nodes, and not included
directly into other nodes now.

> User functions json[b]_build_object_ext() and json[b]_build_array_ext() also
> can be easily removed. But it seems harder to remove new aggregate functions
> json[b]_objectagg() and json[b]_agg_strict(), because they can't be called
> directly from JsonCtorExpr node.
>
>
> I don't see reasons for another reduction now. Can be great if you can
> finalize work what you plan for pg13.
>

> +<->READ_ENUM_FIELD(on_error.btype, JsonBehaviorType);
> +<->READ_NODE_FIELD(on_error.default_expr);
> +<->READ_ENUM_FIELD(on_empty.btype, JsonBehaviorType);
> +<->READ_NODE_FIELD(on_empty.default_expr);
>
> JsonBehavior is node type. Then why you don't write just
>
> READ_NODE_FIELD(on_error);
> READ_NODE_FIELD(on_empty)
>
> ??

JsonBehavior now used in JsonExpr as a pointer to node.

> And maybe the code can be better if you don't use JsonPassing struct
> (or use it only inside gram.y) and pass just List *names, List *values.
>
> Nodes should to contains another nodes or scalar types. Using structs
> (that are not nodes)  inside doesn't look consistently.
>
JsonPassing was replaced with two Lists.

> I found some not finished code in 0003 patch
> +
> +json_name_and_value:
> +/* TODO
> +<-><--><-->KEY c_expr VALUE_P json_value_expr %prec POSTFIXOP
> +<-><--><--><-->{ $$ = makeJsonKeyValue($2, $4); }
> +<-><--><-->|
> +*/
>
This is unsupported variant of standard syntax, because it seems to lead
to unresolvable conflicts. The only supports syntax is:

JSON_OBJECT(key : value)
JSON_OBJECT(key VALUE value)

>
> The support for json type in jsonpath also seems to be immature, so I will try
> to remove it in the next iteration.
>
> What do you think? This patch is little bit off topic, so if you don't
> need it, then can be removed. Is there some dependency for "jsontable" ?

There is a dependency in SQL/JSON query functions executor, because executor
uses new structure JsonItem instead of plain JsonbValue. I will try to
preserve refactoring with JsonItem introduction, but remove json support.
If it will be still unacceptable, I will try to completely remove patch #1.

--
Nikita Glukhov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

čt 19. 3. 2020 v 23:57 odesílatel Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru>
napsal:

> Attached 45th version of the patches.
>
> Nodes JsonFormat, JsonReturning, JsonPassing, JsonBehavior were fixed.
>
>
> On 17.03.2020 21:35, Pavel Stehule wrote:
>
>
> út 17. 3. 2020 v 1:55 odesílatel Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru>
> napsal:
>
>> Attached 44th version of the patches.
>>
>>
>> On 12.03.2020 16:41, Pavel Stehule wrote:
>>
>>
>> On 12.03.2020 00:09 Nikita Glukhov wrote:
>>
>>> Attached 43rd version of the patches.
>>>
>>> The previous patch #4 ("Add function formats") was removed.
>>> Instead, introduced new executor node JsonCtorExpr which is used to wrap
>>> SQL/JSON constructor function calls (FuncExpr, Aggref, WindowFunc).
>>>
>>> Also, JsonIsPredicate node began to be used as executor node.
>>> This helped to remove unnecessary json[b]_is_valid() user functions.
>>>
>>>
>> It looks very well - all tests passed, code looks well.
>>
>> Now, when there are special nodes, then the introduction of
>> COERCE_INTERNAL_CAST is not necessary, and it is only my one and last
>> objection again this patch's set.
>>
>> I have removed patch #3 with COERCE_INTERNAL_CAST too.
>>
>> Coercions in JsonValueExpr in JsonCtorExpr, which were previously hidden with
>> COERCE_INTERNAL_CAST and which were outputted as RETURNING or FORMAT JSON
>> clauses, now moved into separate expressions.
>>
>> I am looking on the code, and although the code is correct it doesn't
> look well (consistently with other node types).
>
> I think so JsonFormat and JsonReturning should be node types, not just
> structs. If these types will be nodes, then you can simplify _readJsonExpr
> and all node operations on this node.
>
>
> JsonFormat and JsonReturning was transformed into nodes, and not included
> directly into other nodes now.
>
>
> User functions json[b]_build_object_ext() and json[b]_build_array_ext() also
>> can be easily removed. But it seems harder to remove new aggregate functions
>> json[b]_objectagg() and json[b]_agg_strict(), because they can't be called
>> directly from JsonCtorExpr node.
>>
>>
> I don't see reasons for another reduction now. Can be great if you can
> finalize work what you plan for pg13.
>
>
> +<->READ_ENUM_FIELD(on_error.btype, JsonBehaviorType);
> +<->READ_NODE_FIELD(on_error.default_expr);
> +<->READ_ENUM_FIELD(on_empty.btype, JsonBehaviorType);
> +<->READ_NODE_FIELD(on_empty.default_expr);
>
> JsonBehavior is node type. Then why you don't write just
>
> READ_NODE_FIELD(on_error);
> READ_NODE_FIELD(on_empty)
>
> ??
>
> JsonBehavior now used in JsonExpr as a pointer to node.
>
>
> And maybe the code can be better if you don't use JsonPassing struct (or
> use it only inside gram.y) and pass just List *names, List *values.
>
> Nodes should to contains another nodes or scalar types. Using structs
> (that are not nodes) inside doesn't look consistently.
>
> JsonPassing was replaced with two Lists.
>
>
> I found some not finished code in 0003 patch
> +
> +json_name_and_value:
> +/* TODO
> +<-><--><-->KEY c_expr VALUE_P json_value_expr %prec POSTFIXOP
> +<-><--><--><-->{ $$ = makeJsonKeyValue($2, $4); }
> +<-><--><-->|
> +*/
>
> This is unsupported variant of standard syntax, because it seems to lead
> to unresolvable conflicts. The only supports syntax is:
>
> JSON_OBJECT(key : value)
> JSON_OBJECT(key VALUE value)
>
> ok. So please change comment from ToDo to this explanation. Maybe note in
doc (unimplemented features can be good idea)

Some these unresolvable conflicts are solved with extra parenthesis in
Postgres.

>
> The support for json type in jsonpath also seems to be immature, so I will try
>> to remove it in the next iteration.
>>
>> What do you think? This patch is little bit off topic, so if you don't
> need it, then can be removed. Is there some dependency for "jsontable" ?
>
> There is a dependency in SQL/JSON query functions executor, because executor
> uses new structure JsonItem instead of plain JsonbValue. I will try to
> preserve refactoring with JsonItem introduction, but remove json support.
> If it will be still unacceptable, I will try to completely remove patch #1.
>
>
I have much better feeling from version 45 (now it looks like Postgres C
:)). Still there are some small issues.

1. commented code

+json_encoding:
+<-><--><-->name<--><--><--><--><--><--><--><--><-->{ $$ =
makeJsonEncoding($1); }
+<->/*
+<-><--><-->| UTF8<><--><--><--><--><--><--><--><-->{ $$ = JS_ENC_UTF8; }
+<-><--><-->| UTF16><--><--><--><--><--><--><--><-->{ $$ = JS_ENC_UTF16; }
+<-><--><-->| UTF32 <--><--><--><--><--><--><--><-->{ $$ = JS_ENC_UTF32; }
+<->*/
+<-><-->;

2. sometimes useless empty rows

silent boolean DEFAULT false)
+RETURNS text
+LANGUAGE INTERNAL
+STRICT IMMUTABLE PARALLEL SAFE
+AS 'jsonb_path_query_first_text';
+
+
+

+<-><-->if (!coerced)
+<-><-->{
+
+<-><--><-->/* If coercion failed, use to_json()/to_jsonb() functions. */

All tests passed
build without any problem

looking for next update

Regards

Pavel

> --
> Nikita Glukhov
> Postgres Professional: http://www.postgrespro.com
> The Russian Postgres Company
>

Attached 46th version of the patches.

On 20.03.2020 22:34, Pavel Stehule wrote:
>
> čt 19. 3. 2020 v 23:57 odesílatel Nikita Glukhov
> <n(dot)gluhov(at)postgrespro(dot)ru <mailto:n(dot)gluhov(at)postgrespro(dot)ru>> napsal:
>
> Attached 45th version of the patches.
>
> Nodes JsonFormat, JsonReturning, JsonPassing, JsonBehavior were fixed.
>
> On 17.03.2020 21:35, Pavel Stehule wrote:
>>
>> út 17. 3. 2020 v 1:55 odesílatel Nikita Glukhov
>> <n(dot)gluhov(at)postgrespro(dot)ru <mailto:n(dot)gluhov(at)postgrespro(dot)ru>> napsal:
>>
>> Attached 44th version of the patches.
>>
>>
>> On 12.03.2020 16:41, Pavel Stehule wrote:
>>>
>>> On 12.03.2020 00:09 Nikita Glukhov wrote:
>>>
>>> Attached 43rd version of the patches.
>>>
>>> The previous patch #4 ("Add function formats") was removed.
>>> Instead, introduced new executor node JsonCtorExpr which is used to wrap
>>> SQL/JSON constructor function calls (FuncExpr, Aggref, WindowFunc).
>>>
>>> Also, JsonIsPredicate node began to be used as executor node.
>>> This helped to remove unnecessary json[b]_is_valid() user functions.
>>>
>>>
>>> It looks very well - all tests passed, code looks well.
>>>
>>> Now, when there are special nodes, then the introduction of
>>> COERCE_INTERNAL_CAST is not necessary, and it is only my one
>>> and last objection again this patch's set.
>>>
>> I have removed patch #3 with COERCE_INTERNAL_CAST too.
>>
>> Coercions in JsonValueExpr in JsonCtorExpr, which were previously hidden with
>> COERCE_INTERNAL_CAST and which were outputted as RETURNING or FORMAT JSON
>> clauses, now moved into separate expressions.
>>
>> I am looking on the code, and although the code is correct it
>> doesn't look well (consistently with other node types).
>>
>> I think so JsonFormat and JsonReturning should be node types, not
>> just structs. If these types will be nodes, then you can simplify
>> _readJsonExpr and all node operations on this node.
>
> JsonFormat and JsonReturning was transformed into nodes, and not included
> directly into other nodes now.
>
>
>> User functions json[b]_build_object_ext() and json[b]_build_array_ext() also
>> can be easily removed. But it seems harder to remove new aggregate functions
>> json[b]_objectagg() and json[b]_agg_strict(), because they can't be called
>> directly from JsonCtorExpr node.
>>
>>
>> I don't see reasons for another reduction now. Can be great if
>> you can finalize work what you plan for pg13.
>>
I have removed json[b]_build_object_ext() and json[b]_build_array_ext().

But json[b]_objectagg() and json[b]_agg_strict() are still present.
It seems that removing them requires majors refactoring of the execution
of Aggref and WindowFunc nodes.

>> +<->READ_ENUM_FIELD(on_error.btype, JsonBehaviorType);
>> +<->READ_NODE_FIELD(on_error.default_expr);
>> +<->READ_ENUM_FIELD(on_empty.btype, JsonBehaviorType);
>> +<->READ_NODE_FIELD(on_empty.default_expr);
>>
>> JsonBehavior is node type. Then why you don't write just
>>
>> READ_NODE_FIELD(on_error);
>> READ_NODE_FIELD(on_empty)
>>
>> ??
>
> JsonBehavior now used in JsonExpr as a pointer to node.
>
>
>> And maybe the code can be better if you don't use JsonPassing
>> struct (or use it only inside gram.y) and pass just List *names,
>> List *values.
>>
>> Nodes should to contains another nodes or scalar types. Using
>> structs (that are not nodes)  inside doesn't look consistently.
>>
> JsonPassing was replaced with two Lists.
>
>
>> I found some not finished code in 0003 patch
>> +
>> +json_name_and_value:
>> +/* TODO
>> +<-><--><-->KEY c_expr VALUE_P json_value_expr %prec POSTFIXOP
>> +<-><--><--><-->{ $$ = makeJsonKeyValue($2, $4); }
>> +<-><--><-->|
>> +*/
>>
> This is unsupported variant of standard syntax, because it seems to lead
> to unresolvable conflicts. The only supports syntax is:
>
> JSON_OBJECT(key : value)
> JSON_OBJECT(key VALUE value)
>
> ok. So please change comment from ToDo to this explanation. Maybe note
> in doc (unimplemented features can be good idea)
>
> Some these unresolvable conflicts are solved with extra parenthesis in
> Postgres.
>
>>
>> The support for json type in jsonpath also seems to be immature, so I will try
>> to remove it in the next iteration.
>>
>> What do you think? This patch is little bit off topic, so if you
>> don't need it, then can be removed. Is there some dependency for
>> "jsontable" ?
>
> There is a dependency in SQL/JSON query functions executor, because executor
> uses new structure JsonItem instead of plain JsonbValue. I will try to
> preserve refactoring with JsonItem introduction, but remove json support.
> If it will be still unacceptable, I will try to completely remove patch #1.
>
I have completely removed patch #1. It turned out to be not so difficult.

>
> I have much better feeling from version 45 (now it looks like Postgres
> C :)). Still there are some small issues.
>
> 1. commented code
>
> +json_encoding:
> +<-><--><-->name<--><--><--><--><--><--><--><--><-->{ $$ =
> makeJsonEncoding($1); }
> +<->/*
> +<-><--><-->| UTF8<><--><--><--><--><--><--><--><-->{ $$ = JS_ENC_UTF8; }
> +<-><--><-->| UTF16><--><--><--><--><--><--><--><-->{ $$ = JS_ENC_UTF16; }
> +<-><--><-->| UTF32 <--><--><--><--><--><--><--><-->{ $$ = JS_ENC_UTF32; }
> +<->*/
> +<-><-->;
>
Fixed.

> 2. sometimes useless empty rows
>
>                               silent boolean DEFAULT false)
> +RETURNS text
> +LANGUAGE INTERNAL
> +STRICT IMMUTABLE PARALLEL SAFE
> +AS 'jsonb_path_query_first_text';
> +
> +
> +
>
>
> +<-><-->if (!coerced)
> +<-><-->{
> +
> +<-><--><-->/* If coercion failed, use to_json()/to_jsonb() functions. */

Fixed.

--
Nikita Glukhov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

so 21. 3. 2020 v 11:07 odesílatel Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru>
napsal:

> Attached 46th version of the patches.
>
>
> On 20.03.2020 22:34, Pavel Stehule wrote:
>
>
> čt 19. 3. 2020 v 23:57 odesílatel Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru>
> napsal:
>
>> Attached 45th version of the patches.
>>
>> Nodes JsonFormat, JsonReturning, JsonPassing, JsonBehavior were fixed.
>>
>>
>> On 17.03.2020 21:35, Pavel Stehule wrote:
>>
>>
>> út 17. 3. 2020 v 1:55 odesílatel Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru>
>> napsal:
>>
>>> Attached 44th version of the patches.
>>>
>>>
>>> On 12.03.2020 16:41, Pavel Stehule wrote:
>>>
>>>
>>> On 12.03.2020 00:09 Nikita Glukhov wrote:
>>>
>>>> Attached 43rd version of the patches.
>>>>
>>>> The previous patch #4 ("Add function formats") was removed.
>>>> Instead, introduced new executor node JsonCtorExpr which is used to wrap
>>>> SQL/JSON constructor function calls (FuncExpr, Aggref, WindowFunc).
>>>>
>>>> Also, JsonIsPredicate node began to be used as executor node.
>>>> This helped to remove unnecessary json[b]_is_valid() user functions.
>>>>
>>>>
>>> It looks very well - all tests passed, code looks well.
>>>
>>> Now, when there are special nodes, then the introduction of
>>> COERCE_INTERNAL_CAST is not necessary, and it is only my one and last
>>> objection again this patch's set.
>>>
>>> I have removed patch #3 with COERCE_INTERNAL_CAST too.
>>>
>>> Coercions in JsonValueExpr in JsonCtorExpr, which were previously hidden with
>>> COERCE_INTERNAL_CAST and which were outputted as RETURNING or FORMAT JSON
>>> clauses, now moved into separate expressions.
>>>
>>> I am looking on the code, and although the code is correct it doesn't
>> look well (consistently with other node types).
>>
>> I think so JsonFormat and JsonReturning should be node types, not just
>> structs. If these types will be nodes, then you can simplify _readJsonExpr
>> and all node operations on this node.
>>
>>
>> JsonFormat and JsonReturning was transformed into nodes, and not included
>> directly into other nodes now.
>>
>>
>> User functions json[b]_build_object_ext() and json[b]_build_array_ext() also
>>> can be easily removed. But it seems harder to remove new aggregate functions
>>> json[b]_objectagg() and json[b]_agg_strict(), because they can't be called
>>> directly from JsonCtorExpr node.
>>>
>>>
>> I don't see reasons for another reduction now. Can be great if you can
>> finalize work what you plan for pg13.
>>
>> I have removed json[b]_build_object_ext() and json[b]_build_array_ext().
>
> But json[b]_objectagg() and json[b]_agg_strict() are still present.
> It seems that removing them requires majors refactoring of the execution
> of Aggref and WindowFunc nodes.
>
>
> +<->READ_ENUM_FIELD(on_error.btype, JsonBehaviorType);
>> +<->READ_NODE_FIELD(on_error.default_expr);
>> +<->READ_ENUM_FIELD(on_empty.btype, JsonBehaviorType);
>> +<->READ_NODE_FIELD(on_empty.default_expr);
>>
>> JsonBehavior is node type. Then why you don't write just
>>
>> READ_NODE_FIELD(on_error);
>> READ_NODE_FIELD(on_empty)
>>
>> ??
>>
>> JsonBehavior now used in JsonExpr as a pointer to node.
>>
>>
>> And maybe the code can be better if you don't use JsonPassing struct (or
>> use it only inside gram.y) and pass just List *names, List *values.
>>
>> Nodes should to contains another nodes or scalar types. Using structs
>> (that are not nodes) inside doesn't look consistently.
>>
>> JsonPassing was replaced with two Lists.
>>
>>
>> I found some not finished code in 0003 patch
>> +
>> +json_name_and_value:
>> +/* TODO
>> +<-><--><-->KEY c_expr VALUE_P json_value_expr %prec POSTFIXOP
>> +<-><--><--><-->{ $$ = makeJsonKeyValue($2, $4); }
>> +<-><--><-->|
>> +*/
>>
>> This is unsupported variant of standard syntax, because it seems to lead
>> to unresolvable conflicts. The only supports syntax is:
>>
>> JSON_OBJECT(key : value)
>> JSON_OBJECT(key VALUE value)
>>
>> ok. So please change comment from ToDo to this explanation. Maybe note in
> doc (unimplemented features can be good idea)
>
> Some these unresolvable conflicts are solved with extra parenthesis in
> Postgres.
>
>
>>
>> The support for json type in jsonpath also seems to be immature, so I will try
>>> to remove it in the next iteration.
>>>
>>> What do you think? This patch is little bit off topic, so if you don't
>> need it, then can be removed. Is there some dependency for "jsontable" ?
>>
>> There is a dependency in SQL/JSON query functions executor, because executor
>> uses new structure JsonItem instead of plain JsonbValue. I will try to
>> preserve refactoring with JsonItem introduction, but remove json support.
>> If it will be still unacceptable, I will try to completely remove patch #1.
>>
>> I have completely removed patch #1. It turned out to be not so difficult.
>
>
>
> I have much better feeling from version 45 (now it looks like Postgres C
> :)). Still there are some small issues.
>
> 1. commented code
>
> +json_encoding:
> +<-><--><-->name<--><--><--><--><--><--><--><--><-->{ $$ =
> makeJsonEncoding($1); }
> +<->/*
> +<-><--><-->| UTF8<><--><--><--><--><--><--><--><-->{ $$ = JS_ENC_UTF8; }
> +<-><--><-->| UTF16><--><--><--><--><--><--><--><-->{ $$ = JS_ENC_UTF16; }
> +<-><--><-->| UTF32 <--><--><--><--><--><--><--><-->{ $$ = JS_ENC_UTF32; }
> +<->*/
> +<-><-->;
>
> Fixed.
>
> 2. sometimes useless empty rows
>
> silent boolean DEFAULT false)
> +RETURNS text
> +LANGUAGE INTERNAL
> +STRICT IMMUTABLE PARALLEL SAFE
> +AS 'jsonb_path_query_first_text';
> +
> +
> +
>
>
> +<-><-->if (!coerced)
> +<-><-->{
> +
> +<-><--><-->/* If coercion failed, use to_json()/to_jsonb() functions. */
>
> Fixed.
>
I like this version. I checked code and I don't see any issue. It looks
very well.

The build is without any problems, all tests passed.

I'll mark this patch as ready for commiters.

Thank your good work.

Regards

Pavel

>
> --
> Nikita Glukhov
> Postgres Professional: http://www.postgrespro.com
> The Russian Postgres Company
>

Attached 47th version of the patches.

On 21.03.2020 22:38, Pavel Stehule wrote:
>
> On 21. 3. 2020 v 11:07 Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru
> <mailto:n(dot)gluhov(at)postgrespro(dot)ru>> wrote:
>
> Attached 46th version of the patches.
>
> On 20.03.2020 22:34, Pavel Stehule wrote:
>>
>> On 19.03.2020 23:57 Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru
>> <mailto:n(dot)gluhov(at)postgrespro(dot)ru>> wrote:
>>
>> Attached 45th version of the patches.
>>
>> Nodes JsonFormat, JsonReturning, JsonPassing, JsonBehavior were fixed.
>>
>> On 17.03.2020 21:35, Pavel Stehule wrote:
>>>
>>> User functions json[b]_build_object_ext() and json[b]_build_array_ext() also
>>> can be easily removed. But it seems harder to remove new aggregate functions
>>> json[b]_objectagg() and json[b]_agg_strict(), because they can't be called
>>> directly from JsonCtorExpr node.
>>>
>>>
>>> I don't see reasons for another reduction now. Can be great
>>> if you can finalize work what you plan for pg13.
>>>
> I have removed json[b]_build_object_ext() and json[b]_build_array_ext().
>
> But json[b]_objectagg() and json[b]_agg_strict() are still present.
> It seems that removing them requires majors refactoring of the execution
> of Aggref and WindowFunc nodes.
>
I have replaced aggregate function

json[b]_objectagg(key any, val any, absent_on_null boolean, unique_keys boolean)

with three separate functions:

json[b]_object_agg_strict(any, any)
json[b]_object_agg_unique(any, any)
json[b]_object_agg_unique_strict(any, any)

This should be more correct than single aggregate with additional parameters.

--
Nikita Glukhov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

On Mon, Mar 23, 2020 at 8:28 PM Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru> wrote:
> Attached 47th version of the patches.
>
> On 21.03.2020 22:38, Pavel Stehule wrote:
>
>
> On 21. 3. 2020 v 11:07 Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru> wrote:
>>
>> Attached 46th version of the patches.
>>
>> On 20.03.2020 22:34, Pavel Stehule wrote:
>>
>>
>> On 19.03.2020 23:57 Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru> wrote:
>>>
>>> Attached 45th version of the patches.
>>>
>>> Nodes JsonFormat, JsonReturning, JsonPassing, JsonBehavior were fixed.
>>>
>>> On 17.03.2020 21:35, Pavel Stehule wrote:
>>>>
>>>> User functions json[b]_build_object_ext() and json[b]_build_array_ext() also
>>>> can be easily removed. But it seems harder to remove new aggregate functions
>>>> json[b]_objectagg() and json[b]_agg_strict(), because they can't be called
>>>> directly from JsonCtorExpr node.
>>>
>>>
>>> I don't see reasons for another reduction now. Can be great if you can finalize work what you plan for pg13.
>>>
>> I have removed json[b]_build_object_ext() and json[b]_build_array_ext().
>>
>> But json[b]_objectagg() and json[b]_agg_strict() are still present.
>> It seems that removing them requires majors refactoring of the execution
>> of Aggref and WindowFunc nodes.
>
> I have replaced aggregate function
>
> json[b]_objectagg(key any, val any, absent_on_null boolean, unique_keys boolean)
>
> with three separate functions:
>
> json[b]_object_agg_strict(any, any)
> json[b]_object_agg_unique(any, any)
> json[b]_object_agg_unique_strict(any, any)
>
>
> This should be more correct than single aggregate with additional parameters.

I've following notes about this patchset.

1) Uniqueness checks using JsonbUniqueCheckContext and
JsonUniqueCheckContext have quadratic complexity over number of keys.
That doesn't look good especially for jsonb, which anyway sorts object
keys before object serialization.
2) We have two uniqueness checks for json type, which use
JsonbUniqueCheckContext and JsonUniqueState. JsonUniqueState uses
stack of hashes, while JsonbUniqueCheckContext have just plain array
of keys. I think we can make JsonUniqueState use single hash, where
object identifies would be part of hash key. And we should replace
JsonbUniqueCheckContext with JsonUniqueState. That would eliminate
extra entities and provide reasonable complexity for cases, which now
use JsonbUniqueCheckContext.
3) New SQL/JSON clauses don't use timezone and considered as immutable
assuming all the children are immutable. Immutability is good, but
ignoring timezone in all the cases is plain wrong. The first thing we
can do is to use timezone and make SQL/JSON clauses stable. But that
limits their usage in functional and partial indexes. I see couple of
things we can do next (one of them or probably both).
3.1) Provide user a way so specify that we should ignore timezone in
particular case (IGNORE TIMEZONE clause or something like that). Then
SQL/JSON clause will be considered as immutable.
3.2) Automatically detect whether jsonpath might use timezone. If
jsonpath doesn't use .datetime() method, it doesn't need timezone for
sure. Also, from the datetime format specifiers we can get that we
don't compare non-timezoned values to timezoned values. So, if we
detect this jsonpath never uses timezone, we can consider SQL/JSON
clause as immutable.

------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

On Mon, Mar 23, 2020 at 08:28:52PM +0300, Nikita Glukhov wrote:
> Attached 47th version of the patches.

The patch checker/cfbot says this doesn't apply ; could you send a rebasified
version ?

--
Justin

On 7/5/20 1:29 PM, Justin Pryzby wrote:
> On Mon, Mar 23, 2020 at 08:28:52PM +0300, Nikita Glukhov wrote:
>> Attached 47th version of the patches.
> The patch checker/cfbot says this doesn't apply ; could you send a rebasified
> version ?
>

To keep things moving, I've rebased these patches. However, 1) the docs
patches use <replaceble class="parameter"> in many cases where they
should now just use <parameter> and b) patch 4 fails when run under
force_parallel=regress.

cheers

andrew

--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 7/14/20 1:00 PM, Andrew Dunstan wrote:
> On 7/5/20 1:29 PM, Justin Pryzby wrote:
>> On Mon, Mar 23, 2020 at 08:28:52PM +0300, Nikita Glukhov wrote:
>>> Attached 47th version of the patches.
>> The patch checker/cfbot says this doesn't apply ; could you send a rebasified
>> version ?
>>
> To keep things moving, I've rebased these patches. However, 1) the docs
> patches use <replaceble class="parameter"> in many cases where they
> should now just use <parameter> and b) patch 4 fails when run under
> force_parallel=regress.
>
>

Turns out these patches also need to get the message on the new way of
writing entries in func.sgml - I'll publish some updates on that in the
next day or so so that "make doc" will succeed.

cheers

andrew

--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attached 49th version of the patches with two new patches #5 and #6.

On 15.07.2020 00:09, Andrew Dunstan wrote:
> On 7/14/20 1:00 PM, Andrew Dunstan wrote:
>> On 7/5/20 1:29 PM, Justin Pryzby wrote:
>>> On Mon, Mar 23, 2020 at 08:28:52PM +0300, Nikita Glukhov wrote:
>>>> Attached 47th version of the patches.
>>> The patch checker/cfbot says this doesn't apply ; could you send a rebasified
>>> version ?
>>>
>> To keep things moving, I've rebased these patches. However, 1) the docs
>> patches use <replaceble class="parameter"> in many cases where they
>> should now just use <parameter>

I haven't touched <replaceable class="parameter"> yet, because I'm not sure
if <replaceable> or <parameter> is correct here at all.

> Turns out these patches also need to get the message on the new way of
> writing entries in func.sgml - I'll publish some updates on that in the
> next day or so so that "make doc" will succeed.

I can do it by myself, but I just need to understand what to fix and how.

>> and b) patch 4 fails when run under force_parallel=regress.

Fixed parallel-safety check for RETURNING clause of JSON_EXISTS().

On 05.04.2020 19:50, Alexander Korotkov wrote:
> 1) Uniqueness checks using JsonbUniqueCheckContext and
> JsonUniqueCheckContext have quadratic complexity over number of keys.
> That doesn't look good especially for jsonb, which anyway sorts object
> keys before object serialization.
> 2) We have two uniqueness checks for json type, which use
> JsonbUniqueCheckContext and JsonUniqueState. JsonUniqueState uses
> stack of hashes, while JsonbUniqueCheckContext have just plain array
> of keys. I think we can make JsonUniqueState use single hash, where
> object identifies would be part of hash key. And we should replace
> JsonbUniqueCheckContext with JsonUniqueState. That would eliminate
> extra entities and provide reasonable complexity for cases, which now
> use JsonbUniqueCheckContext.

Unique checks were refactored as Alexander proposed.

> 3) New SQL/JSON clauses don't use timezone and considered as immutable
> assuming all the children are immutable. Immutability is good, but
> ignoring timezone in all the cases is plain wrong. The first thing we
> can do is to use timezone and make SQL/JSON clauses stable. But that
> limits their usage in functional and partial indexes. I see couple of
> things we can do next (one of them or probably both).
> 3.1) Provide user a way so specify that we should ignore timezone in
> particular case (IGNORE TIMEZONE clause or something like that). Then
> SQL/JSON clause will be considered as immutable.
> 3.2) Automatically detect whether jsonpath might use timezone. If
> jsonpath doesn't use .datetime() method, it doesn't need timezone for
> sure. Also, from the datetime format specifiers we can get that we
> don't compare non-timezoned values to timezoned values. So, if we
> detect this jsonpath never uses timezone, we can consider SQL/JSON
> clause as immutable.

Implemented second variant with automatic detection.

I also tried to add explicit IGNORE TIMEZONE / IMMUTABLE clauses, but all of
them lead to shift/reduce conflicts that seem not easy to resolve.

Patch #5 implements functions for new JSON type that is expected to appear in
the upcoming SQL/JSON standard:

- JSON() is for constructing JSON typed values from JSON text.
It is almost equivalent to text::json[b] cast, except that it has additional
ability to specify WITH UNIQUE KEYS constraint.

- JSON_SCALAR() is for constructing JSON typed values from SQL scalars.
It is equivalent to to_json[b]().

- JSON_SERIALIZE() is for serializing JSON typed values to character strings.
It is almost equivalent to json[b]::character_type cast, but it also
supports output to bytea.

Upcoming Oracle 20c will have JSON datatype and these functions [1], so we
decided also to implement them for compatibility, despite that they do not make
sense for real PG users.

Patch #6 allows the user to use PG jsonb type as an effective implementation of
SQL JSON type. By explicitly setting GUC sql_json = jsonb, JSON will be mapped
to jsonb, and JSON TEXT (may be named named differently) will be mapped to json.

This seems to be a hack, but we failed to propose something more simpler.

Example of usage GUC sql_json:

=# CREATE TABLE test1 (js json, jb jsonb, jt json text);
CREATE TABLE

=# \d test1
Table "public.test1"
Column | Type | Collation | Nullable | Default
--------+-------+-----------+----------+---------
js | json | | |
jb | jsonb | | |
jt | json | | |

=# SET sql_json = jsonb;
SET

=# \d test1
Table "public.test1"
Column | Type | Collation | Nullable | Default
--------+-----------+-----------+----------+---------
js | json text | | |
jb | json | | |
jt | json text | | |

=# CREATE TABLE test2 (js json, jb jsonb, jt json text);
CREATE TABLE

=# \d test2
Table "public.test2"
Column | Type | Collation | Nullable | Default
--------+-----------+-----------+----------+---------
js | json | | |
jb | json | | |
jt | json text | | |

=# SET sql_json = json;
SET
=# \d test2
Table "public.test2"
Column | Type | Collation | Nullable | Default
--------+-------+-----------+----------+---------
js | jsonb | | |
jb | jsonb | | |
jt | json | | |

[1] https://docs.oracle.com/en/database/oracle/oracle-database/20/adjsn/json-in-oracle-database.html#GUID-CBEDC779-39A3-43C9-AF38-861AE3FC0AEC

--
Nikita Glukhov
Postgres Professional:http://www.postgrespro.com
The Russian Postgres Company

On 7/14/20 9:47 PM, Nikita Glukhov wrote:
> Attached 49th version of the patches with two new patches #5 and #6.
>
> On 15.07.2020 00:09, Andrew Dunstan wrote:
>> On 7/14/20 1:00 PM, Andrew Dunstan wrote:
>>> On 7/5/20 1:29 PM, Justin Pryzby wrote:
>>>> On Mon, Mar 23, 2020 at 08:28:52PM +0300, Nikita Glukhov wrote:
>>>>> Attached 47th version of the patches.
>>>> The patch checker/cfbot says this doesn't apply ; could you send a rebasified
>>>> version ?
>>>>
>>> To keep things moving, I've rebased these patches. However, 1) the docs
>>> patches use <replaceble class="parameter"> in many cases where they
>>> should now just use <parameter>
> I haven't touched <replaceable class="parameter"> yet, because I'm not sure
> if <replaceable> or <parameter> is correct here at all.

Here's the relevant commit message that explains the policy:

commit 47046763c3
Author: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Date:   Mon May 4 13:48:30 2020 -0400

    Doc: standardize markup a bit more.
   
    We had a mishmash of <replaceable>, <replaceable class="parameter">,
    and <parameter> markup for operator/function arguments.  Use <parameter>
    consistently for things that are in fact names of parameters (including
    OUT parameters), reserving <replaceable> for things that aren't.  The
    latter class includes some made-up-by-the-docs type class names, like
    "numeric_type", as well as placeholders for arguments that don't have
    well-defined types.  Possibly we could do better with those categories
    as well, but for the moment I'm content not to have parameter names
    marked up in different ways in different places.

>
>> Turns out these patches also need to get the message on the new way of
>> writing entries in func.sgml - I'll publish some updates on that in the
>> next day or so so that "make doc" will succeed.
> I can do it by myself, but I just need to understand what to fix and how.

Here's an example that shows how it was transformed for json_agg:

-     <row>
-      <entry>
-       <indexterm>
-        <primary>json_agg</primary>
-       </indexterm>
-       <function>json_agg(<replaceable
class="parameter">expression</replaceable>)</function>
-      </entry>
-      <entry>
-       <type>any</type>
-      </entry>
-      <entry>
-       <type>json</type>
-      </entry>
-      <entry>No</entry>
-      <entry>aggregates values, including nulls, as a JSON array</entry>
-     </row>
+      <row>
+       <entry role="func_table_entry"><para role="func_signature">
+        <indexterm>
+         <primary>json_agg</primary>
+        </indexterm>
+        <function>json_agg</function> ( <type>anyelement</type> )
+        <returnvalue>json</returnvalue>
+       </para>
+       <para role="func_signature">
+        <indexterm>
+         <primary>jsonb_agg</primary>
+        </indexterm>
+        <function>jsonb_agg</function> ( <type>anyelement</type> )
+        <returnvalue>jsonb</returnvalue>
+       </para>
+       <para>
+        Collects all the input values, including nulls, into a JSON array.
+        Values are converted to JSON as per <function>to_json</function>
+        or <function>to_jsonb</function>.
+       </para></entry>
+       <entry>No</entry>
+      </row>
 

>>> and b) patch 4 fails when run under force_parallel=regress.
> Fixed parallel-safety check for RETURNING clause of JSON_EXISTS().
>
> On 05.04.2020 19:50, Alexander Korotkov wrote:
> > 1) Uniqueness checks using JsonbUniqueCheckContext and
> > JsonUniqueCheckContext have quadratic complexity over number of keys.
> > That doesn't look good especially for jsonb, which anyway sorts object
> > keys before object serialization.
> > 2) We have two uniqueness checks for json type, which use
> > JsonbUniqueCheckContext and JsonUniqueState. JsonUniqueState uses
> > stack of hashes, while JsonbUniqueCheckContext have just plain array
> > of keys. I think we can make JsonUniqueState use single hash, where
> > object identifies would be part of hash key. And we should replace
> > JsonbUniqueCheckContext with JsonUniqueState. That would eliminate
> > extra entities and provide reasonable complexity for cases, which now
> > use JsonbUniqueCheckContext.
>
> Unique checks were refactored as Alexander proposed.
>
> > 3) New SQL/JSON clauses don't use timezone and considered as immutable
> > assuming all the children are immutable. Immutability is good, but
> > ignoring timezone in all the cases is plain wrong. The first thing we
> > can do is to use timezone and make SQL/JSON clauses stable. But that
> > limits their usage in functional and partial indexes. I see couple of
> > things we can do next (one of them or probably both).
> > 3.1) Provide user a way so specify that we should ignore timezone in
> > particular case (IGNORE TIMEZONE clause or something like that). Then
> > SQL/JSON clause will be considered as immutable.
> > 3.2) Automatically detect whether jsonpath might use timezone. If
> > jsonpath doesn't use .datetime() method, it doesn't need timezone for
> > sure. Also, from the datetime format specifiers we can get that we
> > don't compare non-timezoned values to timezoned values. So, if we
> > detect this jsonpath never uses timezone, we can consider SQL/JSON
> > clause as immutable.
>
> Implemented second variant with automatic detection.
>
> I also tried to add explicit IGNORE TIMEZONE / IMMUTABLE clauses, but all of
> them lead to shift/reduce conflicts that seem not easy to resolve.

Not sure if this helps:

https://wiki.postgresql.org/wiki/Debugging_the_PostgreSQL_grammar_(Bison)

>
>
>
> Patch #5 implements functions for new JSON type that is expected to appear in
> the upcoming SQL/JSON standard:
>
> - JSON() is for constructing JSON typed values from JSON text.
> It is almost equivalent to text::json[b] cast, except that it has additional
> ability to specify WITH UNIQUE KEYS constraint.
>
> - JSON_SCALAR() is for constructing JSON typed values from SQL scalars.
> It is equivalent to to_json[b]().
>
> - JSON_SERIALIZE() is for serializing JSON typed values to character strings.
> It is almost equivalent to json[b]::character_type cast, but it also
> supports output to bytea.
>
> Upcoming Oracle 20c will have JSON datatype and these functions [1], so we
> decided also to implement them for compatibility, despite that they do not make
> sense for real PG users.
>

Are these functions in the standard, or are they Oracle extensions? If
the latter maybe they belong in an options extension.

>
> Patch #6 allows the user to use PG jsonb type as an effective implementation of
> SQL JSON type. By explicitly setting GUC sql_json = jsonb, JSON will be mapped
> to jsonb, and JSON TEXT (may be named named differently) will be mapped to json.
>
> This seems to be a hack, but we failed to propose something more simpler.

What is going to be the effect of that on things like index expressions?
This strikes me at first glance as something of a potential footgun, but
maybe I'm being overcautious.

cheers

andrew

--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attached 50th version of the patches. Only the documentation was changed
since the previous version.

On 15.07.2020 14:50, Andrew Dunstan wrote:

> On 7/14/20 9:47 PM, Nikita Glukhov wrote:
>
> On 15.07.2020 00:09, Andrew Dunstan wrote:
>>> On 7/14/20 1:00 PM, Andrew Dunstan wrote:
>>>> To keep things moving, I've rebased these patches. However, 1) the docs
>>>> patches use <replaceble class="parameter"> in many cases where they
>>>> should now just use <parameter>
>> I haven't touched <replaceable class="parameter"> yet, because I'm not sure
>> if <replaceable> or <parameter> is correct here at all.
> Here's the relevant commit message that explains the policy:
>
> commit 47046763c3
> Author: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
> Date:   Mon May 4 13:48:30 2020 -0400
>
>     Doc: standardize markup a bit more.
>
>     We had a mishmash of <replaceable>, <replaceable class="parameter">,
>     and <parameter> markup for operator/function arguments.  Use <parameter>
>     consistently for things that are in fact names of parameters (including
>     OUT parameters), reserving <replaceable> for things that aren't.  The
>     latter class includes some made-up-by-the-docs type class names, like
>     "numeric_type", as well as placeholders for arguments that don't have
>     well-defined types.  Possibly we could do better with those categories
>     as well, but for the moment I'm content not to have parameter names
>     marked up in different ways in different places.
>>> Turns out these patches also need to get the message on the new way of
>>> writing entries in func.sgml - I'll publish some updates on that in the
>>> next day or so so that "make doc" will succeed.
>> I can do it by myself, but I just need to understand what to fix and how.

Ok, I replaced some <replaceable> with <parameter> in SQL/JSON constructors.
Also I replaced all '[]' with <optional>.

>> Patch #5 implements functions for new JSON type that is expected to appear in
>> the upcoming SQL/JSON standard:
>>
>> - JSON() is for constructing JSON typed values from JSON text.
>> It is almost equivalent to text::json[b] cast, except that it has additional
>> ability to specify WITH UNIQUE KEYS constraint.
>>
>> - JSON_SCALAR() is for constructing JSON typed values from SQL scalars.
>> It is equivalent to to_json[b]().
>>
>> - JSON_SERIALIZE() is for serializing JSON typed values to character strings.
>> It is almost equivalent to json[b]::character_type cast, but it also
>> supports output to bytea.
>>
>> Upcoming Oracle 20c will have JSON datatype and these functions [1], so we
>> decided also to implement them for compatibility, despite that they do not make
>> sense for real PG users.
> Are these functions in the standard, or are they Oracle extensions? If
> the latter maybe they belong in an options extension.

The new SQL type JSON and these functions are in to the new standard SQL/JSON 2.0.
It is at the proposal stage now, but Oracle 20c has already implemented it.

The document is not publicly available, so Oleg should have sent it to you in a
private message.

>> Patch #6 allows the user to use PG jsonb type as an effective implementation of
>> SQL JSON type. By explicitly setting GUC sql_json = jsonb, JSON will be mapped
>> to jsonb, and JSON TEXT (may be named named differently) will be mapped to json.
>>
>> This seems to be a hack, but we failed to propose something more simpler.
> What is going to be the effect of that on things like index expressions?
> This strikes me at first glance as something of a potential footgun, but
> maybe I'm being overcautious.

This allows users of 'sql_json=jsonb' to create GIN indexes on SQL type JSON
(but such creation indexes are still not SQL standard conforming). Maybe I do
not correctly understand the question or the consequences of type name
rewriting.

The type names are rewritten on the input at the initial parsing stage and on
the output in format_type_be(), like it is done for "timestamp with timezone" =>
timestamptz, integer => int4. Internal representation of query expressions
remains the same. Affected only representation of JSON types to/from user.

--
Nikita Glukhov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

On 7/17/20 4:26 PM, Nikita Glukhov wrote:
>
>>> Patch #5 implements functions for new JSON type that is expected to appear in
>>> the upcoming SQL/JSON standard:
>>>
>>> - JSON() is for constructing JSON typed values from JSON text.
>>> It is almost equivalent to text::json[b] cast, except that it has additional
>>> ability to specify WITH UNIQUE KEYS constraint.
>>>
>>> - JSON_SCALAR() is for constructing JSON typed values from SQL scalars.
>>> It is equivalent to to_json[b]().
>>>
>>> - JSON_SERIALIZE() is for serializing JSON typed values to character strings.
>>> It is almost equivalent to json[b]::character_type cast, but it also
>>> supports output to bytea.
>>>
>>> Upcoming Oracle 20c will have JSON datatype and these functions [1], so we
>>> decided also to implement them for compatibility, despite that they do not make
>>> sense for real PG users.
>> Are these functions in the standard, or are they Oracle extensions? If
>> the latter maybe they belong in an options extension.
> The new SQL type JSON and these functions are in to the new standard SQL/JSON 2.0.
> It is at the proposal stage now, but Oracle 20c has already implemented it.
>
> The document is not publicly available, so Oleg should have sent it to you in a
> private message.
>
>>> Patch #6 allows the user to use PG jsonb type as an effective implementation of
>>> SQL JSON type. By explicitly setting GUC sql_json = jsonb, JSON will be mapped
>>> to jsonb, and JSON TEXT (may be named named differently) will be mapped to json.
>>>
>>> This seems to be a hack, but we failed to propose something more simpler.
>> What is going to be the effect of that on things like index expressions?
>> This strikes me at first glance as something of a potential footgun, but
>> maybe I'm being overcautious.
> This allows users of 'sql_json=jsonb' to create GIN indexes on SQL type JSON
> (but such creation indexes are still not SQL standard conforming). Maybe I do
> not correctly understand the question or the consequences of type name
> rewriting.
>
> The type names are rewritten on the input at the initial parsing stage and on
> the output in format_type_be(), like it is done for "timestamp with timezone" =>
> timestamptz, integer => int4. Internal representation of query expressions
> remains the same. Affected only representation of JSON types to/from user.
>

I think patches 5 and 6 need to be submitted to the next commitfest,
This is far too much scope creep to be snuck in under the current CF item.

I'll look at patches 1-4.

cheers

andrew

--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Sat, Jul 18, 2020 at 09:24:11AM -0400, Andrew Dunstan wrote:
> I think patches 5 and 6 need to be submitted to the next commitfest,
> This is far too much scope creep to be snuck in under the current CF item.
>
> I'll look at patches 1-4.

Even with that, the patch set has been waiting on author for the last
six weeks, so I am marking it as RwF for now. Please feel free to
resubmit.
--
Michael

On Fri, 17 Jul 2020 at 21:26, Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru> wrote:
>
> Attached 50th version of the patches. Only the documentation was changed
> since the previous version.

I can imagine the effort required to get to v50, so I salute your efforts.

The document for SQL Standard has now been published as CD
9075-2-Foundation (Thanks Peter).

That gives us a clearer picture of what is being voted on and should
allow Nikita to complete his work.

I suggest we move forwards on this now, but if anyone objects to
including this in PG14 in favour of waiting for the vote, please say
so clearly so we can skip to PG15.

--
Simon Riggs http://www.EnterpriseDB.com/

út 15. 12. 2020 v 18:00 odesílatel Simon Riggs <simon(at)2ndquadrant(dot)com>
napsal:

> On Fri, 17 Jul 2020 at 21:26, Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru>
> wrote:
> >
> > Attached 50th version of the patches. Only the documentation was changed
> > since the previous version.
>
> I can imagine the effort required to get to v50, so I salute your efforts.
>
> The document for SQL Standard has now been published as CD
> 9075-2-Foundation (Thanks Peter).
>
> That gives us a clearer picture of what is being voted on and should
> allow Nikita to complete his work.
>
> I suggest we move forwards on this now, but if anyone objects to
> including this in PG14 in favour of waiting for the vote, please say
> so clearly so we can skip to PG15.
>

Maybe this set of patches can be reorganized and divided. Some parts like
json generating functions are almost trivial and without controversions
with clean benefits for users.

The most complexity is related to json_table function. Nikita did a very
good job and implemented this function in maximal conformance with ANSI SQL
with a maximal set of features. On second hand it is hard to do review
because this patch is really complex, and a lot of functionality was not
implemented elsewhere (so isn't possible to compare results). I think it
should be possible to reduce complexity and divide acceptance of json_table
to some steps like basic functionality (on MySQL level), enhanced
functionality (on Oracle level), and full functionality (the Postgres will
be first). This functionality is interesting and maximal conformity with
SQL/JSON is great so I am for merging it. But if it will be divided into
some chronological steps, then there can be higher probability of merging
to upstream in a good time.

Regards

Pavel

> --
> Simon Riggs http://www.EnterpriseDB.com/
>

On Tue, Dec 15, 2020 at 8:01 PM Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
>
> On Fri, 17 Jul 2020 at 21:26, Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru> wrote:
> >
> > Attached 50th version of the patches. Only the documentation was changed
> > since the previous version.
>
> I can imagine the effort required to get to v50, so I salute your efforts.
>
> The document for SQL Standard has now been published as CD
> 9075-2-Foundation (Thanks Peter).

SQL-2016 is already 4 years old and this what we are using for our development.
What is CD 9075-2-Foundation ?

>
> That gives us a clearer picture of what is being voted on and should
> allow Nikita to complete his work.
>
> I suggest we move forwards on this now, but if anyone objects to
> including this in PG14 in favour of waiting for the vote, please say
> so clearly so we can skip to PG15.

There is upcoming JSON v2. SQL Standard, which specifies JSON data type,
that mean we have to protect our users, or SQL-compliant applications will be
slow and developers will be disappointed. I discussed all these in my talk
at Postgres Build
http://www.sai.msu.su/~megera/postgres/talks/json-build-2020.pdf
Andrew Dunstan has volunteered to work with us on completing SQL/JSON standard,
but ...

I think we need SQL/JSON functions in PG14 and SQL/JSON compatibility mode for
json/jsonb.

>
> --
> Simon Riggs http://www.EnterpriseDB.com/

--
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

On Tue, Dec 15, 2020 at 8:37 PM Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com> wrote:
>
>
>
> út 15. 12. 2020 v 18:00 odesílatel Simon Riggs <simon(at)2ndquadrant(dot)com> napsal:
>>
>> On Fri, 17 Jul 2020 at 21:26, Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru> wrote:
>> >
>> > Attached 50th version of the patches. Only the documentation was changed
>> > since the previous version.
>>
>> I can imagine the effort required to get to v50, so I salute your efforts.
>>
>> The document for SQL Standard has now been published as CD
>> 9075-2-Foundation (Thanks Peter).
>>
>> That gives us a clearer picture of what is being voted on and should
>> allow Nikita to complete his work.
>>
>> I suggest we move forwards on this now, but if anyone objects to
>> including this in PG14 in favour of waiting for the vote, please say
>> so clearly so we can skip to PG15.
>
>
> Maybe this set of patches can be reorganized and divided. Some parts like json generating functions are almost trivial and without controversions with clean benefits for users.

I agree with this, most interesting is JSON_TABLE.
>
> The most complexity is related to json_table function. Nikita did a very good job and implemented this function in maximal conformance with ANSI SQL with a maximal set of features. On second hand it is hard to do review because this patch is really complex, and a lot of functionality was not implemented elsewhere (so isn't possible to compare results). I think it should be possible to reduce complexity and divide acceptance of json_table to some steps like basic functionality (on MySQL level), enhanced functionality (on Oracle level), and full functionality (the Postgres will be first). This functionality is interesting and maximal conformity with SQL/JSON is great so I am for merging it. But if it will be divided into some chronological steps, then there can be higher probability of merging to upstream in a good time.

I think it's shame to look up on MySQL and Oracle, since we have much
better and complete implementation of the Standard.

>
> Regards
>
> Pavel
>
>>
>> --
>> Simon Riggs http://www.EnterpriseDB.com/

--
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

út 15. 12. 2020 v 19:56 odesílatel Oleg Bartunov <obartunov(at)postgrespro(dot)ru>
napsal:

> On Tue, Dec 15, 2020 at 8:37 PM Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
> wrote:
> >
> >
> >
> > út 15. 12. 2020 v 18:00 odesílatel Simon Riggs <simon(at)2ndquadrant(dot)com>
> napsal:
> >>
> >> On Fri, 17 Jul 2020 at 21:26, Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru>
> wrote:
> >> >
> >> > Attached 50th version of the patches. Only the documentation was
> changed
> >> > since the previous version.
> >>
> >> I can imagine the effort required to get to v50, so I salute your
> efforts.
> >>
> >> The document for SQL Standard has now been published as CD
> >> 9075-2-Foundation (Thanks Peter).
> >>
> >> That gives us a clearer picture of what is being voted on and should
> >> allow Nikita to complete his work.
> >>
> >> I suggest we move forwards on this now, but if anyone objects to
> >> including this in PG14 in favour of waiting for the vote, please say
> >> so clearly so we can skip to PG15.
> >
> >
> > Maybe this set of patches can be reorganized and divided. Some parts
> like json generating functions are almost trivial and without
> controversions with clean benefits for users.
>
> I agree with this, most interesting is JSON_TABLE.
>

It is very interesting, but it is very complex too. There is not any
similarly complex function in ANSI SQL. This function defines its own
language.

>
> > The most complexity is related to json_table function. Nikita did a very
> good job and implemented this function in maximal conformance with ANSI SQL
> with a maximal set of features. On second hand it is hard to do review
> because this patch is really complex, and a lot of functionality was not
> implemented elsewhere (so isn't possible to compare results). I think it
> should be possible to reduce complexity and divide acceptance of json_table
> to some steps like basic functionality (on MySQL level), enhanced
> functionality (on Oracle level), and full functionality (the Postgres will
> be first). This functionality is interesting and maximal conformity with
> SQL/JSON is great so I am for merging it. But if it will be divided into
> some chronological steps, then there can be higher probability of merging
> to upstream in a good time.
>
> I think it's shame to look up on MySQL and Oracle, since we have much
> better and complete implementation of the Standard.
>

Maybe I used bad words. I would not reduce json_table patch to MySQL or
Oracle level. I proposed merging this patch in a few steps when any step
can be functional.

Standard divides JSON supports to some levels too.

There is about 50% code (and features) when review is not a problem,
because this functionality is very clean and natural. Another 50% can be
possibly problematic because Nikita implementation is first in the world
and the description in standard is complex and hard to read, and hard to
test. Because these patches are not divided we have to do lot of repeated
work in every cycle. I proposed to do work more in style step by step than
in big bang style.

I think there is a lot of code that can be commitable immediately (maybe
half). This can be done quickly without any controversies. This reduces
complexity for 50% so we can concentrate better on the rest of patches.
The final target is full support of standard and full merge of Nikita's
patches. Nikita did hard and good work and it is nonsense to throw away any
part of his work - and it is a pity so the merging process is too long
already . But I understand it is pretty hard to commit to this patch in
complexity like this patch has.

>
>
>
> >
> > Regards
> >
> > Pavel
> >
> >>
> >> --
> >> Simon Riggs http://www.EnterpriseDB.com/
>
>
>
> --
> Postgres Professional: http://www.postgrespro.com
> The Russian Postgres Company
>

On 17.09.2020 08:41, Michael Paquier wrote:

> On Sat, Jul 18, 2020 at 09:24:11AM -0400, Andrew Dunstan wrote:
>> I think patches 5 and 6 need to be submitted to the next commitfest,
>> This is far too much scope creep to be snuck in under the current CF item.
>>
>> I'll look at patches 1-4.
> Even with that, the patch set has been waiting on author for the last
> six weeks, so I am marking it as RwF for now. Please feel free to
> resubmit.

Attached 51st version of the patches rebased onto current master.

There were some shift/reduce conflicts in SQL grammar that have appeared
after "expr AS keyword" refactoring in 06a7c3154f. I'm not sure if I resolved
them correctly. JSON TEXT pseudotype, introduced in #0006, caused a lot of
grammar conflicts, so it was replaced with simple explicit pg_catalog.json.

Also new CoercionForm COERCE_SQL_SYNTAX was introduced, and this reminds custom
function formats that I have used in earlier version of the patches for
deparsing of SQL/JSON constructor expressions that were based on raw json[b]
function calls. These custom function formats were replaced in v43 with
dedicated executor nodes for SQL/JSON constructors. So, I'm not sure is it
worth to try to replace back nodes with new COERCE_SQL_SYNTAX.

--
Nikita Glukhov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

For 0001-Common-SQL-JSON-clauses-v51.patch :

+ /* | implementation_defined_JSON_representation_option (BSON, AVRO
etc) */

I don't find implementation_defined_JSON_representation_option in the
patchset. Maybe rephrase the above as a comment
without implementation_defined_JSON_representation_option ?

For getJsonEncodingConst(), should the method error out for the default
case of switch (encoding) ?

0002-SQL-JSON-constructors-v51.patch :

+ Assert(!OidIsValid(collation)); /* result is always an
json[b] type */

an json -> a json

+ /* XXX TEXTOID is default by standard */
+ returning->typid = JSONOID;

Comment doesn't seem to match the assignment.

For json_object_agg_transfn :

+ if (out->len > 2)
+ appendStringInfoString(out, ", ");

Why length needs to be at least 3 (instead of 2) ?

Cheers

On Fri, Dec 25, 2020 at 12:26 PM Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru>
wrote:

> On 17.09.2020 08:41, Michael Paquier wrote:
>
> On Sat, Jul 18, 2020 at 09:24:11AM -0400, Andrew Dunstan wrote:
>
> I think patches 5 and 6 need to be submitted to the next commitfest,
> This is far too much scope creep to be snuck in under the current CF item.
>
> I'll look at patches 1-4.
>
> Even with that, the patch set has been waiting on author for the last
> six weeks, so I am marking it as RwF for now. Please feel free to
> resubmit.
>
> Attached 51st version of the patches rebased onto current master.
>
>
> There were some shift/reduce conflicts in SQL grammar that have appeared
> after "expr AS keyword" refactoring in 06a7c3154f. I'm not sure if I resolved
> them correctly. JSON TEXT pseudotype, introduced in #0006, caused a lot of
> grammar conflicts, so it was replaced with simple explicit pg_catalog.json.
>
> Also new CoercionForm COERCE_SQL_SYNTAX was introduced, and this reminds custom
> function formats that I have used in earlier version of the patches for
> deparsing of SQL/JSON constructor expressions that were based on raw json[b]
> function calls. These custom function formats were replaced in v43 with
> dedicated executor nodes for SQL/JSON constructors. So, I'm not sure is it
> worth to try to replace back nodes with new COERCE_SQL_SYNTAX.
>
> --
> Nikita Glukhov
> Postgres Professional: http://www.postgrespro.com
> The Russian Postgres Company
>
>

Hi,
For ExecEvalJsonExprSubtrans(), if you check !subtrans first,

+ /* No need to use subtransactions. */
+ return func(op, econtext, res, resnull, p, error);

The return statement would allow omitting the else keyword and left-indent
the code in the current if block.

For ExecEvalJsonExpr()

+ *resnull = !DatumGetPointer(res);
+ if (error && *error)
+ return (Datum) 0;

Suppose *resnull is false and *error is true, 0 would be returned
with *resnull as false. Should the *resnull be consistent with the actual
return value ?

For ExecEvalJson() :

+ Assert(*op->resnull);
+ *op->resnull = true;

I am not sure of the purpose for the assignment since *op->resnull should
be true by the assertion.

For raw_expression_tree_walker :

+ if (walker(jfe->on_empty, context))
+ return true;

Should the if condition include jfe->on_empty prior to walking ?

nit: for contain_mutable_functions_walker, if !IsA(jexpr->path_spec, Const)
is checked first (and return), the current if block can be left indented.

For JsonPathDatatypeStatus,

+ jpdsDateTime, /* unknown datetime type */

Should the enum be named jpdsUnknownDateTime so that its meaning is clear
to people reading the code ?

For get_json_behavior(), I wonder if mapping from behavior->btype to the
string form would shorten the body of switch statement.
e.g.
char* map[] = {
" NULL",
" ERROR",
" EMPTY",
...
};

Cheers

On Fri, Dec 25, 2020 at 5:19 PM Zhihong Yu <zyu(at)yugabyte(dot)com> wrote:

> For 0001-Common-SQL-JSON-clauses-v51.patch :
>
> + /* | implementation_defined_JSON_representation_option (BSON,
> AVRO etc) */
>
> I don't find implementation_defined_JSON_representation_option in the
> patchset. Maybe rephrase the above as a comment
> without implementation_defined_JSON_representation_option ?
>
> For getJsonEncodingConst(), should the method error out for the default
> case of switch (encoding) ?
>
> 0002-SQL-JSON-constructors-v51.patch :
>
> + Assert(!OidIsValid(collation)); /* result is always an
> json[b] type */
>
> an json -> a json
>
> + /* XXX TEXTOID is default by standard */
> + returning->typid = JSONOID;
>
> Comment doesn't seem to match the assignment.
>
> For json_object_agg_transfn :
>
> + if (out->len > 2)
> + appendStringInfoString(out, ", ");
>
> Why length needs to be at least 3 (instead of 2) ?
>
> Cheers
>
> On Fri, Dec 25, 2020 at 12:26 PM Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru>
> wrote:
>
>> On 17.09.2020 08:41, Michael Paquier wrote:
>>
>> On Sat, Jul 18, 2020 at 09:24:11AM -0400, Andrew Dunstan wrote:
>>
>> I think patches 5 and 6 need to be submitted to the next commitfest,
>> This is far too much scope creep to be snuck in under the current CF item.
>>
>> I'll look at patches 1-4.
>>
>> Even with that, the patch set has been waiting on author for the last
>> six weeks, so I am marking it as RwF for now. Please feel free to
>> resubmit.
>>
>> Attached 51st version of the patches rebased onto current master.
>>
>>
>> There were some shift/reduce conflicts in SQL grammar that have appeared
>> after "expr AS keyword" refactoring in 06a7c3154f. I'm not sure if I resolved
>> them correctly. JSON TEXT pseudotype, introduced in #0006, caused a lot of
>> grammar conflicts, so it was replaced with simple explicit pg_catalog.json.
>>
>> Also new CoercionForm COERCE_SQL_SYNTAX was introduced, and this reminds custom
>> function formats that I have used in earlier version of the patches for
>> deparsing of SQL/JSON constructor expressions that were based on raw json[b]
>> function calls. These custom function formats were replaced in v43 with
>> dedicated executor nodes for SQL/JSON constructors. So, I'm not sure is it
>> worth to try to replace back nodes with new COERCE_SQL_SYNTAX.
>>
>> --
>> Nikita Glukhov
>> Postgres Professional: http://www.postgrespro.com
>> The Russian Postgres Company
>>
>>

Hi, Zhihong. Thank your for your review. Attached 52nd version of the
patches rebased onto master and fixed as you suggested.

On 26.12.2020 04:19, Zhihong Yu wrote:
> For 0001-Common-SQL-JSON-clauses-v51.patch :
>
> +       /*  | implementation_defined_JSON_representation_option (BSON,
> AVRO etc) */
>
> I don't find implementation_defined_JSON_representation_option in the
> patchset. Maybe rephrase the above as a comment
> without implementation_defined_JSON_representation_option ?
>
Fixed.

> For getJsonEncodingConst(), should the method error out for the
> default case of switch (encoding) ?

Added default case with elog(ERROR).

>
> 0002-SQL-JSON-constructors-v51.patch :
>
> +                   Assert(!OidIsValid(collation)); /* result is
> always an json[b] type */
> an json -> a json

Fixed.

>
> +           /* XXX TEXTOID is default by standard */
> +           returning->typid = JSONOID;
>
> Comment doesn't seem to match the assignment.

Comment corrected.

> For json_object_agg_transfn :
>
> +       if (out->len > 2)
> +           appendStringInfoString(out, ", ");
>
> Why length needs to be at least 3 (instead of 2) ?

2 is a length of starting string "{ ".  len > 2 means that we have already
outputted some fields and we need to output comma delimiter.

On 26.12.2020 22:12, Zhihong Yu wrote:
> Hi,
> For ExecEvalJsonExprSubtrans(), if you check !subtrans first,
>
> +       /* No need to use subtransactions. */
> +       return func(op, econtext, res, resnull, p, error);
>
> The return statement would allow omitting the else keyword and
> left-indent the code in the current if block.

"If" statement was refactored as you suggest.

> For ExecEvalJsonExpr()
>
> +           *resnull = !DatumGetPointer(res);
> +           if (error && *error)
> +               return (Datum) 0;
>
> Suppose *resnull is false and *error is true, 0 would be returned
> with *resnull as false. Should the *resnull be consistent with the
> actual return value ?

Now *resnull is set to false in case of error.

> For ExecEvalJson() :
>
> +       Assert(*op->resnull);
> +       *op->resnull = true;
>
> I am not sure of the purpose for the assignment since *op->resnull
> should be true by the assertion.

Assignment was removed.

> For raw_expression_tree_walker :
>
> +               if (walker(jfe->on_empty, context))
> +                   return true;
>
> Should the if condition include jfe->on_empty prior to walking ?
>
Yes, jfe->on_empty like jfe->on_error can be NULL, and NULL check here is
a walker's responsibility. But in expression_tree_walker() there is a check
for jfe->on_empty, because only on_empty (not jfe->on_error) can be NULL, and
we are calling walker() on jfe->on_empty->default_expr, not on jfe->on_empty.

>
> nit: for contain_mutable_functions_walker, if !IsA(jexpr->path_spec,
> Const) is checked first (and return), the current if block can be left
> indented.

Code was refactored as you suggested.

> For JsonPathDatatypeStatus,
>
> +   jpdsDateTime,               /* unknown datetime type */
>
> Should the enum be named jpdsUnknownDateTime so that its meaning is
> clear to people reading the code ?

jpdsDateTime was renamed to jpdsUnknownDateTime.

>
> For get_json_behavior(), I wonder if mapping from behavior->btype to
> the string form would shorten the body of switch statement.
> e.g.
> char* map[] = {
>   " NULL",
>   " ERROR",
>   " EMPTY",
> ...
> };
>
"Switch" statement was replaced with array lookup.

--
Nikita Glukhov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

On 2021-01-20 03:49, Nikita Glukhov wrote:

> [0001-Add-common-SQL-JSON-clauses-v52.patch.gz]
> [0002-SQL-JSON-constructors-v52.patch.gz]
> [0003-IS-JSON-predicate-v52.patch.gz]
> [0004-SQL-JSON-query-functions-v52.patch.gz]
> [0005-SQL-JSON-functions-for-json-type-v52.patch.gz]
> [0006-GUC-sql_json-v52.patch.gz]

Hi,

I read through the file func.sgml (only that file) and put the
errors/peculiarities in the attached diff. (Small stuff; typos really)

Your patch includes a CREATE TABLE my_films + INSERT, to run the
examples against. I think this is a great idea and we should do it more
often.

But, the table has a text-column to contain the subsequently inserted
json values. The insert runs fine but it turns out that some later
examples queries only run against a jsonb column. So I propose to
change:
CREATE TABLE my_films (js text);
to:
CREATE TABLE my_films (js jsonb);

This change is not yet included in the attached file. An alternative
would be to cast the text-column in the example queries as js::jsonb

I also noticed that some errors were different in the sgml file than 'in
the event':

SELECT JSON_QUERY(js, '$.favorites[*].kind' ERROR ON ERROR) FROM
my_films_jsonb;
(table 'my_films_jsonb' is the same as your 'my_films', but with js
as a jsonb column)

manual says: "ERROR: more than one SQL/JSON item"
in reality: "ERROR: JSON path expression in JSON_QUERY should return
singleton item without wrapper"
and: "HINT: use WITH WRAPPER clause to wrap SQL/JSON item
sequence into array"

Thanks,

Erik Rijkers

>
> --
> Nikita Glukhov
> Postgres Professional: http://www.postgrespro.com
> The Russian Postgres Company

On Sat, Jan 23, 2021 at 3:37 PM Erik Rijkers <er(at)xs4all(dot)nl> wrote:

> On 2021-01-20 03:49, Nikita Glukhov wrote:
>
> > [0001-Add-common-SQL-JSON-clauses-v52.patch.gz]
> > [0002-SQL-JSON-constructors-v52.patch.gz]
> > [0003-IS-JSON-predicate-v52.patch.gz]
> > [0004-SQL-JSON-query-functions-v52.patch.gz]
> > [0005-SQL-JSON-functions-for-json-type-v52.patch.gz]
> > [0006-GUC-sql_json-v52.patch.gz]
>
> Hi,
>
> I read through the file func.sgml (only that file) and put the
> errors/peculiarities in the attached diff. (Small stuff; typos really)
>
>
> Your patch includes a CREATE TABLE my_films + INSERT, to run the
> examples against. I think this is a great idea and we should do it more
> often.
>
> But, the table has a text-column to contain the subsequently inserted
> json values. The insert runs fine but it turns out that some later
> examples queries only run against a jsonb column. So I propose to
> change:
> CREATE TABLE my_films (js text);
> to:
> CREATE TABLE my_films (js jsonb);
>
> This change is not yet included in the attached file. An alternative
> would be to cast the text-column in the example queries as js::jsonb
>
>
> I also noticed that some errors were different in the sgml file than 'in
> the event':
>
>
> SELECT JSON_QUERY(js, '$.favorites[*].kind' ERROR ON ERROR) FROM
> my_films_jsonb;
> (table 'my_films_jsonb' is the same as your 'my_films', but with js
> as a jsonb column)
>
> manual says: "ERROR: more than one SQL/JSON item"
> in reality: "ERROR: JSON path expression in JSON_QUERY should return
> singleton item without wrapper"
> and: "HINT: use WITH WRAPPER clause to wrap SQL/JSON item
> sequence into array"
>
>
> Thanks,
>
> Erik Rijkers
>
> >
> > --
> > Nikita Glukhov
> > Postgres Professional: http://www.postgrespro.com
> > The Russian Postgres Company
>

The patch (func.sgml.20210123.diff) does not apply successfully.

http://cfbot.cputube.org/patch_32_2901.log

----

=== Applying patches on top of PostgreSQL commit ID
0ce4cd04da558178b0186057b721c50a00b7a945 ===
=== applying patch ./func.sgml.20210123.diff
patching file doc/src/sgml/func.sgml
Hunk #1 FAILED at 16968.
Hunk #2 FAILED at 17034.
...

Hunk #19 FAILED at 18743.
19 out of 19 hunks FAILED -- saving rejects to file doc/src/sgml/func.sgml.rej

----

Can we get a rebase?

I am marking the patch "Waiting on Author".

--
Ibrar Ahmed

On 3/8/21 1:55 PM, Ibrar Ahmed wrote:
>
>
> On Sat, Jan 23, 2021 at 3:37 PM Erik Rijkers <er(at)xs4all(dot)nl
> <mailto:er(at)xs4all(dot)nl>> wrote:
>
> On 2021-01-20 03:49, Nikita Glukhov wrote:
>
> > [0001-Add-common-SQL-JSON-clauses-v52.patch.gz]
> > [0002-SQL-JSON-constructors-v52.patch.gz]
> > [0003-IS-JSON-predicate-v52.patch.gz]
> > [0004-SQL-JSON-query-functions-v52.patch.gz]
> > [0005-SQL-JSON-functions-for-json-type-v52.patch.gz]
> > [0006-GUC-sql_json-v52.patch.gz]
>
> Hi,
>
> I read through the file func.sgml (only that file) and put the
> errors/peculiarities in the attached diff.  (Small stuff; typos
> really)
>
>
> Your patch includes a CREATE TABLE my_films + INSERT, to run the
> examples against.  I think this is a great idea and we should do
> it more
> often.
>
> But, the table has a text-column to contain the subsequently inserted
> json values. The insert runs fine but it turns out that some later
> examples queries only run against a jsonb column.  So I propose to
> change:
>    CREATE TABLE my_films (js text);
> to:
>    CREATE TABLE my_films (js jsonb);
>
> This change is not yet included in the attached file.  An alternative
> would be to cast the text-column in the example queries as js::jsonb
>
>
> I also noticed that some errors were different in the sgml file
> than 'in
> the event':
>
>
>     SELECT JSON_QUERY(js, '$.favorites[*].kind' ERROR ON ERROR) FROM
> my_films_jsonb;
>     (table 'my_films_jsonb' is the same as your 'my_films', but
> with js
> as a jsonb column)
>
> manual says: "ERROR: more than one SQL/JSON item"
>   in reality: "ERROR: JSON path expression in JSON_QUERY should
> return
> singleton item without wrapper"
>          and:   "HINT: use WITH WRAPPER clause to wrap SQL/JSON item
> sequence into array"
>
>
> Thanks,
>
> Erik Rijkers
>
> >
> > --
> > Nikita Glukhov
> > Postgres Professional: http://www.postgrespro.com
> <http://www.postgrespro.com>
> > The Russian Postgres Company
>
>
> The patch (func.sgml.20210123.diff) does not apply successfully.
>
> http://cfbot.cputube.org/patch_32_2901.log
> <http://cfbot.cputube.org/patch_32_2901.log>
>
> ----
> === Applying patches on top of PostgreSQL commit ID 0ce4cd04da558178b0186057b721c50a00b7a945 ===
> === applying patch ./func.sgml.20210123.diff
> patching file doc/src/sgml/func.sgml
> Hunk #1 FAILED at 16968.
> Hunk #2 FAILED at 17034.
> ...
> Hunk #19 FAILED at 18743.
> 19 out of 19 hunks FAILED -- saving rejects to file doc/src/sgml/func.sgml.rej
> ----
>
> Can we get a rebase? 
>
> I am marking the patch "Waiting on Author".

I've rebased this, and applied some of Erik's changes.

I'll set it back to 'Needs Review' if the cfbot is happy.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

On 3/26/21 4:22 PM, Andrew Dunstan wrote:
> On 3/8/21 1:55 PM, Ibrar Ahmed wrote:
>>
>> On Sat, Jan 23, 2021 at 3:37 PM Erik Rijkers <er(at)xs4all(dot)nl
>> <mailto:er(at)xs4all(dot)nl>> wrote:
>>
>> On 2021-01-20 03:49, Nikita Glukhov wrote:
>>
>> > [0001-Add-common-SQL-JSON-clauses-v52.patch.gz]
>> > [0002-SQL-JSON-constructors-v52.patch.gz]
>> > [0003-IS-JSON-predicate-v52.patch.gz]
>> > [0004-SQL-JSON-query-functions-v52.patch.gz]
>> > [0005-SQL-JSON-functions-for-json-type-v52.patch.gz]
>> > [0006-GUC-sql_json-v52.patch.gz]
>>
>> Hi,
>>
>> I read through the file func.sgml (only that file) and put the
>> errors/peculiarities in the attached diff.  (Small stuff; typos
>> really)
>>
>>
>> Your patch includes a CREATE TABLE my_films + INSERT, to run the
>> examples against.  I think this is a great idea and we should do
>> it more
>> often.
>>
>> But, the table has a text-column to contain the subsequently inserted
>> json values. The insert runs fine but it turns out that some later
>> examples queries only run against a jsonb column.  So I propose to
>> change:
>>    CREATE TABLE my_films (js text);
>> to:
>>    CREATE TABLE my_films (js jsonb);
>>
>> This change is not yet included in the attached file.  An alternative
>> would be to cast the text-column in the example queries as js::jsonb
>>
>>
>> I also noticed that some errors were different in the sgml file
>> than 'in
>> the event':
>>
>>
>>     SELECT JSON_QUERY(js, '$.favorites[*].kind' ERROR ON ERROR) FROM
>> my_films_jsonb;
>>     (table 'my_films_jsonb' is the same as your 'my_films', but
>> with js
>> as a jsonb column)
>>
>> manual says: "ERROR: more than one SQL/JSON item"
>>   in reality: "ERROR: JSON path expression in JSON_QUERY should
>> return
>> singleton item without wrapper"
>>          and:   "HINT: use WITH WRAPPER clause to wrap SQL/JSON item
>> sequence into array"
>>
>>
>> Thanks,
>>
>> Erik Rijkers
>>
>> >
>> > --
>> > Nikita Glukhov
>> > Postgres Professional: http://www.postgrespro.com
>> <http://www.postgrespro.com>
>> > The Russian Postgres Company
>>
>>
>> The patch (func.sgml.20210123.diff) does not apply successfully.
>>
>> http://cfbot.cputube.org/patch_32_2901.log
>> <http://cfbot.cputube.org/patch_32_2901.log>
>>
>> ----
>> === Applying patches on top of PostgreSQL commit ID 0ce4cd04da558178b0186057b721c50a00b7a945 ===
>> === applying patch ./func.sgml.20210123.diff
>> patching file doc/src/sgml/func.sgml
>> Hunk #1 FAILED at 16968.
>> Hunk #2 FAILED at 17034.
>> ...
>> Hunk #19 FAILED at 18743.
>> 19 out of 19 hunks FAILED -- saving rejects to file doc/src/sgml/func.sgml.rej
>> ----
>>
>> Can we get a rebase? 
>>
>> I am marking the patch "Waiting on Author".
>
>
>
> I've rebased this, and applied some of Erik's changes.
>
>
> I'll set it back to 'Needs Review' if the cfbot is happy.