postgres-xc-developers Mailing List for Postgres-XC

I tested to include postmaster/autovacuum.h both in master and
REL1_0_STABLE.   I didn't find any problem in master but found
conflicts with other header files in REL1_0_STABLE.

This may be caused by the difference in header files in PG 9.1 and 9.2
because gtm.c is the same between these branches.

Maybe for 1.0, we need to borrow the definition of these functions and
code directly.
----------
Koichi Suzuki


2013/2/11 Koichi Suzuki <koi...@gm...>:
> Michael;
>
> Thanks for pointing this out.   I opened a bug ticket for it and will
> commit the fix ASAP.    I understand there's many more to fix WARNINGS
> at compilation and we should fix them (with some exception, of
> course).
>
> This will be
> ----------
> Koichi Suzuki
>
>
> 2013/2/11 Michael Meskes <me...@po...>:
>> Hi,
>>
>> shouldn't src/backend/access/transam/gtm.c include "postmaster/autovacuum.h"?
>> Without it IsAutoVacuumLauncherProcess() is implicitely defined there.
>>
>> Michael
>> --
>> Michael Meskes
>> Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
>> Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org
>> Jabber: michael.meskes at gmail dot com
>> VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL
>>
>> ------------------------------------------------------------------------------
>> Free Next-Gen Firewall Hardware Offer
>> Buy your Sophos next-gen firewall before the end March 2013
>> and get the hardware for free! Learn more.
>> http://p.sf.net/sfu/sophos-d2d-feb
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers

Michael;

Thanks for pointing this out.   I opened a bug ticket for it and will
commit the fix ASAP.    I understand there's many more to fix WARNINGS
at compilation and we should fix them (with some exception, of
course).

This will be
----------
Koichi Suzuki


2013/2/11 Michael Meskes <me...@po...>:
> Hi,
>
> shouldn't src/backend/access/transam/gtm.c include "postmaster/autovacuum.h"?
> Without it IsAutoVacuumLauncherProcess() is implicitely defined there.
>
> Michael
> --
> Michael Meskes
> Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
> Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org
> Jabber: michael.meskes at gmail dot com
> VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL
>
> ------------------------------------------------------------------------------
> Free Next-Gen Firewall Hardware Offer
> Buy your Sophos next-gen firewall before the end March 2013
> and get the hardware for free! Learn more.
> http://p.sf.net/sfu/sophos-d2d-feb
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers

Hi,

shouldn't src/backend/access/transam/gtm.c include "postmaster/autovacuum.h"?
Without it IsAutoVacuumLauncherProcess() is implicitely defined there.

Michael
-- 
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org
Jabber: michael.meskes at gmail dot com
VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL

On Mon, Feb 11, 2013 at 9:53 AM, Amit Khandekar <
ami...@en...> wrote:

> On 9 February 2013 12:06, Abbas Butt <abb...@en...> wrote:
> > Please use this patch, I forgot to change a comment in the patch I sent
> > earlier.
>
> Yes, the target list handling looks ok now. ( I am not reviewing the
> insert deparsing part)
>
> You seem to have left out statements like :
> +printf("\n[%s][%d][%s]\n", __FUNCTION__, __LINE__, query_string);
>

Thanks for noticing this, It was left by mistake, I will remove it from the
final patch before committing.


>
>
> >
> >
> > On Sat, Feb 9, 2013 at 11:29 AM, Abbas Butt <abb...@en...
> >
> > wrote:
> >>
> >>
> >>
> >> On Fri, Feb 8, 2013 at 5:19 PM, Amit Khandekar
> >> <ami...@en...> wrote:
> >>>
> >>> On 7 February 2013 17:48, Abbas Butt <abb...@en...>
> wrote:
> >>> >
> >>> >
> >>> > On Fri, Feb 1, 2013 at 2:14 AM, Amit Khandekar
> >>> > <ami...@en...> wrote:
> >>> >>
> >>> >>
> >>> >>
> >>> >> On 29 January 2013 18:40, Abbas Butt <abb...@en...>
> >>> >> wrote:
> >>> >>>
> >>> >>> Hi,
> >>> >>> PFA a patch that uses Query structure to construct the query to be
> >>> >>> sent
> >>> >>> to the datanodes.
> >>> >>> The patch removes the functions create_remoteinsert_plan,
> >>> >>> create_remoteupdate_plan and create_remotedelete_plan.
> >>> >>> Instead it adds a single function create_remote_dml_plan.
> >>> >>> The patch has detailed comments to explain all the implementation
> >>> >>> details.
> >>> >>> The patch shows no extra regression failures, and compiles without
> >>> >>> any
> >>> >>> new warnings.
> >>> >>> Applies cleanly on current master
> >>> >>> (f4a3652271d80627dcc28171eeccd7620ad397c7).
> >>> >>>
> >>> >>> Comments are welcome.
> >>> >>
> >>> >>
> >>> >> This is a nice thing that we now do the remote query generation
> using
> >>> >> the
> >>> >> deparse logic, and secondly we now have a single function that has
> >>> >> integrated all the duplicated code that is currently present in the
> >>> >> insert/update/delete operations.
> >>> >>
> >>> >> I have done some sanity testing and not found any issues. Here are
> >>> >> some
> >>> >> comments from looking the patch:
> >>> >>
> >>> >> -------
> >>> >>
> >>> >> insert_query_def() :
> >>> >>
> >>> >> The new XC comment says select_rte  and values_rte are neither
> >>> >> required
> >>> >> nor should be present in XC. And the PG checks have been removed.
> Can
> >>> >> we
> >>> >> instead replace the PG checks with the XC-specific checks ? Like :
> >>> >>     if (select_rte || values_rte)
> >>> >>         elog(ERROR, "neither subquery nor values RTEs should be
> >>> >> present in
> >>> >> INSERT");
> >>> >
> >>> >
> >>> > We can, but these is no point having such a condition in the code.
> >>> > Note that while declaring the variables we set them to null
> >>> >
> >>> >     RangeTblEntry *select_rte = NULL;
> >>> >     RangeTblEntry *values_rte = NULL;
> >>> >
> >>> > and then we skip the for loop that is supposed to put any non null
> >>> > value in
> >>> > them
> >>> > then right after that skipped loop there is no point checking whether
> >>> > the
> >>> > variables we just set to null are null or not.
> >>> >
> >>> >>
> >>> >>
> >>> >> --------------
> >>> >>
> >>> >>
> >>> >> Following query species insert when the operation is a general DML
> >>> >> operation.
> >>> >>
> >>> >>     /*
> >>> >>      * For every result relation, build a remote plan to execute
> >>> >> remote
> >>> >> insert.
> >>> >>      */
> >>> >>
> >>> >
> >>> > Thanks for noticing this, I have changed the comment to refer to a
> DML
> >>> > instead of an INSERT.
> >>> >
> >>> >>
> >>> >>
> >>> >>
> >>> >> ------------
> >>> >>
> >>> >> The source target list resnames are used to find the target list
> entry
> >>> >> from root->parse->targetlist that matches the source plan target
> list.
> >>> >> Can
> >>> >> the resnames be NULL
> >>> >
> >>> >
> >>> > No, we are talking about the SET clause of an UPDATE query.
> >>> >
> >>>
> >>> I was not referring to resnames from query target list. I was talking
> >>> about resnames belonging to *source* target list. Those resnames do
> >>> not represent the correct table attribute name. For e.g. :
> >>>
> >>> postgres=# \d tab1
> >>>             Table "public.tab1"
> >>>   Column   |       Type        | Modifiers
> >>> -----------+-------------------+-----------
> >>>  id        | integer           |
> >>>  tab1name1 | character varying |
> >>>  tab1name2 | character varying |
> >>>
> >>> postgres=# \d tab2
> >>>             Table "public.tab2"
> >>>   Column   |       Type        | Modifiers
> >>> -----------+-------------------+-----------
> >>>  id        | integer           |
> >>>  tab2name1 | character varying |
> >>>
> >>> For the below query:
> >>> update tab1 set tab1name1 = f(tj.tab1name2) from (select tab2.id,
> >>> tab1.tab1name2 from tab1 join tab2 on tab1.id = tab2.id) tj;
> >>> As per the patch, in order to get the correct parameter number for
> >>> "set tab1name = $.." , it will try to search resname tab1name1 in the
> >>> source target list, but the source target list does not have any such
> >>> resname tab1name1 because it does not even select tab1name1. So the
> >>> above query fails with "Source data plan's target list does not
> >>> contain : tab1name1".
> >>>
> >>> Actually we should search an entry in the source tlist that somehow
> >>> *corresponds* to SET tab1name1.
> >>>
> >>> Currently, how does that happen ? Currently, the column to be set is
> >>> *always* at the position equal to the column's table attribute number
> >>> in the source tlist. So we know that the value corresponding to "SET
> >>> tab1name1" is always at position 2 in the source target list because
> >>> tab1name1 is defined as the 2nd column of tab1. This is PG behaviour.
> >>>
> >>> Now, once we try to handle the tle search assuming that the source
> >>> target list is of any arbitrary order, things become complicated. If
> >>> we search by resname, it fails as shown above; if we search by that
> >>> column's attribute number (so for set tab1name = .. search for
> >>> varattno=2,var=1 say), how can we be sure it is the right hit ? Can
> >>> the source tlist have multiple occurences of the same column ? Which
> >>> one to choose then ? Bottom line is,  we should come up with some
> >>> rules about where to *keep* in the source tlist the tle corresponding
> >>> to the SET col, only then we can be sure where to find it. May be we
> >>> can always keep all the SET columns in the very first set of items in
> >>> the source tlist. Or something else ...
> >>>
> >>> So as of now, we haven't talked about how that source tlist is going
> >>> to be ... the current PG way of matching the tles is what I personally
> >>> like. Here, all attributes are present in the source tlist. But then
> >>> as optimization, we can keep NULL constants for tles that are not
> >>> updated (i.e. they do not have SET entry). Ultimately the remote tlist
> >>> won't have these NULLs, they will *only* have the non-NULL columns
> >>> thanks to projection support in RemoteQuery; so we are good.
> >>>
> >>> So in conclusion, I would suggest (unless you have yet another
> >>> approach) :  just assume that there are all attributes there in source
> >>> tlist, and don't change the current way of searching the tles. So you
> >>> need to call get_attnum(qtle->resname) to find the parameter number.
> >>
> >>
> >> Thanks for the explanation, I have changed the patch accordingly.
> >>
> >>>
> >>>
> >>> >>
> >>> >> or can their values be different than the actual table attribute
> names
> >>> >> ?
> >>> >
> >>> >
> >>> > No, again this is the case of the SET clause of an UPDATE query.
> >>> >
> >>> >>
> >>> >> In order to make sure we always find the right target list element,
> we
> >>> >> can
> >>> >> use something else to find out the tle entries of
> >>> >> root->parse->targetlist in
> >>> >> the source tlist. May be we can use get_attnum(qtle->resname) as the
> >>> >> parameter number ?
> >>> >
> >>> >
> >>> > get_attnum should NOT be used, because SET clause of an update query
> >>> > can
> >>> > skip some columns of the table. e.g.
> >>> > suppose we have a table
> >>> >
> >>> > create table tab (a int, b int, c int, d int);
> >>> >
> >>> > and we issue an update query like
> >>> >
> >>> > test=# explain verbose update tab set c=c+1 returning c;
> >>> >                                                   QUERY PLAN
> >>> >
> >>> >
> --------------------------------------------------------------------------------------------------------------
> >>> >  Update on public.tab  (cost=0.00..2.50 rows=1000 width=26)
> >>> >    Output: tab.c
> >>> >    Node/s: data_node_1, data_node_2
> >>> >    Node expr: tab.a
> >>> >    Remote query: UPDATE ONLY tab SET c = $3 WHERE ((tab.ctid = $5)
> AND
> >>> > (tab.xc_node_id = $6)) RETURNING tab.c
> >>> >    ->  Data Node Scan on tab "_REMOTE_TABLE_QUERY_"  (cost=0.00..2.50
> >>> > rows=1000 width=26)
> >>> >          Output: tab.a, tab.b, (tab.c + 1), tab.d, tab.ctid,
> >>> > tab.xc_node_id
> >>> >          Node/s: data_node_1, data_node_2
> >>> >          Remote query: SELECT a, b, c, d, ctid, xc_node_id FROM ONLY
> >>> > tab
> >>> > WHERE true
> >>> > (9 rows)
> >>> >
> >>> > Currently the data node scan selects columns 'a','b' and 'd'
> >>> > un-necessarily.
> >>> > This might change in near future as an optimization and the optimized
> >>> > remote
> >>> > table scan query would become
> >>> >
> >>> > Remote query: SELECT c, ctid, xc_node_id FROM ONLY tab WHERE true
> >>> >
> >>> > Now we need to supply $1 for the SET command clause in the update
> query
> >>> > whereas if we use the columns attribute number in table we will
> wrongly
> >>> > use
> >>> > $3.
> >>> >
> >>>
> >>> >
> >>> >>
> >>> >> Because as far as I understand, the data source target list *must*
> >>> >> match
> >>> >> the order of the table attributes.
> >>> >
> >>> >
> >>> > Its not about the order, its about the number of attributes selected
> in
> >>> > the
> >>> > query to scan the remote table. As explained above it is not
> necessary
> >>> > for a
> >>> > remote scan query to select all the columns of the target table.
> >>> >
> >>> >>
> >>> >>
> >>> >> Or is there an assumption that the source target list order can be
> >>> >> different than the order of table attribtues ? From the way
> >>> >> stle->resno is
> >>> >> used to assign the parameter number (param_num = stle->resno), it
> >>> >> seems
> >>> >> that's the assumption.
> >>> >
> >>> >
> >>> > stle->resno would only show the ordinal position of the column in the
> >>> > select
> >>> > query and that can be different from the position of the column in
> the
> >>> > table
> >>> > as explained in the example above.
> >>> >
> >>> >>
> >>> >>
> >>> >> Also note that the source target list also has the joined table
> vars.
> >>> >> That
> >>> >> might be harmless, because they are always present *after* the
> >>> >> resultRelation target list, so theoretically the joined table target
> >>> >> list
> >>> >> will be never be scanned because we should always find a match in
> the
> >>> >> resultRelation target list. But just in case we don't find a match
> >>> >> (you have
> >>> >> used param_num == -1 to catch that), it will start comparing
> resnames
> >>> >> and
> >>> >> resno of the other table.
> >>> >>
> >>> >> Actually there *is* one place where the joined table tle's are also
> >>> >> scanned :
> >>> >>
> >>> >>     /*
> >>> >>      * Prepare a param list for INSERT queries
> >>> >>      * While doing so note the position of ctid, xc_node_id in
> source
> >>> >> data
> >>> >>      * plan's target list provided by the caller.
> >>> >>      */
> >>> >>     foreach(elt, sourceTargetList)
> >>> >>     {
> >>> >>         TargetEntry    *tle = lfirst(elt);
> >>> >>
> >>> >>         col_att++;
> >>> >>
> >>> >>         /* The position of ctid/xc_node_id is not required for
> INSERT
> >>> >> */
> >>> >>         if (tle->resjunk && (cmdtype == CMD_UPDATE || cmdtype ==
> >>> >> CMD_DELETE))
> >>> >> ........
> >>> >> ........
> >>> >>        if (cmdtype == CMD_INSERT)
> >>> >> ........
> >>> >> ........
> >>> >>      }
> >>> >>
> >>> >> Here, we are comparing var->varattnos, but the var itself can be
> from
> >>> >> the
> >>> >> other table. Instead, we can scan the first n elements where n is
> the
> >>> >> number
> >>> >> of table attribtues.
> >>> >
> >>> >
> >>> > Instead I have used a different condition to take care of this,
> >>> > scanning
> >>> > first n elements is based on the fact that the remote table scan
> query
> >>> > will
> >>> > always contain all the columns of the remote table.
> >>> >
> >>> > I have used
> >>> >         if (tle->resorigtbl != 0 && tle->resorigtbl !=
> res_rel->relid)
> >>>
> >>> This won't be necessary if you don't change the way the tle's are
> >>> searched currently as I mentioned above.
> >>>
> >>> >
> >>> >
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >> --
> >>> >>>
> >>> >>> Abbas
> >>> >>> Architect
> >>> >>> EnterpriseDB Corporation
> >>> >>> The Enterprise PostgreSQL Company
> >>> >>>
> >>> >>> Phone: 92-334-5100153
> >>> >>>
> >>> >>> Website: www.enterprisedb.com
> >>> >>> EnterpriseDB Blog: http://blogs.enterprisedb.com/
> >>> >>> Follow us on Twitter: http://www.twitter.com/enterprisedb
> >>> >>>
> >>> >>> This e-mail message (and any attachment) is intended for the use of
> >>> >>> the individual or entity to whom it is addressed. This message
> >>> >>> contains information from EnterpriseDB Corporation that may be
> >>> >>> privileged, confidential, or exempt from disclosure under
> applicable
> >>> >>> law. If you are not the intended recipient or authorized to receive
> >>> >>> this for the intended recipient, any use, dissemination,
> >>> >>> distribution,
> >>> >>> retention, archiving, or copying of this communication is strictly
> >>> >>> prohibited. If you have received this e-mail in error, please
> notify
> >>> >>> the sender immediately by reply e-mail and delete this message.
> >>> >>>
> >>> >>>
> >>> >>>
> ------------------------------------------------------------------------------
> >>> >>> Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5,
> CSS,
> >>> >>> MVC, Windows 8 Apps, JavaScript and much more. Keep your skills
> >>> >>> current
> >>> >>> with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
> >>> >>> MVPs and experts. ON SALE this month only -- learn more at:
> >>> >>> http://p.sf.net/sfu/learnnow-d2d
> >>> >>> _______________________________________________
> >>> >>> Postgres-xc-developers mailing list
> >>> >>> Pos...@li...
> >>> >>>
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
> >>> >>>
> >>> >>
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > --
> >>> > Abbas
> >>> > Architect
> >>> > EnterpriseDB Corporation
> >>> > The Enterprise PostgreSQL Company
> >>> >
> >>> > Phone: 92-334-5100153
> >>> >
> >>> > Website: www.enterprisedb.com
> >>> > EnterpriseDB Blog: http://blogs.enterprisedb.com/
> >>> > Follow us on Twitter: http://www.twitter.com/enterprisedb
> >>> >
> >>> > This e-mail message (and any attachment) is intended for the use of
> >>> > the individual or entity to whom it is addressed. This message
> >>> > contains information from EnterpriseDB Corporation that may be
> >>> > privileged, confidential, or exempt from disclosure under applicable
> >>> > law. If you are not the intended recipient or authorized to receive
> >>> > this for the intended recipient, any use, dissemination,
> distribution,
> >>> > retention, archiving, or copying of this communication is strictly
> >>> > prohibited. If you have received this e-mail in error, please notify
> >>> > the sender immediately by reply e-mail and delete this message.
> >>
> >>
> >>
> >>
> >> --
> >> --
> >> Abbas
> >> Architect
> >> EnterpriseDB Corporation
> >> The Enterprise PostgreSQL Company
> >>
> >> Phone: 92-334-5100153
> >>
> >> Website: www.enterprisedb.com
> >> EnterpriseDB Blog: http://blogs.enterprisedb.com/
> >> Follow us on Twitter: http://www.twitter.com/enterprisedb
> >>
> >> This e-mail message (and any attachment) is intended for the use of
> >> the individual or entity to whom it is addressed. This message
> >> contains information from EnterpriseDB Corporation that may be
> >> privileged, confidential, or exempt from disclosure under applicable
> >> law. If you are not the intended recipient or authorized to receive
> >> this for the intended recipient, any use, dissemination, distribution,
> >> retention, archiving, or copying of this communication is strictly
> >> prohibited. If you have received this e-mail in error, please notify
> >> the sender immediately by reply e-mail and delete this message.
> >
> >
> >
> >
> > --
> > --
> > Abbas
> > Architect
> > EnterpriseDB Corporation
> > The Enterprise PostgreSQL Company
> >
> > Phone: 92-334-5100153
> >
> > Website: www.enterprisedb.com
> > EnterpriseDB Blog: http://blogs.enterprisedb.com/
> > Follow us on Twitter: http://www.twitter.com/enterprisedb
> >
> > This e-mail message (and any attachment) is intended for the use of
> > the individual or entity to whom it is addressed. This message
> > contains information from EnterpriseDB Corporation that may be
> > privileged, confidential, or exempt from disclosure under applicable
> > law. If you are not the intended recipient or authorized to receive
> > this for the intended recipient, any use, dissemination, distribution,
> > retention, archiving, or copying of this communication is strictly
> > prohibited. If you have received this e-mail in error, please notify
> > the sender immediately by reply e-mail and delete this message.
>



-- 
--
Abbas
Architect
EnterpriseDB Corporation
The Enterprise PostgreSQL Company

Phone: 92-334-5100153

Website: www.enterprisedb.com
EnterpriseDB Blog: http://blogs.enterprisedb.com/
Follow us on Twitter: http://www.twitter.com/enterprisedb

This e-mail message (and any attachment) is intended for the use of
the individual or entity to whom it is addressed. This message
contains information from EnterpriseDB Corporation that may be
privileged, confidential, or exempt from disclosure under applicable
law. If you are not the intended recipient or authorized to receive
this for the intended recipient, any use, dissemination, distribution,
retention, archiving, or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify
the sender immediately by reply e-mail and delete this message.

On 9 February 2013 12:06, Abbas Butt <abb...@en...> wrote:
> Please use this patch, I forgot to change a comment in the patch I sent
> earlier.

Yes, the target list handling looks ok now. ( I am not reviewing the
insert deparsing part)

You seem to have left out statements like :
+printf("\n[%s][%d][%s]\n", __FUNCTION__, __LINE__, query_string);


>
>
> On Sat, Feb 9, 2013 at 11:29 AM, Abbas Butt <abb...@en...>
> wrote:
>>
>>
>>
>> On Fri, Feb 8, 2013 at 5:19 PM, Amit Khandekar
>> <ami...@en...> wrote:
>>>
>>> On 7 February 2013 17:48, Abbas Butt <abb...@en...> wrote:
>>> >
>>> >
>>> > On Fri, Feb 1, 2013 at 2:14 AM, Amit Khandekar
>>> > <ami...@en...> wrote:
>>> >>
>>> >>
>>> >>
>>> >> On 29 January 2013 18:40, Abbas Butt <abb...@en...>
>>> >> wrote:
>>> >>>
>>> >>> Hi,
>>> >>> PFA a patch that uses Query structure to construct the query to be
>>> >>> sent
>>> >>> to the datanodes.
>>> >>> The patch removes the functions create_remoteinsert_plan,
>>> >>> create_remoteupdate_plan and create_remotedelete_plan.
>>> >>> Instead it adds a single function create_remote_dml_plan.
>>> >>> The patch has detailed comments to explain all the implementation
>>> >>> details.
>>> >>> The patch shows no extra regression failures, and compiles without
>>> >>> any
>>> >>> new warnings.
>>> >>> Applies cleanly on current master
>>> >>> (f4a3652271d80627dcc28171eeccd7620ad397c7).
>>> >>>
>>> >>> Comments are welcome.
>>> >>
>>> >>
>>> >> This is a nice thing that we now do the remote query generation using
>>> >> the
>>> >> deparse logic, and secondly we now have a single function that has
>>> >> integrated all the duplicated code that is currently present in the
>>> >> insert/update/delete operations.
>>> >>
>>> >> I have done some sanity testing and not found any issues. Here are
>>> >> some
>>> >> comments from looking the patch:
>>> >>
>>> >> -------
>>> >>
>>> >> insert_query_def() :
>>> >>
>>> >> The new XC comment says select_rte  and values_rte are neither
>>> >> required
>>> >> nor should be present in XC. And the PG checks have been removed. Can
>>> >> we
>>> >> instead replace the PG checks with the XC-specific checks ? Like :
>>> >>     if (select_rte || values_rte)
>>> >>         elog(ERROR, "neither subquery nor values RTEs should be
>>> >> present in
>>> >> INSERT");
>>> >
>>> >
>>> > We can, but these is no point having such a condition in the code.
>>> > Note that while declaring the variables we set them to null
>>> >
>>> >     RangeTblEntry *select_rte = NULL;
>>> >     RangeTblEntry *values_rte = NULL;
>>> >
>>> > and then we skip the for loop that is supposed to put any non null
>>> > value in
>>> > them
>>> > then right after that skipped loop there is no point checking whether
>>> > the
>>> > variables we just set to null are null or not.
>>> >
>>> >>
>>> >>
>>> >> --------------
>>> >>
>>> >>
>>> >> Following query species insert when the operation is a general DML
>>> >> operation.
>>> >>
>>> >>     /*
>>> >>      * For every result relation, build a remote plan to execute
>>> >> remote
>>> >> insert.
>>> >>      */
>>> >>
>>> >
>>> > Thanks for noticing this, I have changed the comment to refer to a DML
>>> > instead of an INSERT.
>>> >
>>> >>
>>> >>
>>> >>
>>> >> ------------
>>> >>
>>> >> The source target list resnames are used to find the target list entry
>>> >> from root->parse->targetlist that matches the source plan target list.
>>> >> Can
>>> >> the resnames be NULL
>>> >
>>> >
>>> > No, we are talking about the SET clause of an UPDATE query.
>>> >
>>>
>>> I was not referring to resnames from query target list. I was talking
>>> about resnames belonging to *source* target list. Those resnames do
>>> not represent the correct table attribute name. For e.g. :
>>>
>>> postgres=# \d tab1
>>>             Table "public.tab1"
>>>   Column   |       Type        | Modifiers
>>> -----------+-------------------+-----------
>>>  id        | integer           |
>>>  tab1name1 | character varying |
>>>  tab1name2 | character varying |
>>>
>>> postgres=# \d tab2
>>>             Table "public.tab2"
>>>   Column   |       Type        | Modifiers
>>> -----------+-------------------+-----------
>>>  id        | integer           |
>>>  tab2name1 | character varying |
>>>
>>> For the below query:
>>> update tab1 set tab1name1 = f(tj.tab1name2) from (select tab2.id,
>>> tab1.tab1name2 from tab1 join tab2 on tab1.id = tab2.id) tj;
>>> As per the patch, in order to get the correct parameter number for
>>> "set tab1name = $.." , it will try to search resname tab1name1 in the
>>> source target list, but the source target list does not have any such
>>> resname tab1name1 because it does not even select tab1name1. So the
>>> above query fails with "Source data plan's target list does not
>>> contain : tab1name1".
>>>
>>> Actually we should search an entry in the source tlist that somehow
>>> *corresponds* to SET tab1name1.
>>>
>>> Currently, how does that happen ? Currently, the column to be set is
>>> *always* at the position equal to the column's table attribute number
>>> in the source tlist. So we know that the value corresponding to "SET
>>> tab1name1" is always at position 2 in the source target list because
>>> tab1name1 is defined as the 2nd column of tab1. This is PG behaviour.
>>>
>>> Now, once we try to handle the tle search assuming that the source
>>> target list is of any arbitrary order, things become complicated. If
>>> we search by resname, it fails as shown above; if we search by that
>>> column's attribute number (so for set tab1name = .. search for
>>> varattno=2,var=1 say), how can we be sure it is the right hit ? Can
>>> the source tlist have multiple occurences of the same column ? Which
>>> one to choose then ? Bottom line is,  we should come up with some
>>> rules about where to *keep* in the source tlist the tle corresponding
>>> to the SET col, only then we can be sure where to find it. May be we
>>> can always keep all the SET columns in the very first set of items in
>>> the source tlist. Or something else ...
>>>
>>> So as of now, we haven't talked about how that source tlist is going
>>> to be ... the current PG way of matching the tles is what I personally
>>> like. Here, all attributes are present in the source tlist. But then
>>> as optimization, we can keep NULL constants for tles that are not
>>> updated (i.e. they do not have SET entry). Ultimately the remote tlist
>>> won't have these NULLs, they will *only* have the non-NULL columns
>>> thanks to projection support in RemoteQuery; so we are good.
>>>
>>> So in conclusion, I would suggest (unless you have yet another
>>> approach) :  just assume that there are all attributes there in source
>>> tlist, and don't change the current way of searching the tles. So you
>>> need to call get_attnum(qtle->resname) to find the parameter number.
>>
>>
>> Thanks for the explanation, I have changed the patch accordingly.
>>
>>>
>>>
>>> >>
>>> >> or can their values be different than the actual table attribute names
>>> >> ?
>>> >
>>> >
>>> > No, again this is the case of the SET clause of an UPDATE query.
>>> >
>>> >>
>>> >> In order to make sure we always find the right target list element, we
>>> >> can
>>> >> use something else to find out the tle entries of
>>> >> root->parse->targetlist in
>>> >> the source tlist. May be we can use get_attnum(qtle->resname) as the
>>> >> parameter number ?
>>> >
>>> >
>>> > get_attnum should NOT be used, because SET clause of an update query
>>> > can
>>> > skip some columns of the table. e.g.
>>> > suppose we have a table
>>> >
>>> > create table tab (a int, b int, c int, d int);
>>> >
>>> > and we issue an update query like
>>> >
>>> > test=# explain verbose update tab set c=c+1 returning c;
>>> >                                                   QUERY PLAN
>>> >
>>> > --------------------------------------------------------------------------------------------------------------
>>> >  Update on public.tab  (cost=0.00..2.50 rows=1000 width=26)
>>> >    Output: tab.c
>>> >    Node/s: data_node_1, data_node_2
>>> >    Node expr: tab.a
>>> >    Remote query: UPDATE ONLY tab SET c = $3 WHERE ((tab.ctid = $5) AND
>>> > (tab.xc_node_id = $6)) RETURNING tab.c
>>> >    ->  Data Node Scan on tab "_REMOTE_TABLE_QUERY_"  (cost=0.00..2.50
>>> > rows=1000 width=26)
>>> >          Output: tab.a, tab.b, (tab.c + 1), tab.d, tab.ctid,
>>> > tab.xc_node_id
>>> >          Node/s: data_node_1, data_node_2
>>> >          Remote query: SELECT a, b, c, d, ctid, xc_node_id FROM ONLY
>>> > tab
>>> > WHERE true
>>> > (9 rows)
>>> >
>>> > Currently the data node scan selects columns 'a','b' and 'd'
>>> > un-necessarily.
>>> > This might change in near future as an optimization and the optimized
>>> > remote
>>> > table scan query would become
>>> >
>>> > Remote query: SELECT c, ctid, xc_node_id FROM ONLY tab WHERE true
>>> >
>>> > Now we need to supply $1 for the SET command clause in the update query
>>> > whereas if we use the columns attribute number in table we will wrongly
>>> > use
>>> > $3.
>>> >
>>>
>>> >
>>> >>
>>> >> Because as far as I understand, the data source target list *must*
>>> >> match
>>> >> the order of the table attributes.
>>> >
>>> >
>>> > Its not about the order, its about the number of attributes selected in
>>> > the
>>> > query to scan the remote table. As explained above it is not necessary
>>> > for a
>>> > remote scan query to select all the columns of the target table.
>>> >
>>> >>
>>> >>
>>> >> Or is there an assumption that the source target list order can be
>>> >> different than the order of table attribtues ? From the way
>>> >> stle->resno is
>>> >> used to assign the parameter number (param_num = stle->resno), it
>>> >> seems
>>> >> that's the assumption.
>>> >
>>> >
>>> > stle->resno would only show the ordinal position of the column in the
>>> > select
>>> > query and that can be different from the position of the column in the
>>> > table
>>> > as explained in the example above.
>>> >
>>> >>
>>> >>
>>> >> Also note that the source target list also has the joined table vars.
>>> >> That
>>> >> might be harmless, because they are always present *after* the
>>> >> resultRelation target list, so theoretically the joined table target
>>> >> list
>>> >> will be never be scanned because we should always find a match in the
>>> >> resultRelation target list. But just in case we don't find a match
>>> >> (you have
>>> >> used param_num == -1 to catch that), it will start comparing resnames
>>> >> and
>>> >> resno of the other table.
>>> >>
>>> >> Actually there *is* one place where the joined table tle's are also
>>> >> scanned :
>>> >>
>>> >>     /*
>>> >>      * Prepare a param list for INSERT queries
>>> >>      * While doing so note the position of ctid, xc_node_id in source
>>> >> data
>>> >>      * plan's target list provided by the caller.
>>> >>      */
>>> >>     foreach(elt, sourceTargetList)
>>> >>     {
>>> >>         TargetEntry    *tle = lfirst(elt);
>>> >>
>>> >>         col_att++;
>>> >>
>>> >>         /* The position of ctid/xc_node_id is not required for INSERT
>>> >> */
>>> >>         if (tle->resjunk && (cmdtype == CMD_UPDATE || cmdtype ==
>>> >> CMD_DELETE))
>>> >> ........
>>> >> ........
>>> >>        if (cmdtype == CMD_INSERT)
>>> >> ........
>>> >> ........
>>> >>      }
>>> >>
>>> >> Here, we are comparing var->varattnos, but the var itself can be from
>>> >> the
>>> >> other table. Instead, we can scan the first n elements where n is the
>>> >> number
>>> >> of table attribtues.
>>> >
>>> >
>>> > Instead I have used a different condition to take care of this,
>>> > scanning
>>> > first n elements is based on the fact that the remote table scan query
>>> > will
>>> > always contain all the columns of the remote table.
>>> >
>>> > I have used
>>> >         if (tle->resorigtbl != 0 && tle->resorigtbl != res_rel->relid)
>>>
>>> This won't be necessary if you don't change the way the tle's are
>>> searched currently as I mentioned above.
>>>
>>> >
>>> >
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >>>
>>> >>> Abbas
>>> >>> Architect
>>> >>> EnterpriseDB Corporation
>>> >>> The Enterprise PostgreSQL Company
>>> >>>
>>> >>> Phone: 92-334-5100153
>>> >>>
>>> >>> Website: www.enterprisedb.com
>>> >>> EnterpriseDB Blog: http://blogs.enterprisedb.com/
>>> >>> Follow us on Twitter: http://www.twitter.com/enterprisedb
>>> >>>
>>> >>> This e-mail message (and any attachment) is intended for the use of
>>> >>> the individual or entity to whom it is addressed. This message
>>> >>> contains information from EnterpriseDB Corporation that may be
>>> >>> privileged, confidential, or exempt from disclosure under applicable
>>> >>> law. If you are not the intended recipient or authorized to receive
>>> >>> this for the intended recipient, any use, dissemination,
>>> >>> distribution,
>>> >>> retention, archiving, or copying of this communication is strictly
>>> >>> prohibited. If you have received this e-mail in error, please notify
>>> >>> the sender immediately by reply e-mail and delete this message.
>>> >>>
>>> >>>
>>> >>> ------------------------------------------------------------------------------
>>> >>> Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS,
>>> >>> MVC, Windows 8 Apps, JavaScript and much more. Keep your skills
>>> >>> current
>>> >>> with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
>>> >>> MVPs and experts. ON SALE this month only -- learn more at:
>>> >>> http://p.sf.net/sfu/learnnow-d2d
>>> >>> _______________________________________________
>>> >>> Postgres-xc-developers mailing list
>>> >>> Pos...@li...
>>> >>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>> >>>
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > --
>>> > Abbas
>>> > Architect
>>> > EnterpriseDB Corporation
>>> > The Enterprise PostgreSQL Company
>>> >
>>> > Phone: 92-334-5100153
>>> >
>>> > Website: www.enterprisedb.com
>>> > EnterpriseDB Blog: http://blogs.enterprisedb.com/
>>> > Follow us on Twitter: http://www.twitter.com/enterprisedb
>>> >
>>> > This e-mail message (and any attachment) is intended for the use of
>>> > the individual or entity to whom it is addressed. This message
>>> > contains information from EnterpriseDB Corporation that may be
>>> > privileged, confidential, or exempt from disclosure under applicable
>>> > law. If you are not the intended recipient or authorized to receive
>>> > this for the intended recipient, any use, dissemination, distribution,
>>> > retention, archiving, or copying of this communication is strictly
>>> > prohibited. If you have received this e-mail in error, please notify
>>> > the sender immediately by reply e-mail and delete this message.
>>
>>
>>
>>
>> --
>> --
>> Abbas
>> Architect
>> EnterpriseDB Corporation
>> The Enterprise PostgreSQL Company
>>
>> Phone: 92-334-5100153
>>
>> Website: www.enterprisedb.com
>> EnterpriseDB Blog: http://blogs.enterprisedb.com/
>> Follow us on Twitter: http://www.twitter.com/enterprisedb
>>
>> This e-mail message (and any attachment) is intended for the use of
>> the individual or entity to whom it is addressed. This message
>> contains information from EnterpriseDB Corporation that may be
>> privileged, confidential, or exempt from disclosure under applicable
>> law. If you are not the intended recipient or authorized to receive
>> this for the intended recipient, any use, dissemination, distribution,
>> retention, archiving, or copying of this communication is strictly
>> prohibited. If you have received this e-mail in error, please notify
>> the sender immediately by reply e-mail and delete this message.
>
>
>
>
> --
> --
> Abbas
> Architect
> EnterpriseDB Corporation
> The Enterprise PostgreSQL Company
>
> Phone: 92-334-5100153
>
> Website: www.enterprisedb.com
> EnterpriseDB Blog: http://blogs.enterprisedb.com/
> Follow us on Twitter: http://www.twitter.com/enterprisedb
>
> This e-mail message (and any attachment) is intended for the use of
> the individual or entity to whom it is addressed. This message
> contains information from EnterpriseDB Corporation that may be
> privileged, confidential, or exempt from disclosure under applicable
> law. If you are not the intended recipient or authorized to receive
> this for the intended recipient, any use, dissemination, distribution,
> retention, archiving, or copying of this communication is strictly
> prohibited. If you have received this e-mail in error, please notify
> the sender immediately by reply e-mail and delete this message.