postgres-xc-general Mailing List for Postgres-XC (Page 24)

Hi Koichi, Masataka,

 Thanks for taking a look. I double checked both memory usage and the dmesg
log. The kill is not happening due to the out of memory. The kernel doesn't
show any processes being killed.

My current suspicion is that something amiss is happening over the network.
This happens consistently when performing copy over large datasets. Few of
datanodes end up with "invalid string in message".
Going over the code in pqformat.c this happens because the invariant
"StringInfo is guaranteed to have a trailing null byte" is not valid (line
682 -- 637). I am not sure why this happens.


After this the datanodes do something illegal (probably buffer overflow on
the port..just a guess) that the OS decides to issue a sigkill.  I am also
not sure why the OS is not logging this killed operation.


Please let me know of  suggestions you may have regarding the "invalid
string in message". In particular, where in the code is StringInfo msg is
getting populated i.e. which code reads the message over the network and
keeps it ready for the copy operation on the datanode. Any other suggestion
would also be helpful to trace this bug.

Best,
Sandeep






On Thu, Nov 14, 2013 at 10:25 AM, Masataka Saito <pg...@gm...> wrote:

> Hi,
>
> I think I may be onto something.
>
> When the memory is exhausted, the Linux kernel kills process randomly
> by SIGKILL.
> The mechanism is called OOM(Out Of Memory) killer.
> OOM killer logs its activity to kernel mesage buffer.
>
> Could you check display message (dmesg) and memory resource?
>
> Regards.
>
> On Tue, Nov 12, 2013 at 3:55 AM, Sandeep Gupta <gup...@gm...>
> wrote:
> > Hi Koichi,
> >
> >    It is a bit of mystery  because it does not happen consistently.
> Thanks
> > for the clarification though. It is indeed helpful.
> >
> > -Sandeep
> >
> >
> >
> >
> > On Mon, Nov 11, 2013 at 1:58 AM, 鈴木 幸市 <ko...@in...> wrote:
> >>
> >> Someone is sending SIGKILL to coordinator/datanode backend processes.
> >> Although XC (originally PG) code has a handler for SIGKILL, I didn’t
> found
> >> any code in XC sending SIGKILL to other processes.   I’m afraid there
> could
> >> be another process sending SIGKILL to them.
> >>
> >> Best Regards;
> >> ---
> >> Koichi Suzuki
> >>
> >> 2013/11/09 2:54、Sandeep Gupta <gup...@gm...> のメール：
> >>
> >> > I am running single instance of postgres-xc with  several datanode
> >> > Each data node runs on its own machine.
> >> > After instantiating the cluster the database sits idle. I do not
> perform
> >> > any create table or insert operations. However, after some time all
> the
> >> > datanode  automatically shuts down (log messages attached).
> >> >
> >> > Any pointers as to why this maybe happening would be very useful.
> >> >
> >> > -Sandeep
> >> >
> >> > The log files across all datanode contain various error messages.
> These
> >> > include:
> >> >
> >> >
> >> > LOG:  statistics collector process (PID 23073) was terminated by
> signal
> >> > 9: Killed
> >> > LOG:  WAL writer process (PID 23071) was terminated by signal 9:
> Killed
> >> > LOG:  terminating any other active server processes
> >> > WARNING:  terminating connection because of crash of another server
> >> > process
> >> > DETAIL:  The postmaster has commanded this server process to roll back
> >> > the current transaction and exit, because another server process
> exited
> >> > abnormally and possibly corrupted shared memory.
> >> > HINT:  In a moment you should be able to reconnect to the database and
> >> > repeat your command.
> >> > LOG:  all server processes terminated; reinitializing
> >> >
> >> >
> >> > LOG:  checkpointer process (PID 21406) was terminated by signal 9:
> >> > Killed
> >> > LOG:  terminating any other active server processes
> >> > WARNING:  terminating connection because of crash of another server
> >> > process
> >> > DETAIL:  The postmaster has commanded this server process to roll back
> >> > the current transaction and exit, because another server process
> exited
> >> > abnormally and possibly corrupted shared memory.
> >> > LOG:  statistics collector process (PID 28881) was terminated by
> signal
> >> > 9: Killed
> >> > LOG:  autovacuum launcher process (PID 28880) was terminated by signal
> >> > 9: Killed
> >> > LOG:  terminating any other active server processes
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> ------------------------------------------------------------------------------
> >> > November Webinars for C, C++, Fortran Developers
> >> > Accelerate application performance with scalable programming models.
> >> > Explore
> >> > techniques for threading, error checking, porting, and tuning. Get the
> >> > most
> >> > from the latest Intel processors and coprocessors. See abstracts and
> >> > register
> >> >
> >> >
> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk_______________________________________________
> >> > Postgres-xc-general mailing list
> >> > Pos...@li...
> >> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> >>
> >
> >
> >
> ------------------------------------------------------------------------------
> > November Webinars for C, C++, Fortran Developers
> > Accelerate application performance with scalable programming models.
> Explore
> > techniques for threading, error checking, porting, and tuning. Get the
> most
> > from the latest Intel processors and coprocessors. See abstracts and
> > register
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
> > _______________________________________________
> > Postgres-xc-general mailing list
> > Pos...@li...
> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> >
>

Hi,

I think I may be onto something.

When the memory is exhausted, the Linux kernel kills process randomly
by SIGKILL.
The mechanism is called OOM(Out Of Memory) killer.
OOM killer logs its activity to kernel mesage buffer.

Could you check display message (dmesg) and memory resource?

Regards.

On Tue, Nov 12, 2013 at 3:55 AM, Sandeep Gupta <gup...@gm...> wrote:
> Hi Koichi,
>
>    It is a bit of mystery  because it does not happen consistently. Thanks
> for the clarification though. It is indeed helpful.
>
> -Sandeep
>
>
>
>
> On Mon, Nov 11, 2013 at 1:58 AM, 鈴木 幸市 <ko...@in...> wrote:
>>
>> Someone is sending SIGKILL to coordinator/datanode backend processes.
>> Although XC (originally PG) code has a handler for SIGKILL, I didn’t found
>> any code in XC sending SIGKILL to other processes.   I’m afraid there could
>> be another process sending SIGKILL to them.
>>
>> Best Regards;
>> ---
>> Koichi Suzuki
>>
>> 2013/11/09 2:54、Sandeep Gupta <gup...@gm...> のメール：
>>
>> > I am running single instance of postgres-xc with  several datanode
>> > Each data node runs on its own machine.
>> > After instantiating the cluster the database sits idle. I do not perform
>> > any create table or insert operations. However, after some time all the
>> > datanode  automatically shuts down (log messages attached).
>> >
>> > Any pointers as to why this maybe happening would be very useful.
>> >
>> > -Sandeep
>> >
>> > The log files across all datanode contain various error messages. These
>> > include:
>> >
>> >
>> > LOG:  statistics collector process (PID 23073) was terminated by signal
>> > 9: Killed
>> > LOG:  WAL writer process (PID 23071) was terminated by signal 9: Killed
>> > LOG:  terminating any other active server processes
>> > WARNING:  terminating connection because of crash of another server
>> > process
>> > DETAIL:  The postmaster has commanded this server process to roll back
>> > the current transaction and exit, because another server process exited
>> > abnormally and possibly corrupted shared memory.
>> > HINT:  In a moment you should be able to reconnect to the database and
>> > repeat your command.
>> > LOG:  all server processes terminated; reinitializing
>> >
>> >
>> > LOG:  checkpointer process (PID 21406) was terminated by signal 9:
>> > Killed
>> > LOG:  terminating any other active server processes
>> > WARNING:  terminating connection because of crash of another server
>> > process
>> > DETAIL:  The postmaster has commanded this server process to roll back
>> > the current transaction and exit, because another server process exited
>> > abnormally and possibly corrupted shared memory.
>> > LOG:  statistics collector process (PID 28881) was terminated by signal
>> > 9: Killed
>> > LOG:  autovacuum launcher process (PID 28880) was terminated by signal
>> > 9: Killed
>> > LOG:  terminating any other active server processes
>> >
>> >
>> >
>> >
>> >
>> >
>> > ------------------------------------------------------------------------------
>> > November Webinars for C, C++, Fortran Developers
>> > Accelerate application performance with scalable programming models.
>> > Explore
>> > techniques for threading, error checking, porting, and tuning. Get the
>> > most
>> > from the latest Intel processors and coprocessors. See abstracts and
>> > register
>> >
>> > http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk_______________________________________________
>> > Postgres-xc-general mailing list
>> > Pos...@li...
>> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>
>
>
> ------------------------------------------------------------------------------
> November Webinars for C, C++, Fortran Developers
> Accelerate application performance with scalable programming models. Explore
> techniques for threading, error checking, porting, and tuning. Get the most
> from the latest Intel processors and coprocessors. See abstracts and
> register
> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>

Hello,

 

we want to use a postgres-xc cluster with two or more master nodes.

If we start the cluster configuration with a GTM, 2 GTM-proxys, 2
master-coordinators and 2 master-datanodes , we can initialize and use
it.

 

Now we want to add or remove datanodes while the database is still
running. 

If we add a new node, we add that one to the coordinators node list. It
seems the datanode now knows the database schema, but didn't contain any
data. The node is also in write protection mode.

If we remove an existing  datanode (shutdown its kvm and remove it from
the coordinators node list), now the database is not able to execute
queries.

 

We are using Postgres-XC 1.1 beta.

 

So, we would like to know which is the right way to add more new master
nodes to a running cluster and how can we use the cluster after one of
the nodes has crashed? 

 

Best regards,

 

Stephan Casper

Hi Koichi,

   It is a bit of mystery  because it does not happen consistently. Thanks
for the clarification though. It is indeed helpful.

-Sandeep




On Mon, Nov 11, 2013 at 1:58 AM, 鈴木 幸市 <ko...@in...> wrote:

> Someone is sending SIGKILL to coordinator/datanode backend processes.
>  Although XC (originally PG) code has a handler for SIGKILL, I didn’t found
> any code in XC sending SIGKILL to other processes.   I’m afraid there could
> be another process sending SIGKILL to them.
>
> Best Regards;
> ---
> Koichi Suzuki
>
> 2013/11/09 2:54、Sandeep Gupta <gup...@gm...> のメール：
>
> > I am running single instance of postgres-xc with  several datanode
> > Each data node runs on its own machine.
> > After instantiating the cluster the database sits idle. I do not perform
> any create table or insert operations. However, after some time all the
> datanode  automatically shuts down (log messages attached).
> >
> > Any pointers as to why this maybe happening would be very useful.
> >
> > -Sandeep
> >
> > The log files across all datanode contain various error messages. These
> include:
> >
> >
> > LOG:  statistics collector process (PID 23073) was terminated by signal
> 9: Killed
> > LOG:  WAL writer process (PID 23071) was terminated by signal 9: Killed
> > LOG:  terminating any other active server processes
> > WARNING:  terminating connection because of crash of another server
> process
> > DETAIL:  The postmaster has commanded this server process to roll back
> the current transaction and exit, because another server process exited
> abnormally and possibly corrupted shared memory.
> > HINT:  In a moment you should be able to reconnect to the database and
> repeat your command.
> > LOG:  all server processes terminated; reinitializing
> >
> >
> > LOG:  checkpointer process (PID 21406) was terminated by signal 9: Killed
> > LOG:  terminating any other active server processes
> > WARNING:  terminating connection because of crash of another server
> process
> > DETAIL:  The postmaster has commanded this server process to roll back
> the current transaction and exit, because another server process exited
> abnormally and possibly corrupted shared memory.
> > LOG:  statistics collector process (PID 28881) was terminated by signal
> 9: Killed
> > LOG:  autovacuum launcher process (PID 28880) was terminated by signal
> 9: Killed
> > LOG:  terminating any other active server processes
> >
> >
> >
> >
> >
> >
> ------------------------------------------------------------------------------
> > November Webinars for C, C++, Fortran Developers
> > Accelerate application performance with scalable programming models.
> Explore
> > techniques for threading, error checking, porting, and tuning. Get the
> most
> > from the latest Intel processors and coprocessors. See abstracts and
> register
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk_______________________________________________
> > Postgres-xc-general mailing list
> > Pos...@li...
> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
>

Hello, here is the process.

When we created table with DISTRIBUTE BY REPLICATION TO NODE ( datanode1 
), it has been inserting maybe 100 000 rows per minute.
When we created table with DISTRIBUTE BY REPLICATION TO NODE ( 
datanode1, datanode2 ), it has been inserting maybe 50 000 rows per minute.
When we created table witch  DISTRIBUTE BY MODULO ( cluster_id ) TO NODE 
( datanode1, datanode2 ) and has been writing to only one node, it was 
inserting 170 000 rows per minute.

But when we created it with partitioning, it has been 5000 rows per minute.

When we insert row into nonpartitioning table, no row was generated into 
the gtm log file. When we insert row into partitioning table, extra row 
is inserted into gtm log file : Saving transaction restoration info, 
backed-up gxid: 409215
LOCATION:  GTM_WriteRestorePointXid, gtm_txn.c:2649.

When deleting or updating from nonpartitioning table, everything is 
going quickly. But when we do the same thing with partitioning table, it 
is 3 - 4 times slower.

Below is example with partitioning. We are importing data to database, 
so first we insert unit_id into table units. This will generate schema 
for the year we are inserting, based on the column gps_timestamp. After 
that it create table for the unit in the schema, which inherits from 
primary talbe - units_data. Then we are inserting only to table units_data.

CREATE TABLE data.units ( unit_id uuid NOT NULL, cluster_id INT NOT NULL );

CREATE SEQUENCE ai_value;

CREATE TABLE data.units_data (
     row_id INTEGER NOT NULL DEFAULT nextval('ai_value'),
     unit_id uuid NOT NULL,
     cluster_id INT NOT NULL,
     gps_timestamp TIMESTAMP WITHOUT TIME ZONE NOT NULL,
     gps_lat DOUBLE PRECISION NOT NULL,
     gps_lng DOUBLE PRECISION NOT NULL,
     others hstore,
     PRIMARY KEY ( unit_id,cluster_id )
);

CREATE OR REPLACE FUNCTION generate_all(IN cluster_id INTEGER,  IN 
unit_id UUID) RETURNS VOID AS $$
DECLARE
     ROK VARCHAR(4);
     DATANODE_ID VARCHAR(1);
     NAZOV_SCHEMY VARCHAR(13);
     NAZOV_TABULKY VARCHAR;
     NAZOV_END_TABULKY VARCHAR;
     NAZOV_ROK_TABULKY VARCHAR;
BEGIN
     -- SELECT 2013 INTO ROK;
     ROK := 2013;
     SELECT cluster_id INTO DATANODE_ID;
     NAZOV_SCHEMY := 'node_'||DATANODE_ID||'_y_'||ROK;
     NAZOV_ROK_TABULKY := 'units_data_2013';
     NAZOV_TABULKY := 'u_'||replace(unit_id::text,'-','');
     NAZOV_END_TABULKY := NAZOV_TABULKY||'_m';
     -- Ide sa vytvarat schema a tabulky
         IF NOT EXISTS(SELECT * FROM information_schema.schemata WHERE 
schema_name=NAZOV_SCHEMY) THEN
             EXECUTE 'CREATE SCHEMA '||NAZOV_SCHEMY||';';
         END IF;
         IF NOT EXISTS(SELECT * FROM information_schema.tables where 
table_name=NAZOV_ROK_TABULKY AND table_schema=NAZOV_SCHEMY) THEN
             EXECUTE 'CREATE TABLE 
'||NAZOV_SCHEMY||'.'||NAZOV_ROK_TABULKY||'( CHECK ( 
date_part(''year''::text, gps_timestamp)=2013 ) ) INHERITS ( 
data.units_data );';
         END IF;
         EXECUTE 'CREATE OR REPLACE RULE 
"year2013_node'||DATANODE_ID||'_choice" AS ON INSERT TO data.units_data 
WHERE date_part(''year''::text, NEW.gps_timestamp)=2013 AND 
NEW.cluster_id='||DATANODE_ID||' DO INSTEAD INSERT INTO 
'||NAZOV_SCHEMY||'.units_data_2013 VALUES(NEW.*);';

         EXECUTE 'CREATE TABLE '||NAZOV_SCHEMY||'.'||NAZOV_TABULKY||'( 
CHECK ( unit_id='''||unit_id||'''::uuid ) ) INHERITS 
('||NAZOV_SCHEMY||'.'||NAZOV_ROK_TABULKY||' );';

         EXECUTE 'CREATE RULE "'||NAZOV_TABULKY||'_choice" AS ON INSERT 
TO '||NAZOV_SCHEMY||'.units_data_2013 WHERE unit_id='''||unit_id||''' DO 
INSTEAD INSERT INTO '||NAZOV_SCHEMY||'.'||NAZOV_TABULKY||' VALUES(NEW.*);';
     RETURN;
END
$$ LANGUAGE plpgsql;

CREATE RULE "generate_all_rule" AS ON INSERT TO data.units DO ALSO 
SELECT generate_all(NEW.cluster_id, NEW.unit_id);

Thank you, best regards
Matej


On 11. 11. 2013 10:11, 鈴木 幸市 wrote:
> This does not show anything special.   As I wrote, the messages in question don’t appear so often.     Could you test how many of these messages are written using another case, for example, without using partitioning.   I think partitioning table has nothing to do with the issue but I’d like to know if the case happens in other DMLs and DDLs.
>
> Thank you.
> ---
> Koichi Suzuki
>
> 2013/11/11 17:49、Matej Jellus <je...@ts...> のメール：
>
>> Our situation :
>> two machines
>> first - primary GTM, coordinator, datanode
>> second - standby GTM, coordinator, datanode
>>
>> GTM configuration below. We are inserting 5 thousand rows per minute, which is too low for us. We need to insert 100 thousand rows per minute.
>> Starting primary GTM : /gtm_ctl -Z gtm start -D /var/pgxc/data_gtm
>> Starting standby GTM : /gtm_ctl -Z gtm start -D /var/pgxc/data_gtm
>> Starting coord : pg_ctl start -D /var/pgxc/ubuntu2_coord -Z coordinator -l /tmp/logfile_ubuntu2_coord
>>
>> Tables in database are using partitioning. So we have one primary table and we are creating child tables from it. For example :
>> primary table :
>> chema - data
>> table - units_data
>>
>> child tables :
>> schema : y_2013_node1 - tables, which will be saved in node 1
>>    table : u_<unit_id> - child table from data.units_data
>> schema : y_2013_node2 - tables, which will be saved in node 2
>>    table : u_<unit_id> - child table from data.units_data
>>
>> Thank you, best regards
>> Matej
>>
>>
>> On 11. 11. 2013 7:49, 鈴木 幸市 wrote:
>>> Message 1. will be written once every 2000 transactions.    The number looks too large.   Message 1 is written in the three cases:
>>>
>>> 1) When GTM starts,
>>> 2) Once every 2000 transactions,
>>> 3) When GTM standby is promoted to a master.
>>>
>>> Could you let me know how you start your cluster?    We’ve tested XC with DBT-1 and DBT-2 but didn’t have such frequent write at gtm.   You are inserting 5000 raws in a second so 2) should happen twice or three times a second.
>>>
>>> Message 2. is written when GTM standby connects to GTM.   Are you using GTM standby?   If so, how you are starting it?
>>>
>>> We’re testing XC in different environment but didn’t have such GTM overload.   I’m interested in the situation.
>>>
>>> Best;
>>> ---
>>> Koichi Suzuki
>>>
>>> 2013/11/09 0:06、Matej Jellus <je...@ts...> のメール：
>>>
>>>> Hello,
>>>>
>>>> We have problem that postgres xc is very slow, gtm is writing to log a
>>>> lot of data. Is it able to be disabled?
>>>> The most repeated lines are :
>>>>
>>>> 1:140130870839040:2013-11-08 15:42:49.357 CET -LOG:  Saving transaction
>>>> restoration info, backed-up gxid: 373106
>>>> LOCATION:  GTM_WriteRestorePointXid, gtm_txn.c:2649
>>>> ( this is maybe 130 times per second )
>>>>
>>>> 1:140130879186688:2013-11-08 15:45:37.572 CET -LOG:  Connection
>>>> established with GTM standby. - 0x1f91398
>>>> LOCATION:  gtm_standby_connect_to_standby_int, gtm_standby.c:400
>>>>
>>>> It is overloading the disk, i/o is 70%.
>>>>
>>>> Now it is inserting 5 thousand inserts per minute. We need more inserts
>>>> per minute.
>>>>
>>>> Thank you, best regards
>>>> Matej
>>>>
>>>> ------------------------------------------------------------------------------
>>>> November Webinars for C, C++, Fortran Developers
>>>> Accelerate application performance with scalable programming models. Explore
>>>> techniques for threading, error checking, porting, and tuning. Get the most
>>>> from the latest Intel processors and coprocessors. See abstracts and register
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
>>>> _______________________________________________
>>>> Postgres-xc-general mailing list
>>>> Pos...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>>
>>

Someone is sending SIGKILL to coordinator/datanode backend processes.    Although XC (originally PG) code has a handler for SIGKILL, I didn’t found any code in XC sending SIGKILL to other processes.   I’m afraid there could be another process sending SIGKILL to them.

Best Regards;
---
Koichi Suzuki

2013/11/09 2:54、Sandeep Gupta <gup...@gm...> のメール：

> I am running single instance of postgres-xc with  several datanode
> Each data node runs on its own machine. 
> After instantiating the cluster the database sits idle. I do not perform any create table or insert operations. However, after some time all the datanode  automatically shuts down (log messages attached). 
> 
> Any pointers as to why this maybe happening would be very useful. 
> 
> -Sandeep
> 
> The log files across all datanode contain various error messages. These include:
> 
> 
> LOG:  statistics collector process (PID 23073) was terminated by signal 9: Killed
> LOG:  WAL writer process (PID 23071) was terminated by signal 9: Killed
> LOG:  terminating any other active server processes
> WARNING:  terminating connection because of crash of another server process
> DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
> HINT:  In a moment you should be able to reconnect to the database and repeat your command.
> LOG:  all server processes terminated; reinitializing
> 
> 
> LOG:  checkpointer process (PID 21406) was terminated by signal 9: Killed
> LOG:  terminating any other active server processes
> WARNING:  terminating connection because of crash of another server process
> DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
> LOG:  statistics collector process (PID 28881) was terminated by signal 9: Killed
> LOG:  autovacuum launcher process (PID 28880) was terminated by signal 9: Killed
> LOG:  terminating any other active server processes
> 
> 
> 
> 
> 
> ------------------------------------------------------------------------------
> November Webinars for C, C++, Fortran Developers
> Accelerate application performance with scalable programming models. Explore
> techniques for threading, error checking, porting, and tuning. Get the most 
> from the latest Intel processors and coprocessors. See abstracts and register
> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk_______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

Message 1. will be written once every 2000 transactions.    The number looks too large.   Message 1 is written in the three cases:

1) When GTM starts,
2) Once every 2000 transactions,
3) When GTM standby is promoted to a master.

Could you let me know how you start your cluster?    We’ve tested XC with DBT-1 and DBT-2 but didn’t have such frequent write at gtm.   You are inserting 5000 raws in a second so 2) should happen twice or three times a second.

Message 2. is written when GTM standby connects to GTM.   Are you using GTM standby?   If so, how you are starting it?

We’re testing XC in different environment but didn’t have such GTM overload.   I’m interested in the situation.

Best;
---
Koichi Suzuki

2013/11/09 0:06、Matej Jellus <je...@ts...> のメール：

> Hello,
> 
> We have problem that postgres xc is very slow, gtm is writing to log a 
> lot of data. Is it able to be disabled?
> The most repeated lines are :
> 
> 1:140130870839040:2013-11-08 15:42:49.357 CET -LOG:  Saving transaction 
> restoration info, backed-up gxid: 373106
> LOCATION:  GTM_WriteRestorePointXid, gtm_txn.c:2649
> ( this is maybe 130 times per second )
> 
> 1:140130879186688:2013-11-08 15:45:37.572 CET -LOG:  Connection 
> established with GTM standby. - 0x1f91398
> LOCATION:  gtm_standby_connect_to_standby_int, gtm_standby.c:400
> 
> It is overloading the disk, i/o is 70%.
> 
> Now it is inserting 5 thousand inserts per minute. We need more inserts 
> per minute.
> 
> Thank you, best regards
> Matej
> 
> ------------------------------------------------------------------------------
> November Webinars for C, C++, Fortran Developers
> Accelerate application performance with scalable programming models. Explore
> techniques for threading, error checking, porting, and tuning. Get the most 
> from the latest Intel processors and coprocessors. See abstracts and register
> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> 

I am running single instance of postgres-xc with  several datanode
Each data node runs on its own machine.
After instantiating the cluster the database sits idle. I do not perform
any create table or insert operations. However, after some time all the
datanode  automatically shuts down (log messages attached).

Any pointers as to why this maybe happening would be very useful.

-Sandeep

The log files across all datanode contain various error messages. These
include:


LOG:  statistics collector process (PID 23073) was terminated by signal 9:
Killed
LOG:  WAL writer process (PID 23071) was terminated by signal 9: Killed
LOG:  terminating any other active server processes
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the
current transaction and exit, because another server process exited
abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and
repeat your command.
LOG:  all server processes terminated; reinitializing


LOG:  checkpointer process (PID 21406) was terminated by signal 9: Killed
LOG:  terminating any other active server processes
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the
current transaction and exit, because another server process exited
abnormally and possibly corrupted shared memory.
LOG:  statistics collector process (PID 28881) was terminated by signal 9:
Killed
LOG:  autovacuum launcher process (PID 28880) was terminated by signal 9:
Killed
LOG:  terminating any other active server processes

Hello,

We have problem that postgres xc is very slow, gtm is writing to log a 
lot of data. Is it able to be disabled?
The most repeated lines are :

1:140130870839040:2013-11-08 15:42:49.357 CET -LOG:  Saving transaction 
restoration info, backed-up gxid: 373106
LOCATION:  GTM_WriteRestorePointXid, gtm_txn.c:2649
( this is maybe 130 times per second )

1:140130879186688:2013-11-08 15:45:37.572 CET -LOG:  Connection 
established with GTM standby. - 0x1f91398
LOCATION:  gtm_standby_connect_to_standby_int, gtm_standby.c:400

It is overloading the disk, i/o is 70%.

Now it is inserting 5 thousand inserts per minute. We need more inserts 
per minute.

Thank you, best regards
Matej

If you find any inconsistency in replicated table replica, here's another means to repair them.   If you find inconsistent replica on datanode D, you can issue ALTER TABLE twice to repair them.

ALTER TABLE T DROP NODE D;
ALTER TABLE T ADD NODE D;

If you have more than one inconsistent replica, you should drop all such replica with ALTER TABLE DROP NODE and then repair them by ALTER TABLE ADD NODE.

This is available in release 1.1 or later.

Regards;
---
Koichi Suzuki

On 2013/11/07, at 2:16, Alex Hudson <ale...@au...<mailto:ale...@au...>> wrote:

Thanks a lot Koichi!!

We are working with replicated tables in a 3 nodes cluster. We’ll use "EXECUTE DIRECT" statement to be sure if all the replica are identical.

Thanks and Regards!

Alex Hudson S.

De: Koichi Suzuki [mailto:koi...@gm...<http://gmail.com>]
Enviado el: martes, 05 de noviembre de 2013 22:48
Para: Alex Hudson
CC: Postgres-XC mailing list
Asunto: Re: [Postgres-xc-general] consistency check on SPI tuple count failed

Thanks a lot for the report.

Could you let me know if you got this message while handling replicated table or distributed table?

Most probable case is inconsistency in replicated table's replica.   Before looking into it, could you check if there's no outstanding 2PC transactions in your datanodes.   Even though you don't issue 2PC commands, XC may issue it internally when more than one node are involved in updating transactions and you may have such outstanding 2PC transactions especially when some node fails (did you have such failure?).

You can check this by selecting pg_prepared_statements catalog.   This catalog is local to each coordinator and datanode so you should issue select statement using EXECUTE DIRECT statement.   If you find such outstanding 2PC transactions, you can rollback them as follows:

1. Connect psql directly to a target datanode/coordinator as a superuser.
2. Issue "SET xc_maintenance_mode to ON" to enable local manual repair.
3. Issue ROLLBACK PREPARED statement to rollback such outstanding 2PCs.
4. Repeat 1 to 3 for all the coordinator/datanodes which contains such outstanding 2PCs.

Postgres-XC provides pgxc_clean utility to do the above step.   This will be found in contrib directory of the source tarball.

If you don't have such outstanding 2PC transactions or you still have the same error, please try the following.

If you are handling replicated tables, we have a chance that there's some inconsistencies in each replica.   To check this, use "EXECUTE DIRECT" statement to visit each datanode and check if all the replica are identical.   If not, you can recover damaged tuples as follows:

1. Connect psql directly to a target datanode as a superuser.
2. Issue "SET xc_maintenance_mode to ON" to enable local manual repair.
3. Issue SQL statement to repair damaged tuples.
4. Repeat 1 to 3 for all the damaged datanode.

If you've got the error while handling distributed tables, there could be another error cause.   I need to analylze the case more in detail.   Please let me know if you are handling distributed tables.

Best Regards;

---
Koichi Suzuki

2013/11/6 Alex Hudson <ale...@au...<mailto:ale...@au...>>
Hi guys,

I’m working in an application using php and Postgrexc clustered.
In the event of save some information, the app shows me the following lines:

Application raised an exception class <b>ADODB_Exception</b> with message <b>'postgres7 error: [-1: ERROR:  consistency check on SPI tuple count failed

Why produces this error? How can i fix it? May be because I’m using  encoding LATIN1?

Thanks and regards,

Alex Hudson S.


------------------------------------------------------------------------------
November Webinars for C, C++, Fortran Developers
Accelerate application performance with scalable programming models. Explore
techniques for threading, error checking, porting, and tuning. Get the most
from the latest Intel processors and coprocessors. See abstracts and register
http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
_______________________________________________
Postgres-xc-general mailing list
Pos...@li...<mailto:Pos...@li...>
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

------------------------------------------------------------------------------
November Webinars for C, C++, Fortran Developers
Accelerate application performance with scalable programming models. Explore
techniques for threading, error checking, porting, and tuning. Get the most
from the latest Intel processors and coprocessors. See abstracts and register
http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk_______________________________________________
Postgres-xc-general mailing list
Pos...@li...
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

Thanks a lot Koichi!! 

 

We are working with replicated tables in a 3 nodes cluster. We'll use
"EXECUTE DIRECT" statement to be sure if all the replica are identical. 

 

Thanks and Regards!

 

Alex Hudson S.

 

De: Koichi Suzuki [mailto:koi...@gm...] 
Enviado el: martes, 05 de noviembre de 2013 22:48
Para: Alex Hudson
CC: Postgres-XC mailing list
Asunto: Re: [Postgres-xc-general] consistency check on SPI tuple count
failed

 

Thanks a lot for the report.

 

Could you let me know if you got this message while handling replicated
table or distributed table?

 

Most probable case is inconsistency in replicated table's replica.   Before
looking into it, could you check if there's no outstanding 2PC transactions
in your datanodes.   Even though you don't issue 2PC commands, XC may issue
it internally when more than one node are involved in updating transactions
and you may have such outstanding 2PC transactions especially when some node
fails (did you have such failure?).

 

You can check this by selecting pg_prepared_statements catalog.   This
catalog is local to each coordinator and datanode so you should issue select
statement using EXECUTE DIRECT statement.   If you find such outstanding 2PC
transactions, you can rollback them as follows:

 

1. Connect psql directly to a target datanode/coordinator as a superuser.

2. Issue "SET xc_maintenance_mode to ON" to enable local manual repair.

3. Issue ROLLBACK PREPARED statement to rollback such outstanding 2PCs.

4. Repeat 1 to 3 for all the coordinator/datanodes which contains such
outstanding 2PCs.

 

Postgres-XC provides pgxc_clean utility to do the above step.   This will be
found in contrib directory of the source tarball.

 

If you don't have such outstanding 2PC transactions or you still have the
same error, please try the following.

 

If you are handling replicated tables, we have a chance that there's some
inconsistencies in each replica.   To check this, use "EXECUTE DIRECT"
statement to visit each datanode and check if all the replica are identical.
If not, you can recover damaged tuples as follows:

 

1. Connect psql directly to a target datanode as a superuser.

2. Issue "SET xc_maintenance_mode to ON" to enable local manual repair.

3. Issue SQL statement to repair damaged tuples.

4. Repeat 1 to 3 for all the damaged datanode.

 

If you've got the error while handling distributed tables, there could be
another error cause.   I need to analylze the case more in detail.   Please
let me know if you are handling distributed tables.

 

Best Regards;




---

Koichi Suzuki

 

2013/11/6 Alex Hudson <ale...@au...>

Hi guys,

 

I'm working in an application using php and Postgrexc clustered.

In the event of save some information, the app shows me the following lines:

 

Application raised an exception class <b>ADODB_Exception</b> with message
<b>'postgres7 error: [-1: ERROR:  consistency check on SPI tuple count
failed

 

Why produces this error? How can i fix it? May be because I'm using
encoding LATIN1?

 

Thanks and regards,

 

Alex Hudson S.

 


----------------------------------------------------------------------------
--
November Webinars for C, C++, Fortran Developers
Accelerate application performance with scalable programming models. Explore
techniques for threading, error checking, porting, and tuning. Get the most
from the latest Intel processors and coprocessors. See abstracts and
register
http://pubads.g.doubleclick.net/gampad/clk?id=60136231
<http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
> &iu=/4140/ostg.clktrk
_______________________________________________
Postgres-xc-general mailing list
Pos...@li...
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

 

Thanks a lot for the report.

Could you let me know if you got this message while handling replicated
table or distributed table?

Most probable case is inconsistency in replicated table's replica.   Before
looking into it, could you check if there's no outstanding 2PC transactions
in your datanodes.   Even though you don't issue 2PC commands, XC may issue
it internally when more than one node are involved in updating transactions
and you may have such outstanding 2PC transactions especially when some
node fails (did you have such failure?).

You can check this by selecting pg_prepared_statements catalog.   This
catalog is local to each coordinator and datanode so you should issue
select statement using EXECUTE DIRECT statement.   If you find such
outstanding 2PC transactions, you can rollback them as follows:

1. Connect psql directly to a target datanode/coordinator as a superuser.
2. Issue "SET xc_maintenance_mode to ON" to enable local manual repair.
3. Issue ROLLBACK PREPARED statement to rollback such outstanding 2PCs.
4. Repeat 1 to 3 for all the coordinator/datanodes which contains such
outstanding 2PCs.

Postgres-XC provides pgxc_clean utility to do the above step.   This will
be found in contrib directory of the source tarball.

If you don't have such outstanding 2PC transactions or you still have the
same error, please try the following.

If you are handling replicated tables, we have a chance that there's some
inconsistencies in each replica.   To check this, use "EXECUTE DIRECT"
statement to visit each datanode and check if all the replica are
identical.   If not, you can recover damaged tuples as follows:

1. Connect psql directly to a target datanode as a superuser.
2. Issue "SET xc_maintenance_mode to ON" to enable local manual repair.
3. Issue SQL statement to repair damaged tuples.
4. Repeat 1 to 3 for all the damaged datanode.

If you've got the error while handling distributed tables, there could be
another error cause.   I need to analylze the case more in detail.   Please
let me know if you are handling distributed tables.

Best Regards;

---
Koichi Suzuki


2013/11/6 Alex Hudson <ale...@au...>

> Hi guys,
>
>
>
> I’m working in an application using php and Postgrexc clustered.
>
> In the event of save some information, the app shows me the following
> lines:
>
>
>
> Application raised an exception class <b>ADODB_Exception</b> with message
> <b>'postgres7 error: [-1: ERROR:  consistency check on SPI tuple count
> failed
>
>
>
> Why produces this error? How can i fix it? May be because I’m using
> encoding LATIN1?
>
>
>
> Thanks and regards,
>
>
>
> Alex Hudson S.
>
>
>
>
> ------------------------------------------------------------------------------
> November Webinars for C, C++, Fortran Developers
> Accelerate application performance with scalable programming models.
> Explore
> techniques for threading, error checking, porting, and tuning. Get the most
> from the latest Intel processors and coprocessors. See abstracts and
> register
> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
>

Hi guys,

 

I'm working in an application using php and Postgrexc clustered.

In the event of save some information, the app shows me the following lines:

 

Application raised an exception class <b>ADODB_Exception</b> with message
<b>'postgres7 error: [-1: ERROR:  consistency check on SPI tuple count
failed

 

Why produces this error? How can i fix it? May be because I'm using
encoding LATIN1?

 

Thanks and regards,

 

Alex Hudson S.

 

Thanks Ashutosh for the correct answer.    So far, Postgres-XC allows constraints which can be enforced without additional infrastructure, I mean, without visiting other datanodes to check if the constraint is maintained.    I'm afraid Alex referred to a distributed table from replicated tables, which cannot be enforced with the current XC infrastructure.     Of course, the opposite should work, referring replicated tables from a distributed table.

Regards;
---
Koichi Suzuki

On 2013/10/29, at 15:29, Ashutosh Bapat <ash...@en...<mailto:ash...@en...>>
 wrote:

Hi Alex,
The error comes because, the referenced key is not guaranteed to on the same node where the row referencing it lies. Thus either of following should hold for adding a foreign key.
1. Both the tables (referenced and referencer) should be replicated on the same set of nodes
2. Both the tables should be distributed by in the same fashion on referenced and referencer columns resp.
3. The referencer table is distributed and referenced table is replicated on all the nodes where the referencer table is distributed.


Koichi Suzuki says that this is current XC limitation, I tried to Change the tables distribution to "replication". I can change the layout, but the problem is when I want to add the foreign key, I get the same error. How can I do to have the keys in my system?

The system has at least 10 nodes…



Regards,

Alex Hudson S.


------------------------------------------------------------------------------
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
_______________________________________________
Postgres-xc-general mailing list
Pos...@li...<mailto:Pos...@li...>
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general




--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
------------------------------------------------------------------------------
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk_______________________________________________
Postgres-xc-general mailing list
Pos...@li...
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

Hi Alex,
The error comes because, the referenced key is not guaranteed to on the
same node where the row referencing it lies. Thus either of following
should hold for adding a foreign key.
1. Both the tables (referenced and referencer) should be replicated on the
same set of nodes
2. Both the tables should be distributed by in the same fashion on
referenced and referencer columns resp.
3. The referencer table is distributed and referenced table is replicated
on all the nodes where the referencer table is distributed.

Koichi Suzuki says that this is current XC limitation, I tried to
Change the tables distribution to "replication". I can change the
layout, but the problem is when I want to add the foreign key, I get
the same error. How can I do to have the keys in my system?****
>
> The system has at least 10 nodes… ****
>
> ** **
>
> Regards,****
>
> ** **
>
> Alex Hudson S.****
>
> ** **
>
>
> ------------------------------------------------------------------------------
> Android is increasing in popularity, but the open development platform that
> developers love is also attractive to malware creators. Download this white
> paper to learn more about secure code signing practices that can help keep
> Android apps secure.
> http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
>


-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Hi guys,

 

I could really use your help. 
I'm a sysadmin. I'm not a DBA, and I don't have any experience with
PostgreSQL.
 
I'm trying PGXC as a Cluster solution for the PGSQL DB Component in our
Car's assistance infrastructure.
The problem I'm currently having, is that when I try to set a foreign key in
any table.
 
"ERROR:  Cannot create foreign key whose evaluation cannot be enforced to
remote nodes"
 
Koichi Suzuki says that this is current XC limitation, I tried to Change the
tables distribution to "replication". I can change the layout, but the
problem is when I want to add the foreign key, I get the same error. How can
I do to have the keys in my system?
The system has at least 10 nodes. 
 
Regards,

 

Alex Hudson S.

 

Ashutosh,

 First, just wanted to confirm about the streaming aspect. There are two
parts to this: one the streaming aggregate operators and making query
planning cognizant about it.
As I said, the whole query planner and how to extend it is not so clear.
You did  give some pointers about how to go about doing this. However,
someone with not complete knowledge of the planner subsystem would have
still have diffculty.

I can help with the documentation first about how to extend the planner if
you are interested and can give  pointers in codebase which needs to
touched in order to achieve this.

-Sandeep





On Thu, Oct 17, 2013 at 8:19 AM, Ashutosh Bapat <
ash...@en...> wrote:

> That sounds a good idea. Can you please provide a patch?
>
>
> On Thu, Oct 17, 2013 at 5:41 PM, Sandeep Gupta <gup...@gm...>wrote:
>
>>  I maybe missing something. Is it the case that postgres or pgxc doesn't
>> support
>> streaming aggregates? That would allow aggregation over sorted streams to
>> produce a
>> aggregated stream that is also sorted.
>> However, I am not too sure about this.
>>
>> -Sandeep
>>
>>
>>
>>
>> On 10/17/2013 07:13 AM, Ashutosh Bapat wrote:
>>
>> That's not possible right now. For grouping, either the grouped input
>> from the datanode gets shuffled at the coordinator or ordered on the
>> grouping column. Thus even if we get the ordered intput from datanode on a
>> particular column that order is disturbed at the coordinator because of
>> grouping.
>>
>>
>> On Thu, Oct 17, 2013 at 4:37 PM, Sandeep Gupta <gup...@gm...>wrote:
>>
>>>  How about a plan where the datanodes perform the sort as well and the
>>> coordinator performs a sorted merge?
>>>
>>> Are such plans not part of the query planner?
>>>
>>> -Sandeep
>>>
>>>
>>> On 10/17/2013 07:04 AM, Ashutosh Bapat wrote:
>>>
>>> There is GROUP BY clause that needs to be evaluated before the result
>>> can be ordered. Thus GROUP BY is sent to the datanode but not ORDER BY.
>>>
>>>
>>>  On Thu, Oct 17, 2013 at 4:31 PM, Sandeep Gupta <gup...@gm...
>>> > wrote:
>>>
>>>>  Hi Ashutosh,
>>>>
>>>>
>>>>    Attached below is the query and the corresponding query plan. I am
>>>> using version 1.1.
>>>>  Thanks for taking a look at this.
>>>>
>>>>  -Sandeep
>>>>
>>>> SELECT exposed_time effectedDate, ROUND(COUNT(a.pid)/10) COUNT FROM public.vt_demography_info_xc d, public.ses_vt_20130805_xc a WHERE d.pid=a.pid AND d.countyid='50015' AND d.age BETWEEN 5 AND 18 AND d.gender=1 GROUP BY exposed_time ORDER BY exposed_time;
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>                                      QUERY PLAN (Coordinator)
>>>>
>>>>  Sort  (cost=10000000005.03..10000000005.03 rows=1 width=8)
>>>>    Output: a.exposed_time, (round(((count((count(a.pid))) / 10))::double precision))
>>>>    Sort Key: a.exposed_time
>>>>    ->  HashAggregate  (cost=5.00..5.02 rows=1 width=8)
>>>>          Output: a.exposed_time, round(((count((count(a.pid))) / 10))::double precision)
>>>>          ->  Data Node Scan on "__REMOTE_GROUP_QUERY__"  (cost=0.00..0.00 rows=1000 width=8)
>>>>                Output: a.exposed_time, (count(a.pid))
>>>>                Node/s: datanode1, datanode10, datanode11, datanode12, datanode13, datanode14, datanode15, datanode16, datanode2, datanode3, datanode4, datanode5, datanode6, datanode7, datanode8, datanode9
>>>>                Remote query: SELECT r.a_1, count(r.a_2) FROM ((SELECT d.pid FROM ONLY public.vt_demography_info_xc d WHERE ((d.age >= 5) AND (d.age <= 18) AND ((d.countyid)::text = '50015'::text) AND (d.gender = 1))) l(a_1) JOIN (SELECT a.exposed_time, a.pid FROM ONLY public.ses_vt_20130805_xc a WHERE ((a.exposed_time >= 4667) AND (a.exposed_time <= 5031))) r(a_1, a_2) ON (true)) WHERE (l.a_1 = r.a_2) GROUP BY 1
>>>> (9 rows)
>>>>
>>>>                                      QUERY PLAN (Datanode)
>>>>
>>>>  GroupAggregate  (cost=0.00..47862.29 rows=225 width=8)
>>>>    Output: a.exposed_time, round(((count(a.pid) / 10))::double precision)
>>>>    ->  Nested Loop  (cost=0.00..47856.05 rows=460 width=8)
>>>>          Output: a.exposed_time, a.pid
>>>>          ->  Index Scan using et_ses on public.ses_vt_20130805_xc a  (cost=0.00..7283.10 rows=129583 width=8)
>>>>                Output: a.pid, a.rep, a.exposed_time, a.infectious_time, a.recovered_time
>>>>                Index Cond: ((a.exposed_time >= 4667) AND (a.exposed_time <= 5031))
>>>>          ->  Index Scan using pid_demo on public.vt_demography_info_xc d  (cost=0.00..0.30 rows=1 width=4)
>>>>                Output: d.pid, d.hid, d.age, d.gender, d.zipode, d.blockgroupid, d.longitude, d.lattitude, d.county, d.countyid
>>>>                Index Cond: (d.pid = a.pid)
>>>>                Filter: ((d.age >= 5) AND (d.age <= 18) AND ((d.countyid)::text = '50015'::text) AND (d.gender = 1))
>>>> (11 rows)
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Oct 17, 2013 at 12:12 AM, Ashutosh Bapat <
>>>> ash...@en...> wrote:
>>>>
>>>>> Sandeep,
>>>>> It would be nice if you mention the version of XC in your mail. Sort
>>>>> push down is available from 1.1 onwards. If you do not see sort getting
>>>>> pushed down in 1.1, please report detailed definitions of the tables, query
>>>>> and the EXPLAIN output.
>>>>>
>>>>>
>>>>>  On Thu, Oct 17, 2013 at 1:09 AM, Sandeep Gupta <
>>>>> gup...@gm...> wrote:
>>>>>
>>>>>>    Hi,
>>>>>>
>>>>>>   In an another query that requires the result to be aggregated and
>>>>>> ordered by a field (lets say timeo)
>>>>>> the query planner currently pulls  the results and then performs a
>>>>>> sort with hash aggregate.
>>>>>>
>>>>>>  The table at the datanodes are clustered by timeo. I was wondering
>>>>>> if it possible
>>>>>> for query planner to push down the order by clause at the datanode
>>>>>> and then perform
>>>>>>  sort-merge aggregate at the coordinator. Surely, that would be a
>>>>>> better query plan.
>>>>>>
>>>>>>  We have tried enable_sort=off etc. but that doesn't work.
>>>>>>
>>>>>>  Thanks.
>>>>>>  Sandeep
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------------------------
>>>>>> October Webinars: Code for Performance
>>>>>> Free Intel webinars can help you accelerate application performance.
>>>>>> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the
>>>>>> most from
>>>>>> the latest Intel processors and coprocessors. See abstracts and
>>>>>> register >
>>>>>>
>>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=60135031&iu=/4140/ostg.clktrk
>>>>>> _______________________________________________
>>>>>> Postgres-xc-general mailing list
>>>>>> Pos...@li...
>>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Wishes,
>>>>> Ashutosh Bapat
>>>>> EnterpriseDB Corporation
>>>>> The Postgres Database Company
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Best Wishes,
>>> Ashutosh Bapat
>>> EnterpriseDB Corporation
>>> The Postgres Database Company
>>>
>>>
>>>
>>
>>
>> --
>> Best Wishes,
>> Ashutosh Bapat
>> EnterpriseDB Corporation
>> The Postgres Database Company
>>
>>
>>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EnterpriseDB Corporation
> The Postgres Database Company
>

That sounds a good idea. Can you please provide a patch?


On Thu, Oct 17, 2013 at 5:41 PM, Sandeep Gupta <gup...@gm...>wrote:

>  I maybe missing something. Is it the case that postgres or pgxc doesn't
> support
> streaming aggregates? That would allow aggregation over sorted streams to
> produce a
> aggregated stream that is also sorted.
> However, I am not too sure about this.
>
> -Sandeep
>
>
>
>
> On 10/17/2013 07:13 AM, Ashutosh Bapat wrote:
>
> That's not possible right now. For grouping, either the grouped input from
> the datanode gets shuffled at the coordinator or ordered on the grouping
> column. Thus even if we get the ordered intput from datanode on a
> particular column that order is disturbed at the coordinator because of
> grouping.
>
>
> On Thu, Oct 17, 2013 at 4:37 PM, Sandeep Gupta <gup...@gm...>wrote:
>
>>  How about a plan where the datanodes perform the sort as well and the
>> coordinator performs a sorted merge?
>>
>> Are such plans not part of the query planner?
>>
>> -Sandeep
>>
>>
>> On 10/17/2013 07:04 AM, Ashutosh Bapat wrote:
>>
>> There is GROUP BY clause that needs to be evaluated before the result can
>> be ordered. Thus GROUP BY is sent to the datanode but not ORDER BY.
>>
>>
>>  On Thu, Oct 17, 2013 at 4:31 PM, Sandeep Gupta <gup...@gm...>wrote:
>>
>>>  Hi Ashutosh,
>>>
>>>
>>>    Attached below is the query and the corresponding query plan. I am
>>> using version 1.1.
>>>  Thanks for taking a look at this.
>>>
>>>  -Sandeep
>>>
>>> SELECT exposed_time effectedDate, ROUND(COUNT(a.pid)/10) COUNT FROM public.vt_demography_info_xc d, public.ses_vt_20130805_xc a WHERE d.pid=a.pid AND d.countyid='50015' AND d.age BETWEEN 5 AND 18 AND d.gender=1 GROUP BY exposed_time ORDER BY exposed_time;
>>>
>>>
>>>
>>>
>>>
>>>                                      QUERY PLAN (Coordinator)
>>>
>>>  Sort  (cost=10000000005.03..10000000005.03 rows=1 width=8)
>>>    Output: a.exposed_time, (round(((count((count(a.pid))) / 10))::double precision))
>>>    Sort Key: a.exposed_time
>>>    ->  HashAggregate  (cost=5.00..5.02 rows=1 width=8)
>>>          Output: a.exposed_time, round(((count((count(a.pid))) / 10))::double precision)
>>>          ->  Data Node Scan on "__REMOTE_GROUP_QUERY__"  (cost=0.00..0.00 rows=1000 width=8)
>>>                Output: a.exposed_time, (count(a.pid))
>>>                Node/s: datanode1, datanode10, datanode11, datanode12, datanode13, datanode14, datanode15, datanode16, datanode2, datanode3, datanode4, datanode5, datanode6, datanode7, datanode8, datanode9
>>>                Remote query: SELECT r.a_1, count(r.a_2) FROM ((SELECT d.pid FROM ONLY public.vt_demography_info_xc d WHERE ((d.age >= 5) AND (d.age <= 18) AND ((d.countyid)::text = '50015'::text) AND (d.gender = 1))) l(a_1) JOIN (SELECT a.exposed_time, a.pid FROM ONLY public.ses_vt_20130805_xc a WHERE ((a.exposed_time >= 4667) AND (a.exposed_time <= 5031))) r(a_1, a_2) ON (true)) WHERE (l.a_1 = r.a_2) GROUP BY 1
>>> (9 rows)
>>>
>>>                                      QUERY PLAN (Datanode)
>>>
>>>  GroupAggregate  (cost=0.00..47862.29 rows=225 width=8)
>>>    Output: a.exposed_time, round(((count(a.pid) / 10))::double precision)
>>>    ->  Nested Loop  (cost=0.00..47856.05 rows=460 width=8)
>>>          Output: a.exposed_time, a.pid
>>>          ->  Index Scan using et_ses on public.ses_vt_20130805_xc a  (cost=0.00..7283.10 rows=129583 width=8)
>>>                Output: a.pid, a.rep, a.exposed_time, a.infectious_time, a.recovered_time
>>>                Index Cond: ((a.exposed_time >= 4667) AND (a.exposed_time <= 5031))
>>>          ->  Index Scan using pid_demo on public.vt_demography_info_xc d  (cost=0.00..0.30 rows=1 width=4)
>>>                Output: d.pid, d.hid, d.age, d.gender, d.zipode, d.blockgroupid, d.longitude, d.lattitude, d.county, d.countyid
>>>                Index Cond: (d.pid = a.pid)
>>>                Filter: ((d.age >= 5) AND (d.age <= 18) AND ((d.countyid)::text = '50015'::text) AND (d.gender = 1))
>>> (11 rows)
>>>
>>>
>>>
>>>
>>> On Thu, Oct 17, 2013 at 12:12 AM, Ashutosh Bapat <
>>> ash...@en...> wrote:
>>>
>>>> Sandeep,
>>>> It would be nice if you mention the version of XC in your mail. Sort
>>>> push down is available from 1.1 onwards. If you do not see sort getting
>>>> pushed down in 1.1, please report detailed definitions of the tables, query
>>>> and the EXPLAIN output.
>>>>
>>>>
>>>>  On Thu, Oct 17, 2013 at 1:09 AM, Sandeep Gupta <
>>>> gup...@gm...> wrote:
>>>>
>>>>>    Hi,
>>>>>
>>>>>   In an another query that requires the result to be aggregated and
>>>>> ordered by a field (lets say timeo)
>>>>> the query planner currently pulls  the results and then performs a
>>>>> sort with hash aggregate.
>>>>>
>>>>>  The table at the datanodes are clustered by timeo. I was wondering if
>>>>> it possible
>>>>> for query planner to push down the order by clause at the datanode and
>>>>> then perform
>>>>>  sort-merge aggregate at the coordinator. Surely, that would be a
>>>>> better query plan.
>>>>>
>>>>>  We have tried enable_sort=off etc. but that doesn't work.
>>>>>
>>>>>  Thanks.
>>>>>  Sandeep
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> October Webinars: Code for Performance
>>>>> Free Intel webinars can help you accelerate application performance.
>>>>> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the
>>>>> most from
>>>>> the latest Intel processors and coprocessors. See abstracts and
>>>>> register >
>>>>>
>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=60135031&iu=/4140/ostg.clktrk
>>>>> _______________________________________________
>>>>> Postgres-xc-general mailing list
>>>>> Pos...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Best Wishes,
>>>> Ashutosh Bapat
>>>> EnterpriseDB Corporation
>>>> The Postgres Database Company
>>>>
>>>
>>>
>>
>>
>> --
>> Best Wishes,
>> Ashutosh Bapat
>> EnterpriseDB Corporation
>> The Postgres Database Company
>>
>>
>>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EnterpriseDB Corporation
> The Postgres Database Company
>
>
>


-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

I maybe missing something. Is it the case that postgres or pgxc doesn't 
support
streaming aggregates? That would allow aggregation over sorted streams 
to produce a
aggregated stream that is also sorted.
However, I am not too sure about this.

-Sandeep



On 10/17/2013 07:13 AM, Ashutosh Bapat wrote:
> That's not possible right now. For grouping, either the grouped input 
> from the datanode gets shuffled at the coordinator or ordered on the 
> grouping column. Thus even if we get the ordered intput from datanode 
> on a particular column that order is disturbed at the coordinator 
> because of grouping.
>
>
> On Thu, Oct 17, 2013 at 4:37 PM, Sandeep Gupta 
> <gup...@gm... <mailto:gup...@gm...>> wrote:
>
>     How about a plan where the datanodes perform the sort as well and the
>     coordinator performs a sorted merge?
>
>     Are such plans not part of the query planner?
>
>     -Sandeep
>
>
>     On 10/17/2013 07:04 AM, Ashutosh Bapat wrote:
>>     There is GROUP BY clause that needs to be evaluated before the
>>     result can be ordered. Thus GROUP BY is sent to the datanode but
>>     not ORDER BY.
>>
>>
>>     On Thu, Oct 17, 2013 at 4:31 PM, Sandeep Gupta
>>     <gup...@gm... <mailto:gup...@gm...>> wrote:
>>
>>         Hi Ashutosh,
>>
>>
>>           Attached below is the query and the corresponding query
>>         plan. I am using version 1.1.
>>         Thanks for taking a look at this.
>>
>>         -Sandeep
>>
>>         SELECT exposed_time effectedDate, ROUND(COUNT(a.pid)/10) COUNT FROM public.vt_demography_info_xc d, public.ses_vt_20130805_xc a WHERE d.pid=a.pid AND d.countyid='50015' AND d.age BETWEEN 5 AND 18 AND d.gender=1 GROUP BY exposed_time ORDER BY exposed_time;
>>
>>
>>
>>
>>                                             QUERY PLAN (Coordinator)
>>
>>           Sort  (cost=10000000005.03..10000000005.03 rows=1 width=8)
>>             Output: a.exposed_time, (round(((count((count(a.pid))) / 10))::double precision))
>>             Sort Key: a.exposed_time
>>             ->  HashAggregate  (cost=5.00..5.02 rows=1 width=8)
>>                   Output: a.exposed_time, round(((count((count(a.pid))) / 10))::double precision)
>>                   ->  Data Node Scan on "__REMOTE_GROUP_QUERY__"  (cost=0.00..0.00 rows=1000 width=8)
>>                         Output: a.exposed_time, (count(a.pid))
>>                         Node/s: datanode1, datanode10, datanode11, datanode12, datanode13, datanode14, datanode15, datanode16, datanode2, datanode3, datanode4, datanode5, datanode6, datanode7, datanode8, datanode9
>>                         Remote query: SELECT r.a_1, count(r.a_2) FROM ((SELECT d.pid FROM ONLY public.vt_demography_info_xc d WHERE ((d.age >= 5) AND (d.age <= 18) AND ((d.countyid)::text = '50015'::text) AND (d.gender = 1))) l(a_1) JOIN (SELECT a.exposed_time, a.pid FROM ONLY public.ses_vt_20130805_xc a WHERE ((a.exposed_time >= 4667) AND (a.exposed_time <= 5031))) r(a_1, a_2) ON (true)) WHERE (l.a_1 = r.a_2) GROUP BY 1
>>         (9 rows)
>>
>>                                               QUERY PLAN (Datanode)
>>
>>           GroupAggregate  (cost=0.00..47862.29 rows=225 width=8)
>>             Output: a.exposed_time, round(((count(a.pid) / 10))::double precision)
>>             ->  Nested Loop  (cost=0.00..47856.05 rows=460 width=8)
>>                   Output: a.exposed_time, a.pid
>>                   ->  Index Scan using et_ses on public.ses_vt_20130805_xc a  (cost=0.00..7283.10 rows=129583 width=8)
>>                         Output: a.pid, a.rep, a.exposed_time, a.infectious_time, a.recovered_time
>>                         Index Cond: ((a.exposed_time >= 4667) AND (a.exposed_time <= 5031))
>>                   ->  Index Scan using pid_demo on public.vt_demography_info_xc d  (cost=0.00..0.30 rows=1 width=4)
>>                         Output: d.pid, d.hid, d.age, d.gender, d.zipode, d.blockgroupid, d.longitude, d.lattitude, d.county, d.countyid
>>                         Index Cond: (d.pid = a.pid)
>>                         Filter: ((d.age >= 5) AND (d.age <= 18) AND ((d.countyid)::text = '50015'::text) AND (d.gender = 1))
>>         (11 rows)
>>
>>
>>
>>
>>         On Thu, Oct 17, 2013 at 12:12 AM, Ashutosh Bapat
>>         <ash...@en...
>>         <mailto:ash...@en...>> wrote:
>>
>>             Sandeep,
>>             It would be nice if you mention the version of XC in your
>>             mail. Sort push down is available from 1.1 onwards. If
>>             you do not see sort getting pushed down in 1.1, please
>>             report detailed definitions of the tables, query and the
>>             EXPLAIN output.
>>
>>
>>             On Thu, Oct 17, 2013 at 1:09 AM, Sandeep Gupta
>>             <gup...@gm...
>>             <mailto:gup...@gm...>> wrote:
>>
>>                 Hi,
>>
>>                  In an another query that requires the result to be
>>                 aggregated and ordered by a field (lets say timeo)
>>                 the query planner currently pulls  the results and
>>                 then performs a sort with hash aggregate.
>>
>>                 The table at the datanodes are clustered by timeo. I
>>                 was wondering if it possible
>>                 for query planner to push down the order by clause at
>>                 the datanode and then perform
>>                 sort-merge aggregate at the coordinator. Surely, that
>>                 would be a better query plan.
>>
>>                 We have tried enable_sort=off etc. but that doesn't
>>                 work.
>>
>>                 Thanks.
>>                 Sandeep
>>
>>
>>
>>                 ------------------------------------------------------------------------------
>>                 October Webinars: Code for Performance
>>                 Free Intel webinars can help you accelerate
>>                 application performance.
>>                 Explore tips for MPI, OpenMP, advanced profiling, and
>>                 more. Get the most from
>>                 the latest Intel processors and coprocessors. See
>>                 abstracts and register >
>>                 http://pubads.g.doubleclick.net/gampad/clk?id=60135031&iu=/4140/ostg.clktrk
>>                 _______________________________________________
>>                 Postgres-xc-general mailing list
>>                 Pos...@li...
>>                 <mailto:Pos...@li...>
>>                 https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>
>>
>>
>>
>>             -- 
>>             Best Wishes,
>>             Ashutosh Bapat
>>             EnterpriseDB Corporation
>>             The Postgres Database Company
>>
>>
>>
>>
>>
>>     -- 
>>     Best Wishes,
>>     Ashutosh Bapat
>>     EnterpriseDB Corporation
>>     The Postgres Database Company
>
>
>
>
> -- 
> Best Wishes,
> Ashutosh Bapat
> EnterpriseDB Corporation
> The Postgres Database Company

This description is for distributed tables. For replicated tables,
everything shippable is shipped.


On Thu, Oct 17, 2013 at 4:43 PM, Ashutosh Bapat <
ash...@en...> wrote:

> That's not possible right now. For grouping, either the grouped input from
> the datanode gets shuffled at the coordinator or ordered on the grouping
> column. Thus even if we get the ordered intput from datanode on a
> particular column that order is disturbed at the coordinator because of
> grouping.
>
>
> On Thu, Oct 17, 2013 at 4:37 PM, Sandeep Gupta <gup...@gm...>wrote:
>
>>  How about a plan where the datanodes perform the sort as well and the
>> coordinator performs a sorted merge?
>>
>> Are such plans not part of the query planner?
>>
>> -Sandeep
>>
>>
>> On 10/17/2013 07:04 AM, Ashutosh Bapat wrote:
>>
>> There is GROUP BY clause that needs to be evaluated before the result can
>> be ordered. Thus GROUP BY is sent to the datanode but not ORDER BY.
>>
>>
>>  On Thu, Oct 17, 2013 at 4:31 PM, Sandeep Gupta <gup...@gm...>wrote:
>>
>>>  Hi Ashutosh,
>>>
>>>
>>>    Attached below is the query and the corresponding query plan. I am
>>> using version 1.1.
>>>  Thanks for taking a look at this.
>>>
>>>  -Sandeep
>>>
>>> SELECT exposed_time effectedDate, ROUND(COUNT(a.pid)/10) COUNT FROM public.vt_demography_info_xc d, public.ses_vt_20130805_xc a WHERE d.pid=a.pid AND d.countyid='50015' AND d.age BETWEEN 5 AND 18 AND d.gender=1 GROUP BY exposed_time ORDER BY exposed_time;
>>>
>>>
>>>
>>>
>>>
>>>                                      QUERY PLAN (Coordinator)
>>>
>>>  Sort  (cost=10000000005.03..10000000005.03 rows=1 width=8)
>>>    Output: a.exposed_time, (round(((count((count(a.pid))) / 10))::double precision))
>>>    Sort Key: a.exposed_time
>>>    ->  HashAggregate  (cost=5.00..5.02 rows=1 width=8)
>>>          Output: a.exposed_time, round(((count((count(a.pid))) / 10))::double precision)
>>>          ->  Data Node Scan on "__REMOTE_GROUP_QUERY__"  (cost=0.00..0.00 rows=1000 width=8)
>>>                Output: a.exposed_time, (count(a.pid))
>>>                Node/s: datanode1, datanode10, datanode11, datanode12, datanode13, datanode14, datanode15, datanode16, datanode2, datanode3, datanode4, datanode5, datanode6, datanode7, datanode8, datanode9
>>>                Remote query: SELECT r.a_1, count(r.a_2) FROM ((SELECT d.pid FROM ONLY public.vt_demography_info_xc d WHERE ((d.age >= 5) AND (d.age <= 18) AND ((d.countyid)::text = '50015'::text) AND (d.gender = 1))) l(a_1) JOIN (SELECT a.exposed_time, a.pid FROM ONLY public.ses_vt_20130805_xc a WHERE ((a.exposed_time >= 4667) AND (a.exposed_time <= 5031))) r(a_1, a_2) ON (true)) WHERE (l.a_1 = r.a_2) GROUP BY 1
>>> (9 rows)
>>>
>>>                                      QUERY PLAN (Datanode)
>>>
>>>  GroupAggregate  (cost=0.00..47862.29 rows=225 width=8)
>>>    Output: a.exposed_time, round(((count(a.pid) / 10))::double precision)
>>>    ->  Nested Loop  (cost=0.00..47856.05 rows=460 width=8)
>>>          Output: a.exposed_time, a.pid
>>>          ->  Index Scan using et_ses on public.ses_vt_20130805_xc a  (cost=0.00..7283.10 rows=129583 width=8)
>>>                Output: a.pid, a.rep, a.exposed_time, a.infectious_time, a.recovered_time
>>>                Index Cond: ((a.exposed_time >= 4667) AND (a.exposed_time <= 5031))
>>>          ->  Index Scan using pid_demo on public.vt_demography_info_xc d  (cost=0.00..0.30 rows=1 width=4)
>>>                Output: d.pid, d.hid, d.age, d.gender, d.zipode, d.blockgroupid, d.longitude, d.lattitude, d.county, d.countyid
>>>                Index Cond: (d.pid = a.pid)
>>>                Filter: ((d.age >= 5) AND (d.age <= 18) AND ((d.countyid)::text = '50015'::text) AND (d.gender = 1))
>>> (11 rows)
>>>
>>>
>>>
>>>
>>> On Thu, Oct 17, 2013 at 12:12 AM, Ashutosh Bapat <
>>> ash...@en...> wrote:
>>>
>>>> Sandeep,
>>>> It would be nice if you mention the version of XC in your mail. Sort
>>>> push down is available from 1.1 onwards. If you do not see sort getting
>>>> pushed down in 1.1, please report detailed definitions of the tables, query
>>>> and the EXPLAIN output.
>>>>
>>>>
>>>>  On Thu, Oct 17, 2013 at 1:09 AM, Sandeep Gupta <
>>>> gup...@gm...> wrote:
>>>>
>>>>>    Hi,
>>>>>
>>>>>   In an another query that requires the result to be aggregated and
>>>>> ordered by a field (lets say timeo)
>>>>> the query planner currently pulls  the results and then performs a
>>>>> sort with hash aggregate.
>>>>>
>>>>>  The table at the datanodes are clustered by timeo. I was wondering if
>>>>> it possible
>>>>> for query planner to push down the order by clause at the datanode and
>>>>> then perform
>>>>>  sort-merge aggregate at the coordinator. Surely, that would be a
>>>>> better query plan.
>>>>>
>>>>>  We have tried enable_sort=off etc. but that doesn't work.
>>>>>
>>>>>  Thanks.
>>>>>  Sandeep
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> October Webinars: Code for Performance
>>>>> Free Intel webinars can help you accelerate application performance.
>>>>> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the
>>>>> most from
>>>>> the latest Intel processors and coprocessors. See abstracts and
>>>>> register >
>>>>>
>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=60135031&iu=/4140/ostg.clktrk
>>>>> _______________________________________________
>>>>> Postgres-xc-general mailing list
>>>>> Pos...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Best Wishes,
>>>> Ashutosh Bapat
>>>> EnterpriseDB Corporation
>>>> The Postgres Database Company
>>>>
>>>
>>>
>>
>>
>> --
>> Best Wishes,
>> Ashutosh Bapat
>> EnterpriseDB Corporation
>> The Postgres Database Company
>>
>>
>>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EnterpriseDB Corporation
> The Postgres Database Company
>



-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

That's not possible right now. For grouping, either the grouped input from
the datanode gets shuffled at the coordinator or ordered on the grouping
column. Thus even if we get the ordered intput from datanode on a
particular column that order is disturbed at the coordinator because of
grouping.


On Thu, Oct 17, 2013 at 4:37 PM, Sandeep Gupta <gup...@gm...>wrote:

>  How about a plan where the datanodes perform the sort as well and the
> coordinator performs a sorted merge?
>
> Are such plans not part of the query planner?
>
> -Sandeep
>
>
> On 10/17/2013 07:04 AM, Ashutosh Bapat wrote:
>
> There is GROUP BY clause that needs to be evaluated before the result can
> be ordered. Thus GROUP BY is sent to the datanode but not ORDER BY.
>
>
>  On Thu, Oct 17, 2013 at 4:31 PM, Sandeep Gupta <gup...@gm...>wrote:
>
>>  Hi Ashutosh,
>>
>>
>>    Attached below is the query and the corresponding query plan. I am
>> using version 1.1.
>>  Thanks for taking a look at this.
>>
>>  -Sandeep
>>
>> SELECT exposed_time effectedDate, ROUND(COUNT(a.pid)/10) COUNT FROM public.vt_demography_info_xc d, public.ses_vt_20130805_xc a WHERE d.pid=a.pid AND d.countyid='50015' AND d.age BETWEEN 5 AND 18 AND d.gender=1 GROUP BY exposed_time ORDER BY exposed_time;
>>
>>
>>
>>
>>
>>                                      QUERY PLAN (Coordinator)
>>
>>  Sort  (cost=10000000005.03..10000000005.03 rows=1 width=8)
>>    Output: a.exposed_time, (round(((count((count(a.pid))) / 10))::double precision))
>>    Sort Key: a.exposed_time
>>    ->  HashAggregate  (cost=5.00..5.02 rows=1 width=8)
>>          Output: a.exposed_time, round(((count((count(a.pid))) / 10))::double precision)
>>          ->  Data Node Scan on "__REMOTE_GROUP_QUERY__"  (cost=0.00..0.00 rows=1000 width=8)
>>                Output: a.exposed_time, (count(a.pid))
>>                Node/s: datanode1, datanode10, datanode11, datanode12, datanode13, datanode14, datanode15, datanode16, datanode2, datanode3, datanode4, datanode5, datanode6, datanode7, datanode8, datanode9
>>                Remote query: SELECT r.a_1, count(r.a_2) FROM ((SELECT d.pid FROM ONLY public.vt_demography_info_xc d WHERE ((d.age >= 5) AND (d.age <= 18) AND ((d.countyid)::text = '50015'::text) AND (d.gender = 1))) l(a_1) JOIN (SELECT a.exposed_time, a.pid FROM ONLY public.ses_vt_20130805_xc a WHERE ((a.exposed_time >= 4667) AND (a.exposed_time <= 5031))) r(a_1, a_2) ON (true)) WHERE (l.a_1 = r.a_2) GROUP BY 1
>> (9 rows)
>>
>>                                      QUERY PLAN (Datanode)
>>
>>  GroupAggregate  (cost=0.00..47862.29 rows=225 width=8)
>>    Output: a.exposed_time, round(((count(a.pid) / 10))::double precision)
>>    ->  Nested Loop  (cost=0.00..47856.05 rows=460 width=8)
>>          Output: a.exposed_time, a.pid
>>          ->  Index Scan using et_ses on public.ses_vt_20130805_xc a  (cost=0.00..7283.10 rows=129583 width=8)
>>                Output: a.pid, a.rep, a.exposed_time, a.infectious_time, a.recovered_time
>>                Index Cond: ((a.exposed_time >= 4667) AND (a.exposed_time <= 5031))
>>          ->  Index Scan using pid_demo on public.vt_demography_info_xc d  (cost=0.00..0.30 rows=1 width=4)
>>                Output: d.pid, d.hid, d.age, d.gender, d.zipode, d.blockgroupid, d.longitude, d.lattitude, d.county, d.countyid
>>                Index Cond: (d.pid = a.pid)
>>                Filter: ((d.age >= 5) AND (d.age <= 18) AND ((d.countyid)::text = '50015'::text) AND (d.gender = 1))
>> (11 rows)
>>
>>
>>
>>
>> On Thu, Oct 17, 2013 at 12:12 AM, Ashutosh Bapat <
>> ash...@en...> wrote:
>>
>>> Sandeep,
>>> It would be nice if you mention the version of XC in your mail. Sort
>>> push down is available from 1.1 onwards. If you do not see sort getting
>>> pushed down in 1.1, please report detailed definitions of the tables, query
>>> and the EXPLAIN output.
>>>
>>>
>>>  On Thu, Oct 17, 2013 at 1:09 AM, Sandeep Gupta <gup...@gm...
>>> > wrote:
>>>
>>>>    Hi,
>>>>
>>>>   In an another query that requires the result to be aggregated and
>>>> ordered by a field (lets say timeo)
>>>> the query planner currently pulls  the results and then performs a sort
>>>> with hash aggregate.
>>>>
>>>>  The table at the datanodes are clustered by timeo. I was wondering if
>>>> it possible
>>>> for query planner to push down the order by clause at the datanode and
>>>> then perform
>>>>  sort-merge aggregate at the coordinator. Surely, that would be a
>>>> better query plan.
>>>>
>>>>  We have tried enable_sort=off etc. but that doesn't work.
>>>>
>>>>  Thanks.
>>>>  Sandeep
>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> October Webinars: Code for Performance
>>>> Free Intel webinars can help you accelerate application performance.
>>>> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the
>>>> most from
>>>> the latest Intel processors and coprocessors. See abstracts and
>>>> register >
>>>>
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=60135031&iu=/4140/ostg.clktrk
>>>> _______________________________________________
>>>> Postgres-xc-general mailing list
>>>> Pos...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>>
>>>>
>>>
>>>
>>> --
>>> Best Wishes,
>>> Ashutosh Bapat
>>> EnterpriseDB Corporation
>>> The Postgres Database Company
>>>
>>
>>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EnterpriseDB Corporation
> The Postgres Database Company
>
>
>


-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

How about a plan where the datanodes perform the sort as well and the
coordinator performs a sorted merge?

Are such plans not part of the query planner?

-Sandeep

On 10/17/2013 07:04 AM, Ashutosh Bapat wrote:
> There is GROUP BY clause that needs to be evaluated before the result 
> can be ordered. Thus GROUP BY is sent to the datanode but not ORDER BY.
>
>
> On Thu, Oct 17, 2013 at 4:31 PM, Sandeep Gupta 
> <gup...@gm... <mailto:gup...@gm...>> wrote:
>
>     Hi Ashutosh,
>
>
>       Attached below is the query and the corresponding query plan. I
>     am using version 1.1.
>     Thanks for taking a look at this.
>
>     -Sandeep
>
>     SELECT exposed_time effectedDate, ROUND(COUNT(a.pid)/10) COUNT FROM public.vt_demography_info_xc d, public.ses_vt_20130805_xc a WHERE d.pid=a.pid AND d.countyid='50015' AND d.age BETWEEN 5 AND 18 AND d.gender=1 GROUP BY exposed_time ORDER BY exposed_time;
>
>
>
>
>                                         QUERY PLAN (Coordinator)
>
>       Sort  (cost=10000000005.03..10000000005.03 rows=1 width=8)
>         Output: a.exposed_time, (round(((count((count(a.pid))) / 10))::double precision))
>         Sort Key: a.exposed_time
>         ->  HashAggregate  (cost=5.00..5.02 rows=1 width=8)
>               Output: a.exposed_time, round(((count((count(a.pid))) / 10))::double precision)
>               ->  Data Node Scan on "__REMOTE_GROUP_QUERY__"  (cost=0.00..0.00 rows=1000 width=8)
>                     Output: a.exposed_time, (count(a.pid))
>                     Node/s: datanode1, datanode10, datanode11, datanode12, datanode13, datanode14, datanode15, datanode16, datanode2, datanode3, datanode4, datanode5, datanode6, datanode7, datanode8, datanode9
>                     Remote query: SELECT r.a_1, count(r.a_2) FROM ((SELECT d.pid FROM ONLY public.vt_demography_info_xc d WHERE ((d.age >= 5) AND (d.age <= 18) AND ((d.countyid)::text = '50015'::text) AND (d.gender = 1))) l(a_1) JOIN (SELECT a.exposed_time, a.pid FROM ONLY public.ses_vt_20130805_xc a WHERE ((a.exposed_time >= 4667) AND (a.exposed_time <= 5031))) r(a_1, a_2) ON (true)) WHERE (l.a_1 = r.a_2) GROUP BY 1
>     (9 rows)
>
>                                           QUERY PLAN (Datanode)
>
>       GroupAggregate  (cost=0.00..47862.29 rows=225 width=8)
>         Output: a.exposed_time, round(((count(a.pid) / 10))::double precision)
>         ->  Nested Loop  (cost=0.00..47856.05 rows=460 width=8)
>               Output: a.exposed_time, a.pid
>               ->  Index Scan using et_ses on public.ses_vt_20130805_xc a  (cost=0.00..7283.10 rows=129583 width=8)
>                     Output: a.pid, a.rep, a.exposed_time, a.infectious_time, a.recovered_time
>                     Index Cond: ((a.exposed_time >= 4667) AND (a.exposed_time <= 5031))
>               ->  Index Scan using pid_demo on public.vt_demography_info_xc d  (cost=0.00..0.30 rows=1 width=4)
>                     Output: d.pid, d.hid, d.age, d.gender, d.zipode, d.blockgroupid, d.longitude, d.lattitude, d.county, d.countyid
>                     Index Cond: (d.pid = a.pid)
>                     Filter: ((d.age >= 5) AND (d.age <= 18) AND ((d.countyid)::text = '50015'::text) AND (d.gender = 1))
>     (11 rows)
>
>
>
>
>     On Thu, Oct 17, 2013 at 12:12 AM, Ashutosh Bapat
>     <ash...@en...
>     <mailto:ash...@en...>> wrote:
>
>         Sandeep,
>         It would be nice if you mention the version of XC in your
>         mail. Sort push down is available from 1.1 onwards. If you do
>         not see sort getting pushed down in 1.1, please report
>         detailed definitions of the tables, query and the EXPLAIN output.
>
>
>         On Thu, Oct 17, 2013 at 1:09 AM, Sandeep Gupta
>         <gup...@gm... <mailto:gup...@gm...>> wrote:
>
>             Hi,
>
>              In an another query that requires the result to be
>             aggregated and ordered by a field (lets say timeo)
>             the query planner currently pulls the results and then
>             performs a sort with hash aggregate.
>
>             The table at the datanodes are clustered by timeo. I was
>             wondering if it possible
>             for query planner to push down the order by clause at the
>             datanode and then perform
>             sort-merge aggregate at the coordinator. Surely, that
>             would be a better query plan.
>
>             We have tried enable_sort=off etc. but that doesn't work.
>
>             Thanks.
>             Sandeep
>
>
>
>             ------------------------------------------------------------------------------
>             October Webinars: Code for Performance
>             Free Intel webinars can help you accelerate application
>             performance.
>             Explore tips for MPI, OpenMP, advanced profiling, and
>             more. Get the most from
>             the latest Intel processors and coprocessors. See
>             abstracts and register >
>             http://pubads.g.doubleclick.net/gampad/clk?id=60135031&iu=/4140/ostg.clktrk
>             _______________________________________________
>             Postgres-xc-general mailing list
>             Pos...@li...
>             <mailto:Pos...@li...>
>             https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
>
>
>
>         -- 
>         Best Wishes,
>         Ashutosh Bapat
>         EnterpriseDB Corporation
>         The Postgres Database Company
>
>
>
>
>
> -- 
> Best Wishes,
> Ashutosh Bapat
> EnterpriseDB Corporation
> The Postgres Database Company

There is GROUP BY clause that needs to be evaluated before the result can
be ordered. Thus GROUP BY is sent to the datanode but not ORDER BY.


On Thu, Oct 17, 2013 at 4:31 PM, Sandeep Gupta <gup...@gm...>wrote:

> Hi Ashutosh,
>
>
>   Attached below is the query and the corresponding query plan. I am using
> version 1.1.
> Thanks for taking a look at this.
>
> -Sandeep
>
> SELECT exposed_time effectedDate, ROUND(COUNT(a.pid)/10) COUNT FROM public.vt_demography_info_xc d, public.ses_vt_20130805_xc a WHERE d.pid=a.pid AND d.countyid='50015' AND d.age BETWEEN 5 AND 18 AND d.gender=1 GROUP BY exposed_time ORDER BY exposed_time;
>
>
>                                    QUERY PLAN (Coordinator)
>
>  Sort  (cost=10000000005.03..10000000005.03 rows=1 width=8)
>    Output: a.exposed_time, (round(((count((count(a.pid))) / 10))::double precision))
>    Sort Key: a.exposed_time
>    ->  HashAggregate  (cost=5.00..5.02 rows=1 width=8)
>          Output: a.exposed_time, round(((count((count(a.pid))) / 10))::double precision)
>          ->  Data Node Scan on "__REMOTE_GROUP_QUERY__"  (cost=0.00..0.00 rows=1000 width=8)
>                Output: a.exposed_time, (count(a.pid))
>                Node/s: datanode1, datanode10, datanode11, datanode12, datanode13, datanode14, datanode15, datanode16, datanode2, datanode3, datanode4, datanode5, datanode6, datanode7, datanode8, datanode9
>                Remote query: SELECT r.a_1, count(r.a_2) FROM ((SELECT d.pid FROM ONLY public.vt_demography_info_xc d WHERE ((d.age >= 5) AND (d.age <= 18) AND ((d.countyid)::text = '50015'::text) AND (d.gender = 1))) l(a_1) JOIN (SELECT a.exposed_time, a.pid FROM ONLY public.ses_vt_20130805_xc a WHERE ((a.exposed_time >= 4667) AND (a.exposed_time <= 5031))) r(a_1, a_2) ON (true)) WHERE (l.a_1 = r.a_2) GROUP BY 1
> (9 rows)
>
>                                      QUERY PLAN (Datanode)
>
>  GroupAggregate  (cost=0.00..47862.29 rows=225 width=8)
>    Output: a.exposed_time, round(((count(a.pid) / 10))::double precision)
>    ->  Nested Loop  (cost=0.00..47856.05 rows=460 width=8)
>          Output: a.exposed_time, a.pid
>          ->  Index Scan using et_ses on public.ses_vt_20130805_xc a  (cost=0.00..7283.10 rows=129583 width=8)
>                Output: a.pid, a.rep, a.exposed_time, a.infectious_time, a.recovered_time
>                Index Cond: ((a.exposed_time >= 4667) AND (a.exposed_time <= 5031))
>          ->  Index Scan using pid_demo on public.vt_demography_info_xc d  (cost=0.00..0.30 rows=1 width=4)
>                Output: d.pid, d.hid, d.age, d.gender, d.zipode, d.blockgroupid, d.longitude, d.lattitude, d.county, d.countyid
>                Index Cond: (d.pid = a.pid)
>                Filter: ((d.age >= 5) AND (d.age <= 18) AND ((d.countyid)::text = '50015'::text) AND (d.gender = 1))
> (11 rows)
>
>
>
>
> On Thu, Oct 17, 2013 at 12:12 AM, Ashutosh Bapat <
> ash...@en...> wrote:
>
>> Sandeep,
>> It would be nice if you mention the version of XC in your mail. Sort push
>> down is available from 1.1 onwards. If you do not see sort getting pushed
>> down in 1.1, please report detailed definitions of the tables, query and
>> the EXPLAIN output.
>>
>>
>> On Thu, Oct 17, 2013 at 1:09 AM, Sandeep Gupta <gup...@gm...>wrote:
>>
>>> Hi,
>>>
>>>  In an another query that requires the result to be aggregated and
>>> ordered by a field (lets say timeo)
>>> the query planner currently pulls  the results and then performs a sort
>>> with hash aggregate.
>>>
>>> The table at the datanodes are clustered by timeo. I was wondering if it
>>> possible
>>> for query planner to push down the order by clause at the datanode and
>>> then perform
>>> sort-merge aggregate at the coordinator. Surely, that would be a better
>>> query plan.
>>>
>>> We have tried enable_sort=off etc. but that doesn't work.
>>>
>>> Thanks.
>>> Sandeep
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> October Webinars: Code for Performance
>>> Free Intel webinars can help you accelerate application performance.
>>> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
>>> from
>>> the latest Intel processors and coprocessors. See abstracts and register
>>> >
>>>
>>> http://pubads.g.doubleclick.net/gampad/clk?id=60135031&iu=/4140/ostg.clktrk
>>> _______________________________________________
>>> Postgres-xc-general mailing list
>>> Pos...@li...
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>
>>>
>>
>>
>> --
>> Best Wishes,
>> Ashutosh Bapat
>> EnterpriseDB Corporation
>> The Postgres Database Company
>>
>
>


-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Hi Ashutosh,


  Attached below is the query and the corresponding query plan. I am using
version 1.1.
Thanks for taking a look at this.

-Sandeep

SELECT exposed_time effectedDate, ROUND(COUNT(a.pid)/10) COUNT FROM
public.vt_demography_info_xc d, public.ses_vt_20130805_xc a WHERE
d.pid=a.pid AND d.countyid='50015' AND d.age BETWEEN 5 AND 18 AND
d.gender=1 GROUP BY exposed_time ORDER BY exposed_time;


                                   QUERY PLAN (Coordinator)

 Sort  (cost=10000000005.03..10000000005.03 rows=1 width=8)
   Output: a.exposed_time, (round(((count((count(a.pid))) /
10))::double precision))
   Sort Key: a.exposed_time
   ->  HashAggregate  (cost=5.00..5.02 rows=1 width=8)
         Output: a.exposed_time, round(((count((count(a.pid))) /
10))::double precision)
         ->  Data Node Scan on "__REMOTE_GROUP_QUERY__"
(cost=0.00..0.00 rows=1000 width=8)
               Output: a.exposed_time, (count(a.pid))
               Node/s: datanode1, datanode10, datanode11, datanode12,
datanode13, datanode14, datanode15, datanode16, datanode2, datanode3,
datanode4, datanode5, datanode6, datanode7, datanode8, datanode9
               Remote query: SELECT r.a_1, count(r.a_2) FROM ((SELECT
d.pid FROM ONLY public.vt_demography_info_xc d WHERE ((d.age >= 5) AND
(d.age <= 18) AND ((d.countyid)::text = '50015'::text) AND (d.gender =
1))) l(a_1) JOIN (SELECT a.exposed_time, a.pid FROM ONLY
public.ses_vt_20130805_xc a WHERE ((a.exposed_time >= 4667) AND
(a.exposed_time <= 5031))) r(a_1, a_2) ON (true)) WHERE (l.a_1 =
r.a_2) GROUP BY 1
(9 rows)

                                     QUERY PLAN (Datanode)

 GroupAggregate  (cost=0.00..47862.29 rows=225 width=8)
   Output: a.exposed_time, round(((count(a.pid) / 10))::double precision)
   ->  Nested Loop  (cost=0.00..47856.05 rows=460 width=8)
         Output: a.exposed_time, a.pid
         ->  Index Scan using et_ses on public.ses_vt_20130805_xc a
(cost=0.00..7283.10 rows=129583 width=8)
               Output: a.pid, a.rep, a.exposed_time,
a.infectious_time, a.recovered_time
               Index Cond: ((a.exposed_time >= 4667) AND
(a.exposed_time <= 5031))
         ->  Index Scan using pid_demo on public.vt_demography_info_xc
d  (cost=0.00..0.30 rows=1 width=4)
               Output: d.pid, d.hid, d.age, d.gender, d.zipode,
d.blockgroupid, d.longitude, d.lattitude, d.county, d.countyid
               Index Cond: (d.pid = a.pid)
               Filter: ((d.age >= 5) AND (d.age <= 18) AND
((d.countyid)::text = '50015'::text) AND (d.gender = 1))
(11 rows)




On Thu, Oct 17, 2013 at 12:12 AM, Ashutosh Bapat <
ash...@en...> wrote:

> Sandeep,
> It would be nice if you mention the version of XC in your mail. Sort push
> down is available from 1.1 onwards. If you do not see sort getting pushed
> down in 1.1, please report detailed definitions of the tables, query and
> the EXPLAIN output.
>
>
> On Thu, Oct 17, 2013 at 1:09 AM, Sandeep Gupta <gup...@gm...>wrote:
>
>> Hi,
>>
>>  In an another query that requires the result to be aggregated and
>> ordered by a field (lets say timeo)
>> the query planner currently pulls  the results and then performs a sort
>> with hash aggregate.
>>
>> The table at the datanodes are clustered by timeo. I was wondering if it
>> possible
>> for query planner to push down the order by clause at the datanode and
>> then perform
>> sort-merge aggregate at the coordinator. Surely, that would be a better
>> query plan.
>>
>> We have tried enable_sort=off etc. but that doesn't work.
>>
>> Thanks.
>> Sandeep
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> October Webinars: Code for Performance
>> Free Intel webinars can help you accelerate application performance.
>> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
>> from
>> the latest Intel processors and coprocessors. See abstracts and register >
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=60135031&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Postgres-xc-general mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>
>>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EnterpriseDB Corporation
> The Postgres Database Company
>

Yes, foreign data is not a part of XC cluster.   Foreign data's transaction management is separate from the cluster and we cannot enforce data integrity.   Even though we support FDW, it is just foreign data, not a part of XC cluster.   To make PostgreSQL data as a part of XC cluster, PG needs to accept GXID and snapshot from XC, as well as sequence, if it is shared with other tables in XC.

It does not sound simple because in this case, PostgreSQL database is not autonomous and cannot operate by its own.

Regards;
---
Koichi Suzuki

On 2013/10/17, at 14:52, Amit Khandekar <ami...@en...<mailto:ami...@en...>>
 wrote:

I think Aris's expectation is that by using FDW support we can create a cluster of heterogeneous nodes. This is not going to happen just by supporting  foreign data wrappers and foreign tables.

Note that allowing a foreign table/server to be created in Postgres-XC means a foreign table will be created on a machine which is completely outside the Postgres-XC cluster. Making that machine a part of the cluster is a different thing, and that does not require FDWs.


On 17 October 2013 10:48, 鈴木 幸市 <ko...@in...<mailto:ko...@in...>> wrote:
So far, CREATE FOREIGN DATA WRAPPER, CREATE SERVER and CREATE USER MAPPING are blocked.   As Michael suggests, yes, it would be nice to connect to foreign data through coordinators or even datanodes.

It's welcome if more people are involved in the test, not just development.   Contribution of the code is more than welcome.

Unfortunately, nobody dis these work mainly due to the resource.   it will be wonderful if anybody can join and contribute the code.   There are not reason that XC doesn't have to support FDW.

Best;
---
Koichi Suzuki

On 2013/10/17, at 13:11, Michael Paquier <mic...@gm...<mailto:mic...@gm...>> wrote:

> On Thu, Oct 17, 2013 at 12:43 PM, Aris Setyawan <ari...@gm...<mailto:ari...@gm...>> wrote:
>> Hi,
>>
>> Can XC be used with [write-able] foreign-data wrapper? What I mean
>> here are push down optimization and data distribution.
> XC does not support itself fdw, but I don't see why there would be
> problems to have a Postgres server with a postgres_fdw connect to
> Coordinators for read/write operations or even Datanodes for read-only
> operations as the communication interface is the same as vanilla. Take
> care to use at least XC 1.1~ for the latter though.
>
>> I imagine, If yes, we can have a cluster of not just postgresql node.
>> But we can have oracle or mysql or redis or unlimited cluster.
> Yep. Supporting FDW in XC would be fun as well. Patches welcome.
> --
> Michael
>
> ------------------------------------------------------------------------------
> October Webinars: Code for Performance
> Free Intel webinars can help you accelerate application performance.
> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from
> the latest Intel processors and coprocessors. See abstracts and register >
> http://pubads.g.doubleclick.net/gampad/clk?id=60135031&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...<mailto:Pos...@li...>
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>


------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60135031&iu=/4140/ostg.clktrk
_______________________________________________
Postgres-xc-general mailing list
Pos...@li...<mailto:Pos...@li...>
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general