postgres-xc-developers Mailing List for Postgres-XC (Page 126)

On Tue, Apr 12, 2011 at 5:37 PM, Andrei Martsinchyk <
and...@gm...> wrote:

>
> XC aggregate is implemented by three functions: first function is invoked
> on data node once per row and added argument to internal accumulated value,
> which is sent to coordinator after group is processed, second function is
> invoked on coordinator once per received pre-aggregated row and combine
> pre-aggregated values together, third function is invoked on coordinator per
> group and convert accumulated value to aggregation result.
> Regarding sum() and count(), they have to perform typecast on final step.
> In Postgres sum(int4)int8, accumulating function is defined like
> sum_agg(int8, int4):int8; in XC this function performs pre-aggregation.
> Combining function is sum_agg(numeric, int8):numeric, Postgres does not have
> sum_agg(int8, int8):int8. So XC has to convert numeric to int8 to return
> value of declared type, while in Postgres accumulated value can be returned
> without conversion.
>

The plain PG pg_aggregate entries for sum look like
postgres=# select * from pg_aggregate where aggfnoid in (select oid from
pg_proc where proname = 'sum');
    aggfnoid    | aggtransfn  | aggcollectfn |   aggfinalfn    | aggsortop |
aggtranstype | aggcollecttype | agginitval | agginitcollect
----------------+-------------+--------------+-----------------+-----------+--------------+----------------+------------+----------------
 pg_catalog.sum | int8_sum    | numeric_add  | -               |         0
|         1700 |           1700 |            |
 pg_catalog.sum | int4_sum    | int8_sum     | pg_catalog.int8 |         0
|           20 |           1700 |            |
 pg_catalog.sum | int2_sum    | int8_sum     | pg_catalog.int8 |         0
|           20 |           1700 |            |


And the PGXC entries look like
testdb=# select * from pg_aggregate where aggfnoid in (select oid from
pg_proc where proname = 'sum');
    aggfnoid    | aggtransfn  | aggfinalfn | aggsortop | aggtranstype |
agginitval
----------------+-------------+------------+-----------+--------------+------------
 pg_catalog.sum | int8_sum    | -          |         0 |         1700 |
 pg_catalog.sum | int4_sum    | -          |         0 |           20 |
 pg_catalog.sum | int2_sum    | -          |         0 |           20 |

In PG, the sum of integers all result in int8, whereas in PGXC they result
into numeric and casted back to int8. May be we should use a new function
int8_sum(int8, int8):int8 instead of int8_sum(numeric, int8):numeric. That
way we don't need any final function for sum, just like PG.

This will help us set the finalfn_oid in ExecInitAgg() and we will have
group by running (albeit slow). This has another impact. The plain
aggregates (without any group by) with JOINs are not working currently. For
example, query postgres=# select avg(emp.val * dept.val) from emp, dept;
returns 0 (1 row with value 0) even if there is some non-zero data in those
tables. This is because we do not set finalfn_oid in ExecInitAgg() and the
tree for above query looks like
AggState(NestedLoop(RemoteQuery (select val from emp), RemoteQuery(select
val from dept))). Thus the aggregate is not pushed down to the data node.
While finalising the aggregate result, it does not find finalfnoid and thus
returns false results. If we can set finalfnoid as done in the attached
patch, we will get the group by running albeit suboptimally.

Any thoughts?

I guess in your code one of aggregation steps is missing.
> Hope this helps.
>
> 2011/4/12 Ashutosh Bapat <ash...@en...>
>
>> Hi,
>> I took outputs of query "select aggfnoid, aggfinalfn from pg_aggregate
>> where aggfinalfn != 0;" against plain postgres and PGXC. It showed following
>> difference
>> [ashutosh@anand PG_HEAD]diff /tmp/pgxc_aggfinalfn.out
>> /tmp/pg_aggfinalfn.out
>> 10,13d9
>> <  pg_catalog.sum         | pg_catalog.int8
>> <  pg_catalog.sum         | pg_catalog.int8
>> <  pg_catalog.count       | pg_catalog.int8
>> <  pg_catalog.count       | pg_catalog.int8
>> 50d45
>> <  regr_count             | pg_catalog.int8
>> 62c57,59
>> < (59 rows)
>> ---
>> >  array_agg              | array_agg_finalfn
>> >  string_agg             | string_agg_finalfn
>> > (56 rows)
>>
>>
>> XC has final functions set for aggregates sum, count whereas plain
>> postgres has those. Plain postgres has the final functions for array_agg and
>> string_agg but XC does not have those. Why is this difference?
>>
>> As of now, in XC, for GROUP BY queries, the coordinators receives plain
>> data from data nodes, stripped of any aggregates or GROUP BY clause. I was
>> trying to use PG mechanism to calculate the aggregates (so as to enable
>> group by clauses quickly). It worked for AVG, but for sum it ended up
>> calling numeric_int8() because of above entries which hit segfault since the
>> data passed to it is not numeric. In that case, it's important to know
>> whether those differences are important. NOTE: This won't be the final
>> version of GROUP BY support. I am trying to design it such a way that we can
>> push GROUP BY down to datanodes.
>>
>> The changes are added by commit 8326f619.
>>
>> --
>> Best Wishes,
>> Ashutosh Bapat
>> EntepriseDB Corporation
>> The Enterprise Postgres Company
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Forrester Wave Report - Recovery time is now measured in hours and minutes
>> not days. Key insights are discussed in the 2010 Forrester Wave Report as
>> part of an in-depth evaluation of disaster recovery service providers.
>> Forrester found the best-in-class provider in terms of services and
>> vision.
>> Read this report now!  http://p.sf.net/sfu/ibm-webcastpromo
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>>
>
>
> --
> Best regards,
> Andrei Martsinchyk                   mailto:and...@gm...
>



-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

Hi Ashutosh,

In XC aggregates were changed to perform pre-aggregation on data nodes, that
was done to reduce amount of data sent over network.
If aggregation happened on coordinator each data node would return entire
result set, but instead it returns one row (one row per group, if GROUP BY
exists).
Naturally, definitions of aggregate functions in XC were changed.
Postgres aggregate is implemented by two functions: first function is
invoked once per row and added argument to internal accumulated value,
second function is invoked once per group and convert accumulated value to
aggregation result.
XC aggregate is implemented by three functions: first function is invoked on
data node once per row and added argument to internal accumulated value,
which is sent to coordinator after group is processed, second function is
invoked on coordinator once per received pre-aggregated row and combine
pre-aggregated values together, third function is invoked on coordinator per
group and convert accumulated value to aggregation result.
Regarding sum() and count(), they have to perform typecast on final step.
In Postgres sum(int4)int8, accumulating function is defined like
sum_agg(int8, int4):int8; in XC this function performs pre-aggregation.
Combining function is sum_agg(numeric, int8):numeric, Postgres does not have
sum_agg(int8, int8):int8. So XC has to convert numeric to int8 to return
value of declared type, while in Postgres accumulated value can be returned
without conversion.
I guess in your code one of aggregation steps is missing.
Hope this helps.

2011/4/12 Ashutosh Bapat <ash...@en...>

> Hi,
> I took outputs of query "select aggfnoid, aggfinalfn from pg_aggregate
> where aggfinalfn != 0;" against plain postgres and PGXC. It showed following
> difference
> [ashutosh@anand PG_HEAD]diff /tmp/pgxc_aggfinalfn.out
> /tmp/pg_aggfinalfn.out
> 10,13d9
> <  pg_catalog.sum         | pg_catalog.int8
> <  pg_catalog.sum         | pg_catalog.int8
> <  pg_catalog.count       | pg_catalog.int8
> <  pg_catalog.count       | pg_catalog.int8
> 50d45
> <  regr_count             | pg_catalog.int8
> 62c57,59
> < (59 rows)
> ---
> >  array_agg              | array_agg_finalfn
> >  string_agg             | string_agg_finalfn
> > (56 rows)
>
>
> XC has final functions set for aggregates sum, count whereas plain postgres
> has those. Plain postgres has the final functions for array_agg and
> string_agg but XC does not have those. Why is this difference?
>
> As of now, in XC, for GROUP BY queries, the coordinators receives plain
> data from data nodes, stripped of any aggregates or GROUP BY clause. I was
> trying to use PG mechanism to calculate the aggregates (so as to enable
> group by clauses quickly). It worked for AVG, but for sum it ended up
> calling numeric_int8() because of above entries which hit segfault since the
> data passed to it is not numeric. In that case, it's important to know
> whether those differences are important. NOTE: This won't be the final
> version of GROUP BY support. I am trying to design it such a way that we can
> push GROUP BY down to datanodes.
>
> The changes are added by commit 8326f619.
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Enterprise Postgres Company
>
>
>
> ------------------------------------------------------------------------------
> Forrester Wave Report - Recovery time is now measured in hours and minutes
> not days. Key insights are discussed in the 2010 Forrester Wave Report as
> part of an in-depth evaluation of disaster recovery service providers.
> Forrester found the best-in-class provider in terms of services and vision.
> Read this report now!  http://p.sf.net/sfu/ibm-webcastpromo
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>


-- 
Best regards,
Andrei Martsinchyk                   mailto:and...@gm...

Hi,
I took outputs of query "select aggfnoid, aggfinalfn from pg_aggregate where
aggfinalfn != 0;" against plain postgres and PGXC. It showed following
difference
[ashutosh@anand PG_HEAD]diff /tmp/pgxc_aggfinalfn.out /tmp/pg_aggfinalfn.out

10,13d9
<  pg_catalog.sum         | pg_catalog.int8
<  pg_catalog.sum         | pg_catalog.int8
<  pg_catalog.count       | pg_catalog.int8
<  pg_catalog.count       | pg_catalog.int8
50d45
<  regr_count             | pg_catalog.int8
62c57,59
< (59 rows)
---
>  array_agg              | array_agg_finalfn
>  string_agg             | string_agg_finalfn
> (56 rows)


XC has final functions set for aggregates sum, count whereas plain postgres
has those. Plain postgres has the final functions for array_agg and
string_agg but XC does not have those. Why is this difference?

As of now, in XC, for GROUP BY queries, the coordinators receives plain data
from data nodes, stripped of any aggregates or GROUP BY clause. I was trying
to use PG mechanism to calculate the aggregates (so as to enable group by
clauses quickly). It worked for AVG, but for sum it ended up calling
numeric_int8() because of above entries which hit segfault since the data
passed to it is not numeric. In that case, it's important to know whether
those differences are important. NOTE: This won't be the final version of
GROUP BY support. I am trying to design it such a way that we can push GROUP
BY down to datanodes.

The changes are added by commit 8326f619.

-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

Hi all,

Postgres-XC 0.9.4 was available in the GIT repository since last week,
but now the tar file and all the manuals have been released there:
https://sourceforge.net/projects/postgres-xc/files/Version_0.9.4/
-- 
Thanks,

Michael Paquier
http://michael.otacoo.com

Year.   I believe 1st cut is very important to make the solution
general.  We used to take the second cut first and this influenced the
cover range of the statment.  (It was very useful to run DBT-1
though).

Regards;
----------
Koichi Suzuki



2011/4/8 Ashutosh Bapat <ash...@en...>:
> Before I forget, Koichi, your sourceforge address bounced, so my mail only
> reached Mason. Thanks Mason for including others in the thread again.
>
> On Fri, Apr 8, 2011 at 6:04 AM, Koichi Suzuki <koi...@gm...> wrote:
>>
>> Thank you for valuable advice.   We also should think how GROUP BY can
>> be pushed down to datanodes so that coordinator can simply merge the
>> result.
>
> If I understand it right, Mason has already given solution to that problem.
> We push the groupby down to datanodes with additional order by clause
> (ordering based on the group by expressions) on top of them (this looks
> tricky if there are already other order by clauses). Thus the data we
> collect at coordinator is already grouped per datanode and all coordinator
> has to do is to consolidate each row from the data nodes in order like we do
> in merge sort.
>
> I see that we may have to do this in steps.
> 1. first cut - implement to apply group by only at coordinator. Not so
> efficient, but will make group by work
> 2. second cut - implement pushing down group by to the data nodes, better
> than one but still the grouping at coordinator is not that efficient
> 3. third cut - implement above idea fully
>
> We might be able to do 1 and 2 in the first cut itself. But this is too
> early to say anything. I will get back once, I know things better.
>
> Mason has also pointed to the possibility of distributing grouping phase at
> coordinator across datanodes (in third cut) so that coordinator is not
> loaded if there are too many columns in the group by. But that requires the
> infrastructure to ship rows from coordinator to datanodes. This
> infrastructure is not place, I think. So that is a far possibility for now.
>
>
>>
>> ----------
>> Koichi Suzuki
>>
>>
>>
>> 2011/4/7 Mason <ma...@us...>:
>> > I looked at the schedule.
>> >
>> > I am not sure about the planned design for GROUP BY, but originally
>> > Andrei was planning on making it somewhat similar to ORDER BY, how
>> > ORDER BY does a merge sort on the coordinator, based on sorted results
>> > from the data nodes. Each data node could do the beginning phase of
>> > aggregation in groups and then sort the output in the same manner of
>> > the groups. Then, the coordinator could do the last step of
>> > aggregation with like groups, which is easy to get them on the fly
>> > because of the sorting coming in from the data nodes (and avoiding
>> > materialization).
>> >
>> > This should work pretty well.  One drawback is if they chose a GROUP
>> > BY clause with many groups (many = thousands+). Then some parallelism
>> > is lost because of the final phase being done in only one place, on
>> > the Coordinator. GridSQL spreads out the final aggregation phase
>> > amongst all the data nodes, moving like groups to the same node to get
>> > more parallelism. I think row shipping infrastructure might have to be
>> > in place first before implementing that, and there will be a
>> > noticeable benefit only once there are many, many groups, so I don't
>> > see it being a critical thing at this phase and can be added later.
>> >
>> > Regards,
>> >
>> > Mason
>> >
>> >
>> >
>> > On Thu, Apr 7, 2011 at 5:27 AM, Koichi Suzuki
>> > <koi...@us...> wrote:
>> >> Project "Postgres-XC documentation".
>> >>
>> >> The branch, master has been updated
>> >>       via  62434399fdd57aff2701e3e5e97fed619f6d6820 (commit)
>> >>      from  252519c2be5309a3682b0ee895cf040083ae1784 (commit)
>> >>
>> >>
>> >> - Log -----------------------------------------------------------------
>> >> commit 62434399fdd57aff2701e3e5e97fed619f6d6820
>> >> Author: Koichi Suzuki <koi...@gm...>
>> >> Date:   Thu Apr 7 18:27:26 2011 +0900
>> >>
>> >>    1. Added 2011FYQ1 schedule for each member.
>> >>    2. Modified my progress sheet of Reference Manual.
>> >>
>> >>                     -- Koichi Suzuki
>> >>
>> >> diff --git a/progress/2011FYQ1_Schedule.ods
>> >> b/progress/2011FYQ1_Schedule.ods
>> >> new file mode 100755
>> >> index 0000000..5e24d37
>> >> Binary files /dev/null and b/progress/2011FYQ1_Schedule.ods differ
>> >> diff --git a/progress/documentation-progress.ods
>> >> b/progress/documentation-progress.ods
>> >> index 277aade..2c8577e 100644
>> >> Binary files a/progress/documentation-progress.ods and
>> >> b/progress/documentation-progress.ods differ
>> >>
>> >> -----------------------------------------------------------------------
>> >>
>> >> Summary of changes:
>> >>  progress/2011FYQ1_Schedule.ods      |  Bin 0 -> 22147 bytes
>> >>  progress/documentation-progress.ods |  Bin 16883 -> 19519 bytes
>> >>  2 files changed, 0 insertions(+), 0 deletions(-)
>> >>  create mode 100755 progress/2011FYQ1_Schedule.ods
>> >>
>> >>
>> >> hooks/post-receive
>> >> --
>> >> Postgres-XC documentation
>> >>
>> >>
>> >> ------------------------------------------------------------------------------
>> >> Xperia(TM) PLAY
>> >> It's a major breakthrough. An authentic gaming
>> >> smartphone on the nation's most reliable network.
>> >> And it wants your games.
>> >> http://p.sf.net/sfu/verizon-sfdev
>> >> _______________________________________________
>> >> Postgres-xc-committers mailing list
>> >> Pos...@li...
>> >> https://lists.sourceforge.net/lists/listinfo/postgres-xc-committers
>> >>
>> >
>> >
>> > ------------------------------------------------------------------------------
>> > Xperia(TM) PLAY
>> > It's a major breakthrough. An authentic gaming
>> > smartphone on the nation's most reliable network.
>> > And it wants your games.
>> > http://p.sf.net/sfu/verizon-sfdev
>> > _______________________________________________
>> > Postgres-xc-committers mailing list
>> > Pos...@li...
>> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-committers
>> >
>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Enterprise Postgres Company
>
>

Hi,

Actually the ORDER BY is not always has to be pushed down to data nodes.
Postgres may decide to group by hash. In this case sorting is unnecessary
operation.
However it may be a problem to determine on coordinator, if data node is
going to use sort or hash grouping.
Aggregate functions are already changed so values are pre-aggregated on
datanodes, and coordinator completes aggregation. This should not be a
problem.

2011/4/8 Ashutosh Bapat <ash...@en...>

> Before I forget, Koichi, your sourceforge address bounced, so my mail only
> reached Mason. Thanks Mason for including others in the thread again.
>
> On Fri, Apr 8, 2011 at 6:04 AM, Koichi Suzuki <koi...@gm...>wrote:
>
>> Thank you for valuable advice.   We also should think how GROUP BY can
>> be pushed down to datanodes so that coordinator can simply merge the
>> result.
>>
>
> If I understand it right, Mason has already given solution to that problem.
> We push the groupby down to datanodes with additional order by clause
> (ordering based on the group by expressions) on top of them (this looks
> tricky if there are already other order by clauses). Thus the data we
> collect at coordinator is already grouped per datanode and all coordinator
> has to do is to consolidate each row from the data nodes in order like we do
> in merge sort.
>
> I see that we may have to do this in steps.
> 1. first cut - implement to apply group by only at coordinator. Not so
> efficient, but will make group by work
> 2. second cut - implement pushing down group by to the data nodes, better
> than one but still the grouping at coordinator is not that efficient
> 3. third cut - implement above idea fully
>
> We might be able to do 1 and 2 in the first cut itself. But this is too
> early to say anything. I will get back once, I know things better.
>
> Mason has also pointed to the possibility of distributing grouping phase at
> coordinator across datanodes (in third cut) so that coordinator is not
> loaded if there are too many columns in the group by. But that requires the
> infrastructure to ship rows from coordinator to datanodes. This
> infrastructure is not place, I think. So that is a far possibility for now.
>
>
>
>>
>> ----------
>> Koichi Suzuki
>>
>>
>>
>> 2011/4/7 Mason <ma...@us...>:
>> > I looked at the schedule.
>> >
>> > I am not sure about the planned design for GROUP BY, but originally
>> > Andrei was planning on making it somewhat similar to ORDER BY, how
>> > ORDER BY does a merge sort on the coordinator, based on sorted results
>> > from the data nodes. Each data node could do the beginning phase of
>> > aggregation in groups and then sort the output in the same manner of
>> > the groups. Then, the coordinator could do the last step of
>> > aggregation with like groups, which is easy to get them on the fly
>> > because of the sorting coming in from the data nodes (and avoiding
>> > materialization).
>> >
>> > This should work pretty well.  One drawback is if they chose a GROUP
>> > BY clause with many groups (many = thousands+). Then some parallelism
>> > is lost because of the final phase being done in only one place, on
>> > the Coordinator. GridSQL spreads out the final aggregation phase
>> > amongst all the data nodes, moving like groups to the same node to get
>> > more parallelism. I think row shipping infrastructure might have to be
>> > in place first before implementing that, and there will be a
>> > noticeable benefit only once there are many, many groups, so I don't
>> > see it being a critical thing at this phase and can be added later.
>> >
>> > Regards,
>> >
>> > Mason
>> >
>> >
>> >
>> > On Thu, Apr 7, 2011 at 5:27 AM, Koichi Suzuki
>> > <koi...@us...> wrote:
>> >> Project "Postgres-XC documentation".
>> >>
>> >> The branch, master has been updated
>> >>       via  62434399fdd57aff2701e3e5e97fed619f6d6820 (commit)
>> >>      from  252519c2be5309a3682b0ee895cf040083ae1784 (commit)
>> >>
>> >>
>> >> - Log -----------------------------------------------------------------
>> >> commit 62434399fdd57aff2701e3e5e97fed619f6d6820
>> >> Author: Koichi Suzuki <koi...@gm...>
>> >> Date:   Thu Apr 7 18:27:26 2011 +0900
>> >>
>> >>    1. Added 2011FYQ1 schedule for each member.
>> >>    2. Modified my progress sheet of Reference Manual.
>> >>
>> >>                     -- Koichi Suzuki
>> >>
>> >> diff --git a/progress/2011FYQ1_Schedule.ods
>> b/progress/2011FYQ1_Schedule.ods
>> >> new file mode 100755
>> >> index 0000000..5e24d37
>> >> Binary files /dev/null and b/progress/2011FYQ1_Schedule.ods differ
>> >> diff --git a/progress/documentation-progress.ods
>> b/progress/documentation-progress.ods
>> >> index 277aade..2c8577e 100644
>> >> Binary files a/progress/documentation-progress.ods and
>> b/progress/documentation-progress.ods differ
>> >>
>> >> -----------------------------------------------------------------------
>> >>
>> >> Summary of changes:
>> >>  progress/2011FYQ1_Schedule.ods      |  Bin 0 -> 22147 bytes
>> >>  progress/documentation-progress.ods |  Bin 16883 -> 19519 bytes
>> >>  2 files changed, 0 insertions(+), 0 deletions(-)
>> >>  create mode 100755 progress/2011FYQ1_Schedule.ods
>> >>
>> >>
>> >> hooks/post-receive
>> >> --
>> >> Postgres-XC documentation
>> >>
>> >>
>> ------------------------------------------------------------------------------
>> >> Xperia(TM) PLAY
>> >> It's a major breakthrough. An authentic gaming
>> >> smartphone on the nation's most reliable network.
>> >> And it wants your games.
>> >> http://p.sf.net/sfu/verizon-sfdev
>> >> _______________________________________________
>> >> Postgres-xc-committers mailing list
>> >> Pos...@li...
>> >> https://lists.sourceforge.net/lists/listinfo/postgres-xc-committers
>> >>
>> >
>> >
>> ------------------------------------------------------------------------------
>> > Xperia(TM) PLAY
>> > It's a major breakthrough. An authentic gaming
>> > smartphone on the nation's most reliable network.
>> > And it wants your games.
>> > http://p.sf.net/sfu/verizon-sfdev
>> > _______________________________________________
>> > Postgres-xc-committers mailing list
>> > Pos...@li...
>> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-committers
>> >
>>
>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Enterprise Postgres Company
>
>
>
> ------------------------------------------------------------------------------
> Xperia(TM) PLAY
> It's a major breakthrough. An authentic gaming
> smartphone on the nation's most reliable network.
> And it wants your games.
> http://p.sf.net/sfu/verizon-sfdev
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>


-- 
Best regards,
Andrei Martsinchyk                   mailto:and...@gm...

Before I forget, Koichi, your sourceforge address bounced, so my mail only
reached Mason. Thanks Mason for including others in the thread again.

On Fri, Apr 8, 2011 at 6:04 AM, Koichi Suzuki <koi...@gm...> wrote:

> Thank you for valuable advice.   We also should think how GROUP BY can
> be pushed down to datanodes so that coordinator can simply merge the
> result.
>

If I understand it right, Mason has already given solution to that problem.
We push the groupby down to datanodes with additional order by clause
(ordering based on the group by expressions) on top of them (this looks
tricky if there are already other order by clauses). Thus the data we
collect at coordinator is already grouped per datanode and all coordinator
has to do is to consolidate each row from the data nodes in order like we do
in merge sort.

I see that we may have to do this in steps.
1. first cut - implement to apply group by only at coordinator. Not so
efficient, but will make group by work
2. second cut - implement pushing down group by to the data nodes, better
than one but still the grouping at coordinator is not that efficient
3. third cut - implement above idea fully

We might be able to do 1 and 2 in the first cut itself. But this is too
early to say anything. I will get back once, I know things better.

Mason has also pointed to the possibility of distributing grouping phase at
coordinator across datanodes (in third cut) so that coordinator is not
loaded if there are too many columns in the group by. But that requires the
infrastructure to ship rows from coordinator to datanodes. This
infrastructure is not place, I think. So that is a far possibility for now.



>
> ----------
> Koichi Suzuki
>
>
>
> 2011/4/7 Mason <ma...@us...>:
> > I looked at the schedule.
> >
> > I am not sure about the planned design for GROUP BY, but originally
> > Andrei was planning on making it somewhat similar to ORDER BY, how
> > ORDER BY does a merge sort on the coordinator, based on sorted results
> > from the data nodes. Each data node could do the beginning phase of
> > aggregation in groups and then sort the output in the same manner of
> > the groups. Then, the coordinator could do the last step of
> > aggregation with like groups, which is easy to get them on the fly
> > because of the sorting coming in from the data nodes (and avoiding
> > materialization).
> >
> > This should work pretty well.  One drawback is if they chose a GROUP
> > BY clause with many groups (many = thousands+). Then some parallelism
> > is lost because of the final phase being done in only one place, on
> > the Coordinator. GridSQL spreads out the final aggregation phase
> > amongst all the data nodes, moving like groups to the same node to get
> > more parallelism. I think row shipping infrastructure might have to be
> > in place first before implementing that, and there will be a
> > noticeable benefit only once there are many, many groups, so I don't
> > see it being a critical thing at this phase and can be added later.
> >
> > Regards,
> >
> > Mason
> >
> >
> >
> > On Thu, Apr 7, 2011 at 5:27 AM, Koichi Suzuki
> > <koi...@us...> wrote:
> >> Project "Postgres-XC documentation".
> >>
> >> The branch, master has been updated
> >>       via  62434399fdd57aff2701e3e5e97fed619f6d6820 (commit)
> >>      from  252519c2be5309a3682b0ee895cf040083ae1784 (commit)
> >>
> >>
> >> - Log -----------------------------------------------------------------
> >> commit 62434399fdd57aff2701e3e5e97fed619f6d6820
> >> Author: Koichi Suzuki <koi...@gm...>
> >> Date:   Thu Apr 7 18:27:26 2011 +0900
> >>
> >>    1. Added 2011FYQ1 schedule for each member.
> >>    2. Modified my progress sheet of Reference Manual.
> >>
> >>                     -- Koichi Suzuki
> >>
> >> diff --git a/progress/2011FYQ1_Schedule.ods
> b/progress/2011FYQ1_Schedule.ods
> >> new file mode 100755
> >> index 0000000..5e24d37
> >> Binary files /dev/null and b/progress/2011FYQ1_Schedule.ods differ
> >> diff --git a/progress/documentation-progress.ods
> b/progress/documentation-progress.ods
> >> index 277aade..2c8577e 100644
> >> Binary files a/progress/documentation-progress.ods and
> b/progress/documentation-progress.ods differ
> >>
> >> -----------------------------------------------------------------------
> >>
> >> Summary of changes:
> >>  progress/2011FYQ1_Schedule.ods      |  Bin 0 -> 22147 bytes
> >>  progress/documentation-progress.ods |  Bin 16883 -> 19519 bytes
> >>  2 files changed, 0 insertions(+), 0 deletions(-)
> >>  create mode 100755 progress/2011FYQ1_Schedule.ods
> >>
> >>
> >> hooks/post-receive
> >> --
> >> Postgres-XC documentation
> >>
> >>
> ------------------------------------------------------------------------------
> >> Xperia(TM) PLAY
> >> It's a major breakthrough. An authentic gaming
> >> smartphone on the nation's most reliable network.
> >> And it wants your games.
> >> http://p.sf.net/sfu/verizon-sfdev
> >> _______________________________________________
> >> Postgres-xc-committers mailing list
> >> Pos...@li...
> >> https://lists.sourceforge.net/lists/listinfo/postgres-xc-committers
> >>
> >
> >
> ------------------------------------------------------------------------------
> > Xperia(TM) PLAY
> > It's a major breakthrough. An authentic gaming
> > smartphone on the nation's most reliable network.
> > And it wants your games.
> > http://p.sf.net/sfu/verizon-sfdev
> > _______________________________________________
> > Postgres-xc-committers mailing list
> > Pos...@li...
> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-committers
> >
>



-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

Thank you for valuable advice.   We also should think how GROUP BY can
be pushed down to datanodes so that coordinator can simply merge the
result.

----------
Koichi Suzuki



2011/4/7 Mason <ma...@us...>:
> I looked at the schedule.
>
> I am not sure about the planned design for GROUP BY, but originally
> Andrei was planning on making it somewhat similar to ORDER BY, how
> ORDER BY does a merge sort on the coordinator, based on sorted results
> from the data nodes. Each data node could do the beginning phase of
> aggregation in groups and then sort the output in the same manner of
> the groups. Then, the coordinator could do the last step of
> aggregation with like groups, which is easy to get them on the fly
> because of the sorting coming in from the data nodes (and avoiding
> materialization).
>
> This should work pretty well.  One drawback is if they chose a GROUP
> BY clause with many groups (many = thousands+). Then some parallelism
> is lost because of the final phase being done in only one place, on
> the Coordinator. GridSQL spreads out the final aggregation phase
> amongst all the data nodes, moving like groups to the same node to get
> more parallelism. I think row shipping infrastructure might have to be
> in place first before implementing that, and there will be a
> noticeable benefit only once there are many, many groups, so I don't
> see it being a critical thing at this phase and can be added later.
>
> Regards,
>
> Mason
>
>
>
> On Thu, Apr 7, 2011 at 5:27 AM, Koichi Suzuki
> <koi...@us...> wrote:
>> Project "Postgres-XC documentation".
>>
>> The branch, master has been updated
>>       via  62434399fdd57aff2701e3e5e97fed619f6d6820 (commit)
>>      from  252519c2be5309a3682b0ee895cf040083ae1784 (commit)
>>
>>
>> - Log -----------------------------------------------------------------
>> commit 62434399fdd57aff2701e3e5e97fed619f6d6820
>> Author: Koichi Suzuki <koi...@gm...>
>> Date:   Thu Apr 7 18:27:26 2011 +0900
>>
>>    1. Added 2011FYQ1 schedule for each member.
>>    2. Modified my progress sheet of Reference Manual.
>>
>>                     -- Koichi Suzuki
>>
>> diff --git a/progress/2011FYQ1_Schedule.ods b/progress/2011FYQ1_Schedule.ods
>> new file mode 100755
>> index 0000000..5e24d37
>> Binary files /dev/null and b/progress/2011FYQ1_Schedule.ods differ
>> diff --git a/progress/documentation-progress.ods b/progress/documentation-progress.ods
>> index 277aade..2c8577e 100644
>> Binary files a/progress/documentation-progress.ods and b/progress/documentation-progress.ods differ
>>
>> -----------------------------------------------------------------------
>>
>> Summary of changes:
>>  progress/2011FYQ1_Schedule.ods      |  Bin 0 -> 22147 bytes
>>  progress/documentation-progress.ods |  Bin 16883 -> 19519 bytes
>>  2 files changed, 0 insertions(+), 0 deletions(-)
>>  create mode 100755 progress/2011FYQ1_Schedule.ods
>>
>>
>> hooks/post-receive
>> --
>> Postgres-XC documentation
>>
>> ------------------------------------------------------------------------------
>> Xperia(TM) PLAY
>> It's a major breakthrough. An authentic gaming
>> smartphone on the nation's most reliable network.
>> And it wants your games.
>> http://p.sf.net/sfu/verizon-sfdev
>> _______________________________________________
>> Postgres-xc-committers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-committers
>>
>
> ------------------------------------------------------------------------------
> Xperia(TM) PLAY
> It's a major breakthrough. An authentic gaming
> smartphone on the nation's most reliable network.
> And it wants your games.
> http://p.sf.net/sfu/verizon-sfdev
> _______________________________________________
> Postgres-xc-committers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-committers
>

I looked at the schedule.

I am not sure about the planned design for GROUP BY, but originally
Andrei was planning on making it somewhat similar to ORDER BY, how
ORDER BY does a merge sort on the coordinator, based on sorted results
from the data nodes. Each data node could do the beginning phase of
aggregation in groups and then sort the output in the same manner of
the groups. Then, the coordinator could do the last step of
aggregation with like groups, which is easy to get them on the fly
because of the sorting coming in from the data nodes (and avoiding
materialization).

This should work pretty well.  One drawback is if they chose a GROUP
BY clause with many groups (many = thousands+). Then some parallelism
is lost because of the final phase being done in only one place, on
the Coordinator. GridSQL spreads out the final aggregation phase
amongst all the data nodes, moving like groups to the same node to get
more parallelism. I think row shipping infrastructure might have to be
in place first before implementing that, and there will be a
noticeable benefit only once there are many, many groups, so I don't
see it being a critical thing at this phase and can be added later.

Regards,

Mason



On Thu, Apr 7, 2011 at 5:27 AM, Koichi Suzuki
<koi...@us...> wrote:
> Project "Postgres-XC documentation".
>
> The branch, master has been updated
>       via  62434399fdd57aff2701e3e5e97fed619f6d6820 (commit)
>      from  252519c2be5309a3682b0ee895cf040083ae1784 (commit)
>
>
> - Log -----------------------------------------------------------------
> commit 62434399fdd57aff2701e3e5e97fed619f6d6820
> Author: Koichi Suzuki <koi...@gm...>
> Date:   Thu Apr 7 18:27:26 2011 +0900
>
>    1. Added 2011FYQ1 schedule for each member.
>    2. Modified my progress sheet of Reference Manual.
>
>                     -- Koichi Suzuki
>
> diff --git a/progress/2011FYQ1_Schedule.ods b/progress/2011FYQ1_Schedule.ods
> new file mode 100755
> index 0000000..5e24d37
> Binary files /dev/null and b/progress/2011FYQ1_Schedule.ods differ
> diff --git a/progress/documentation-progress.ods b/progress/documentation-progress.ods
> index 277aade..2c8577e 100644
> Binary files a/progress/documentation-progress.ods and b/progress/documentation-progress.ods differ
>
> -----------------------------------------------------------------------
>
> Summary of changes:
>  progress/2011FYQ1_Schedule.ods      |  Bin 0 -> 22147 bytes
>  progress/documentation-progress.ods |  Bin 16883 -> 19519 bytes
>  2 files changed, 0 insertions(+), 0 deletions(-)
>  create mode 100755 progress/2011FYQ1_Schedule.ods
>
>
> hooks/post-receive
> --
> Postgres-XC documentation
>
> ------------------------------------------------------------------------------
> Xperia(TM) PLAY
> It's a major breakthrough. An authentic gaming
> smartphone on the nation's most reliable network.
> And it wants your games.
> http://p.sf.net/sfu/verizon-sfdev
> _______________________________________________
> Postgres-xc-committers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-committers
>

I added the latest version of the patch in the patch tracker.
-- 
Thanks,

Michael Paquier
http://michael.otacoo.com

Hi,

Just to let everybody know about the status of DBT-1 with latest Postgres-XC
release...

Related to bug 3240359 and some others, it is important to know that
runnning DBT-1 with version 0.9.4 may result in a high number of errors.
The origin of this problem is the aggregate function count.
Because of the fact that when doing a count XC returns 0 rows instead of 1
row with 0 as result, this is resulting in a huge amount of errors when
running DBT1.
So basically we are unable to make proper DBT-1 measurements with 0.9.4.

The query being the origin of the error is from addToSC.h:
SELECT count(scl_sc_id)
FROM   shopping_cart_line
WHERE  scl_sc_id=%lld;
-- 
Thanks,

Michael Paquier
http://michael.otacoo.com

Benny;

I'm afraid I didn't find the the patch attached.   I'll register it to
the patch track for the review.

Thank you;
----------
Koichi Suzuki



2011/4/6 wangxiong.gm <wan...@gm...>:
>
> Dears,
>
> When a table is dropped, the relative distribution information will be kept
> in pgxc_class catalog now. The patch can fix such a problem. The relative
> infomation on the table distribution will be removed when drop a table.
>
> I hope it's helpful.
>
> Best regards,
> Benny
> 2011-04-06
> ________________________________
> wangxiong.gm
> ------------------------------------------------------------------------------
> Xperia(TM) PLAY
> It's a major breakthrough. An authentic gaming
> smartphone on the nation's most reliable network.
> And it wants your games.
> http://p.sf.net/sfu/verizon-sfdev
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>

Thanks for your help.

I went as far as, setting up a cluster with 1 coordinator, 2 data nodes and
a gtm. I could create a table, insert data into it and fire a query on
coordinator and on individual data nodes and drop the table.

On Wed, Apr 6, 2011 at 5:29 AM, Michael Paquier
<mic...@gm...>wrote:

> About pgxc.conf, it was used by a utility called pgxc_ddl that is not
> really necessary since DDL Synchronization in done through all the
> Coordinators.
> I made a cleanup of this utility and moved it to folder
> src/bin/pgxc/pgxc_ddl with commit 490c08dd37fa44af57cd3a2b3e931ef4f3a94853.
>
>
> --
> Thanks,
>
> Michael Paquier
> http://michael.otacoo.com
>



-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

Dears,

When a table is dropped, the relative distribution information will be kept in pgxc_class catalog now. The patch can fix such a problem. The relative infomation on the table distribution will be removed when drop a table.

I hope it's helpful.

Best regards,
Benny
2011-04-06 



wangxiong.gm 

Thanks a lot , I appreciate your efforts and understand the reason of
failure. I will re-test your patch and would let you know soon.

On Wed, Apr 6, 2011 at 1:01 PM, wangxiong.gm <wan...@gm...> wrote:

>
> Hi Abbas,
>
> The enclosure is a modified patch for new Postgres-xc branch on
> bug#3170712.
>
> The reason why your test cases failed is you didn't apply the patch I
> submitted to the old Postgres-xc branch.
>
> Best regards,
> Benny
>
> ------------------
> wangxiong.gm
> 2011-04-06
>
> -------------------------------------------------------------
> 发件人：Abbas Butt
> 发送日期：2011-03-22 14:50:02
> 收件人：xiong wang
> 抄送：
> 主题：Re: Patch for PGXC Bug ID 3170712
>
> Hi,
> Thanks
>
> On Tue, Mar 22, 2011 at 7:54 AM, xiong wang <wan...@gm...>
> wrote:
>
> > Hi Abbas,
> >
> > Thanks for your test.
> >
> > I have tested on the master branch, it works well. So, I think the
> > crash is relative with the merge_postgres_9_0_3 because my patch was
> > tested on the postgres-xc which was not merged with postgres 9.0.3.
> >
> > I will check the bug soon on the branch merge_postgres_9_0_3.
> >
> > Best regards,
> > Benny
> >
> > >2011/3/21 Abbas Butt <abb...@en...>:
> > > Hi,
> > > Just finished testing the submitted patch to fix that issue and found
> > that
> > > there is still a server crash reproducible using the following
> statements
> > > create table rule_test_hs (a int, b int);
> > > create table rule_test_rp (a int, b int) distribute by replication;
> > > create table rule_test_hs1 (a int, b int);
> > > create table rule_test_mo (a int, b int) distribute by modulo(a);
> > > create table rule_test_rb (a int, b int) distribute by round robin;
> > > --instead
> > > create rule ff_instead_ins as on insert to rule_test_rp do instead
> > nothing;
> > > create rule gg_instead_ins as on insert to rule_test_hs do instead
> > nothing;
> > > create rule hh_instead_ins as on insert to rule_test_mo do instead
> > nothing;
> > > create rule ii_instead_ins as on insert to rule_test_rb do instead
> > nothing;
> > > insert into rule_test_rp values (1,2),(2,3);
> > > insert into rule_test_hs values (1,2),(2,3);
> > > insert into rule_test_mo values (1,2),(2,3);
> > > insert into rule_test_rb values (1,2),(2,3);
> > > drop rule ff_instead_ins on rule_test_rp;
> > > drop rule gg_instead_ins on rule_test_hs;
> > > drop rule hh_instead_ins on rule_test_mo;
> > > drop rule ii_instead_ins on rule_test_rb;
> > > create rule bb_also_ins as on insert to rule_test_hs do also insert
> into
> > > rule_test_rp values (new.a,new.b);
> > > create rule cc_also_ins as on insert to rule_test_hs do also insert
> into
> > > rule_test_hs1 values (new.a,new.b);
> > > create rule dd_also_ins as on insert to rule_test_hs do also insert
> into
> > > rule_test_mo values (new.a,new.b);
> > > create rule ee_also_ins as on insert to rule_test_hs do also insert
> into
> > > rule_test_rb values (new.a,new.b);
> > > insert into rule_test_hs values (1,2),(2,3);
> > > PS: I have taken this test from the test cases that you submitted with
> > the
> > > patch.
> > > --
> > > Abbas
> > > Architect
> > > EnterpriseDB Corporation
> > > The Enterprise PostgreSQL Company
> > >
> > > Phone: 92-334-5100153
> > >
> > > Website: www.enterprisedb.com
> > > EnterpriseDB Blog: http://blogs.enterprisedb.com/
> > > Follow us on Twitter: http://www.twitter.com/enterprisedb
> > >
> > > This e-mail message (and any attachment) is intended for the use of
> > > the individual or entity to whom it is addressed. This message
> > > contains information from EnterpriseDB Corporation that may be
> > > privileged, confidential, or exempt from disclosure under applicable
> > > law. If you are not the intended recipient or authorized to receive
> > > this for the intended recipient, any use, dissemination, distribution,
> > > retention, archiving, or copying of this communication is strictly
> > > prohibited. If you have received this e-mail in error, please notify
> > > the sender immediately by reply e-mail and delete this message.
> > >
> >
>
>
>
> --
> --
> Abbas
> Architect
> EnterpriseDB Corporation
> The Enterprise PostgreSQL Company
>
> Phone: 92-334-5100153
>
> Website: www.enterprisedb.com
> EnterpriseDB Blog: http://blogs.enterprisedb.com/
> Follow us on Twitter: http://www.twitter.com/enterprisedb
>
> This e-mail message (and any attachment) is intended for the use of
> the individual or entity to whom it is addressed. This message
> contains information from EnterpriseDB Corporation that may be
> privileged, confidential, or exempt from disclosure under applicable
> law. If you are not the intended recipient or authorized to receive
> this for the intended recipient, any use, dissemination, distribution,
> retention, archiving, or copying of this communication is strictly
> prohibited. If you have received this e-mail in error, please notify
> the sender immediately by reply e-mail and delete this message.
>
>
> ------------------------------------------------------------------------------
> Xperia(TM) PLAY
> It's a major breakthrough. An authentic gaming
> smartphone on the nation's most reliable network.
> And it wants your games.
> http://p.sf.net/sfu/verizon-sfdev
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>

Hi Abbas,

The enclosure is a modified patch for new Postgres-xc branch on bug#3170712.

The reason why your test cases failed is you didn't apply the patch I submitted to the old Postgres-xc branch.

Best regards,
Benny

------------------				 
wangxiong.gm
2011-04-06

-------------------------------------------------------------
发件人：Abbas Butt
发送日期：2011-03-22 14:50:02
收件人：xiong wang
抄送：
主题：Re: Patch for PGXC Bug ID 3170712

Hi,
Thanks

On Tue, Mar 22, 2011 at 7:54 AM, xiong wang <wan...@gm...> wrote:

> Hi Abbas,
>
> Thanks for your test.
>
> I have tested on the master branch, it works well. So, I think the
> crash is relative with the merge_postgres_9_0_3 because my patch was
> tested on the postgres-xc which was not merged with postgres 9.0.3.
>
> I will check the bug soon on the branch merge_postgres_9_0_3.
>
> Best regards,
> Benny
>
> >2011/3/21 Abbas Butt <abb...@en...>:
> > Hi,
> > Just finished testing the submitted patch to fix that issue and found
> that
> > there is still a server crash reproducible using the following statements
> > create table rule_test_hs (a int, b int);
> > create table rule_test_rp (a int, b int) distribute by replication;
> > create table rule_test_hs1 (a int, b int);
> > create table rule_test_mo (a int, b int) distribute by modulo(a);
> > create table rule_test_rb (a int, b int) distribute by round robin;
> > --instead
> > create rule ff_instead_ins as on insert to rule_test_rp do instead
> nothing;
> > create rule gg_instead_ins as on insert to rule_test_hs do instead
> nothing;
> > create rule hh_instead_ins as on insert to rule_test_mo do instead
> nothing;
> > create rule ii_instead_ins as on insert to rule_test_rb do instead
> nothing;
> > insert into rule_test_rp values (1,2),(2,3);
> > insert into rule_test_hs values (1,2),(2,3);
> > insert into rule_test_mo values (1,2),(2,3);
> > insert into rule_test_rb values (1,2),(2,3);
> > drop rule ff_instead_ins on rule_test_rp;
> > drop rule gg_instead_ins on rule_test_hs;
> > drop rule hh_instead_ins on rule_test_mo;
> > drop rule ii_instead_ins on rule_test_rb;
> > create rule bb_also_ins as on insert to rule_test_hs do also insert into
> > rule_test_rp values (new.a,new.b);
> > create rule cc_also_ins as on insert to rule_test_hs do also insert into
> > rule_test_hs1 values (new.a,new.b);
> > create rule dd_also_ins as on insert to rule_test_hs do also insert into
> > rule_test_mo values (new.a,new.b);
> > create rule ee_also_ins as on insert to rule_test_hs do also insert into
> > rule_test_rb values (new.a,new.b);
> > insert into rule_test_hs values (1,2),(2,3);
> > PS: I have taken this test from the test cases that you submitted with
> the
> > patch.
> > --
> > Abbas
> > Architect
> > EnterpriseDB Corporation
> > The Enterprise PostgreSQL Company
> >
> > Phone: 92-334-5100153
> >
> > Website: www.enterprisedb.com
> > EnterpriseDB Blog: http://blogs.enterprisedb.com/
> > Follow us on Twitter: http://www.twitter.com/enterprisedb
> >
> > This e-mail message (and any attachment) is intended for the use of
> > the individual or entity to whom it is addressed. This message
> > contains information from EnterpriseDB Corporation that may be
> > privileged, confidential, or exempt from disclosure under applicable
> > law. If you are not the intended recipient or authorized to receive
> > this for the intended recipient, any use, dissemination, distribution,
> > retention, archiving, or copying of this communication is strictly
> > prohibited. If you have received this e-mail in error, please notify
> > the sender immediately by reply e-mail and delete this message.
> >
>



-- 
--
Abbas
Architect
EnterpriseDB Corporation
The Enterprise PostgreSQL Company

Phone: 92-334-5100153

Website: www.enterprisedb.com
EnterpriseDB Blog: http://blogs.enterprisedb.com/
Follow us on Twitter: http://www.twitter.com/enterprisedb

This e-mail message (and any attachment) is intended for the use of
the individual or entity to whom it is addressed. This message
contains information from EnterpriseDB Corporation that may be
privileged, confidential, or exempt from disclosure under applicable
law. If you are not the intended recipient or authorized to receive
this for the intended recipient, any use, dissemination, distribution,
retention, archiving, or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify
the sender immediately by reply e-mail and delete this message.

About pgxc.conf, it was used by a utility called pgxc_ddl that is not really
necessary since DDL Synchronization in done through all the Coordinators.
I made a cleanup of this utility and moved it to folder
src/bin/pgxc/pgxc_ddl with commit 490c08dd37fa44af57cd3a2b3e931ef4f3a94853.

-- 
Thanks,

Michael Paquier
http://michael.otacoo.com

On Tue, Apr 5, 2011 at 6:19 PM, Ashutosh Bapat <
ash...@en...> wrote:

> Hi All,
> I am going through Postgres-XC, Install Manual, Version 0.9.3 for setting
> up PG-XC on my laptop. On page 7, section 2.4.1 it says to set gtm_coord_id
> in postgresql.conf. But in the installation I have, I didn't find any such
> parameter mentioned. Neither I see it in code. Is this parameter valid
> anymore?
>
Indeed, there is an error in the manuals.
The parameter is called pgxc_node_id. It has been introduced by node
registration feature.
-- 
Thanks,

Michael Paquier
http://michael.otacoo.com

It was jut for 0.9.2 but survived in 0.9.3.   Please let me double
check tomorrow if remains in the later version.

Regards;
----------
Koichi Suzuki



2011/4/5 Ashutosh Bapat <ash...@en...>:
> Hi Suzuki-san,
> Section 2.4.3 talks about pgxc.conf. That too is missing from the
> installation, I have. May be that too is obsolete.
>
> On Tue, Apr 5, 2011 at 2:58 PM, Koichi Suzuki <koi...@gm...> wrote:
>>
>> Thank you for pointing this out.
>>
>> When Michael is back, I'd like to check manuals, especially GUCs,
>> catalogues and their entries.
>>
>> Regards;
>> ----------
>> Koichi Suzuki
>>
>>
>>
>> 2011/4/5 Ashutosh Bapat <ash...@en...>:
>> > Hi All,
>> > I am going through Postgres-XC, Install Manual, Version 0.9.3 for
>> > setting up
>> > PG-XC on my laptop. On page 7, section 2.4.1 it says to set gtm_coord_id
>> > in
>> > postgresql.conf. But in the installation I have, I didn't find any such
>> > parameter mentioned. Neither I see it in code. Is this parameter valid
>> > anymore?
>> >
>> >
>> > --
>> > Best Wishes,
>> > Ashutosh Bapat
>> > EntepriseDB Corporation
>> > The Enterprise Postgres Company
>> >
>> >
>> >
>> > ------------------------------------------------------------------------------
>> > Xperia(TM) PLAY
>> > It's a major breakthrough. An authentic gaming
>> > smartphone on the nation's most reliable network.
>> > And it wants your games.
>> > http://p.sf.net/sfu/verizon-sfdev
>> > _______________________________________________
>> > Postgres-xc-developers mailing list
>> > Pos...@li...
>> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>> >
>> >
>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Enterprise Postgres Company
>
>

Hi Suzuki-san,
Section 2.4.3 talks about pgxc.conf. That too is missing from the
installation, I have. May be that too is obsolete.

On Tue, Apr 5, 2011 at 2:58 PM, Koichi Suzuki <koi...@gm...> wrote:

> Thank you for pointing this out.
>
> When Michael is back, I'd like to check manuals, especially GUCs,
> catalogues and their entries.
>
> Regards;
> ----------
> Koichi Suzuki
>
>
>
> 2011/4/5 Ashutosh Bapat <ash...@en...>:
> > Hi All,
> > I am going through Postgres-XC, Install Manual, Version 0.9.3 for setting
> up
> > PG-XC on my laptop. On page 7, section 2.4.1 it says to set gtm_coord_id
> in
> > postgresql.conf. But in the installation I have, I didn't find any such
> > parameter mentioned. Neither I see it in code. Is this parameter valid
> > anymore?
> >
> >
> > --
> > Best Wishes,
> > Ashutosh Bapat
> > EntepriseDB Corporation
> > The Enterprise Postgres Company
> >
> >
> >
> ------------------------------------------------------------------------------
> > Xperia(TM) PLAY
> > It's a major breakthrough. An authentic gaming
> > smartphone on the nation's most reliable network.
> > And it wants your games.
> > http://p.sf.net/sfu/verizon-sfdev
> > _______________________________________________
> > Postgres-xc-developers mailing list
> > Pos...@li...
> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
> >
> >
>



-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

Thank you for pointing this out.

When Michael is back, I'd like to check manuals, especially GUCs,
catalogues and their entries.

Regards;
----------
Koichi Suzuki



2011/4/5 Ashutosh Bapat <ash...@en...>:
> Hi All,
> I am going through Postgres-XC, Install Manual, Version 0.9.3 for setting up
> PG-XC on my laptop. On page 7, section 2.4.1 it says to set gtm_coord_id in
> postgresql.conf. But in the installation I have, I didn't find any such
> parameter mentioned. Neither I see it in code. Is this parameter valid
> anymore?
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Enterprise Postgres Company
>
>
> ------------------------------------------------------------------------------
> Xperia(TM) PLAY
> It's a major breakthrough. An authentic gaming
> smartphone on the nation's most reliable network.
> And it wants your games.
> http://p.sf.net/sfu/verizon-sfdev
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>

Hi All,
I am going through Postgres-XC, Install Manual, Version 0.9.3 for setting up
PG-XC on my laptop. On page 7, section 2.4.1 it says to set gtm_coord_id in
postgresql.conf. But in the installation I have, I didn't find any such
parameter mentioned. Neither I see it in code. Is this parameter valid
anymore?


-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

On Tue, Apr 5, 2011 at 1:47 PM, Koichi Suzuki <koi...@gm...> wrote:

> Done.  Please review the sourceforge page.
> ----------
> Koichi Suzuki
>
>
Thanks. I can see my name in newest members.


>
>
> 2011/4/5 Ashutosh Bapat <ash...@en...>:
> >
> >
> > On Tue, Apr 5, 2011 at 1:30 PM, Koichi Suzuki <ko...@in...>
> > wrote:
> >>
> >> Please let me know your sourceforge account name.   You need access
> >> pribilege to the resource.
> >>
> >
> > It's ashutoshbapat.
> >
> >>
> >> Regards;
> >> ---
> >> Koichi
> >>
> >>
> >> On Tue, 5 Apr 2011 11:00:50 +0530
> >> Ashutosh Bapat <ash...@en...> wrote:
> >>
> >> > On Mon, Apr 4, 2011 at 6:56 AM, Koichi Suzuki
> >> > <ko...@in...>wrote:
> >> >
> >> > > Hi, Ashutosh;
> >> > >
> >> > > Welcome to the project!  It's so nice that you are already familiar
> >> > > with
> >> > > the internal.
> >> > >
> >> > >
> >> > Thanks Suzuki-san. Looking forward to contribute to this wonderful
> >> > project.
> >> >
> >> >
> >> > > Best Regards;
> >> > > ---
> >> > > Koichi Suzuki
> >> > >
> >> > > On Fri, 1 Apr 2011 17:10:45 +0530
> >> > > Ashutosh Bapat <ash...@en...> wrote:
> >> > >
> >> > > > Hi Michael,
> >> > > > You want to make sure that the user which makes connection to data
> >> > > > nodes
> >> > > has
> >> > > > privileges to execute set role on data node. I think, not everyone
> >> > > > is
> >> > > > allowed to execute 'set role' using any user name.
> >> > > >
> >> > > > On Fri, Apr 1, 2011 at 4:59 PM, Michael Paquier
> >> > > > <mic...@gm...>wrote:
> >> > > >
> >> > > > > Just another point. XC is using session user to build pooler
> >> > > connections.
> >> > > > > When you change the current user with set role, the same session
> >> > > > > connections are being used, but access to data or tables is
> >> > > > > managed by
> >> > > > > Coordinators with current user value.
> >> > > > >
> >> > > > > --
> >> > > > > Thanks,
> >> > > > >
> >> > > > > Michael Paquier
> >> > > > > http://michael.otacoo.com
> >> > > > >
> >> > > > >
> >> > > > >
> >> > >
> >> > >
> ------------------------------------------------------------------------------
> >> > > > > Create and publish websites with WebMatrix
> >> > > > > Use the most popular FREE web apps or write code yourself;
> >> > > > > WebMatrix provides all the features you need to develop and
> >> > > > > publish your website. http://p.sf.net/sfu/ms-webmatrix-sf
> >> > > > >
> >> > > > > _______________________________________________
> >> > > > > Postgres-xc-developers mailing list
> >> > > > > Pos...@li...
> >> > > > >
> >> > > > >
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
> >> > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Best Wishes,
> >> > > > Ashutosh Bapat
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Best Wishes,
> >> > Ashutosh Bapat
> >
> >
> >
> > --
> > Best Wishes,
> > Ashutosh Bapat
> >
> >
> >
> ------------------------------------------------------------------------------
> > Xperia(TM) PLAY
> > It's a major breakthrough. An authentic gaming
> > smartphone on the nation's most reliable network.
> > And it wants your games.
> > http://p.sf.net/sfu/verizon-sfdev
> > _______________________________________________
> > Postgres-xc-developers mailing list
> > Pos...@li...
> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
> >
> >
>



-- 
Best Wishes,
Ashutosh Bapat

Done.  Please review the sourceforge page.
----------
Koichi Suzuki



2011/4/5 Ashutosh Bapat <ash...@en...>:
>
>
> On Tue, Apr 5, 2011 at 1:30 PM, Koichi Suzuki <ko...@in...>
> wrote:
>>
>> Please let me know your sourceforge account name.   You need access
>> pribilege to the resource.
>>
>
> It's ashutoshbapat.
>
>>
>> Regards;
>> ---
>> Koichi
>>
>>
>> On Tue, 5 Apr 2011 11:00:50 +0530
>> Ashutosh Bapat <ash...@en...> wrote:
>>
>> > On Mon, Apr 4, 2011 at 6:56 AM, Koichi Suzuki
>> > <ko...@in...>wrote:
>> >
>> > > Hi, Ashutosh;
>> > >
>> > > Welcome to the project!  It's so nice that you are already familiar
>> > > with
>> > > the internal.
>> > >
>> > >
>> > Thanks Suzuki-san. Looking forward to contribute to this wonderful
>> > project.
>> >
>> >
>> > > Best Regards;
>> > > ---
>> > > Koichi Suzuki
>> > >
>> > > On Fri, 1 Apr 2011 17:10:45 +0530
>> > > Ashutosh Bapat <ash...@en...> wrote:
>> > >
>> > > > Hi Michael,
>> > > > You want to make sure that the user which makes connection to data
>> > > > nodes
>> > > has
>> > > > privileges to execute set role on data node. I think, not everyone
>> > > > is
>> > > > allowed to execute 'set role' using any user name.
>> > > >
>> > > > On Fri, Apr 1, 2011 at 4:59 PM, Michael Paquier
>> > > > <mic...@gm...>wrote:
>> > > >
>> > > > > Just another point. XC is using session user to build pooler
>> > > connections.
>> > > > > When you change the current user with set role, the same session
>> > > > > connections are being used, but access to data or tables is
>> > > > > managed by
>> > > > > Coordinators with current user value.
>> > > > >
>> > > > > --
>> > > > > Thanks,
>> > > > >
>> > > > > Michael Paquier
>> > > > > http://michael.otacoo.com
>> > > > >
>> > > > >
>> > > > >
>> > >
>> > > ------------------------------------------------------------------------------
>> > > > > Create and publish websites with WebMatrix
>> > > > > Use the most popular FREE web apps or write code yourself;
>> > > > > WebMatrix provides all the features you need to develop and
>> > > > > publish your website. http://p.sf.net/sfu/ms-webmatrix-sf
>> > > > >
>> > > > > _______________________________________________
>> > > > > Postgres-xc-developers mailing list
>> > > > > Pos...@li...
>> > > > >
>> > > > > https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > > > --
>> > > > Best Wishes,
>> > > > Ashutosh Bapat
>> > >
>> >
>> >
>> >
>> > --
>> > Best Wishes,
>> > Ashutosh Bapat
>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
>
>
> ------------------------------------------------------------------------------
> Xperia(TM) PLAY
> It's a major breakthrough. An authentic gaming
> smartphone on the nation's most reliable network.
> And it wants your games.
> http://p.sf.net/sfu/verizon-sfdev
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>

On Tue, Apr 5, 2011 at 1:30 PM, Koichi Suzuki <ko...@in...>wrote:

> Please let me know your sourceforge account name.   You need access
> pribilege to the resource.
>
>
It's ashutoshbapat.


> Regards;
> ---
> Koichi
>
>
> On Tue, 5 Apr 2011 11:00:50 +0530
> Ashutosh Bapat <ash...@en...> wrote:
>
> > On Mon, Apr 4, 2011 at 6:56 AM, Koichi Suzuki <ko...@in...
> >wrote:
> >
> > > Hi, Ashutosh;
> > >
> > > Welcome to the project!  It's so nice that you are already familiar
> with
> > > the internal.
> > >
> > >
> > Thanks Suzuki-san. Looking forward to contribute to this wonderful
> project.
> >
> >
> > > Best Regards;
> > > ---
> > > Koichi Suzuki
> > >
> > > On Fri, 1 Apr 2011 17:10:45 +0530
> > > Ashutosh Bapat <ash...@en...> wrote:
> > >
> > > > Hi Michael,
> > > > You want to make sure that the user which makes connection to data
> nodes
> > > has
> > > > privileges to execute set role on data node. I think, not everyone is
> > > > allowed to execute 'set role' using any user name.
> > > >
> > > > On Fri, Apr 1, 2011 at 4:59 PM, Michael Paquier
> > > > <mic...@gm...>wrote:
> > > >
> > > > > Just another point. XC is using session user to build pooler
> > > connections.
> > > > > When you change the current user with set role, the same session
> > > > > connections are being used, but access to data or tables is managed
> by
> > > > > Coordinators with current user value.
> > > > >
> > > > > --
> > > > > Thanks,
> > > > >
> > > > > Michael Paquier
> > > > > http://michael.otacoo.com
> > > > >
> > > > >
> > > > >
> > >
> ------------------------------------------------------------------------------
> > > > > Create and publish websites with WebMatrix
> > > > > Use the most popular FREE web apps or write code yourself;
> > > > > WebMatrix provides all the features you need to develop and
> > > > > publish your website. http://p.sf.net/sfu/ms-webmatrix-sf
> > > > >
> > > > > _______________________________________________
> > > > > Postgres-xc-developers mailing list
> > > > > Pos...@li...
> > > > >
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Best Wishes,
> > > > Ashutosh Bapat
> > >
> >
> >
> >
> > --
> > Best Wishes,
> > Ashutosh Bapat
>



-- 
Best Wishes,
Ashutosh Bapat