postgres-xc-developers Mailing List for Postgres-XC (Page 130)

Hi,

> XCM module is added to Postgres-XC (ha_support branch so far). I
> added the following file in sourceforge development web-site.
>
> XCM_Module_Document_20110221.pdf

I finished reading this document now.
I think that XCM is a great idea.

> Misc is created to store temporary materials intended to be a part of
> further releases.

I'd like to know details more.
(1)How to replicate the data among gtm and gtm-standby(s)?

(2)What failure do you assume?  Only crash failure? or more?
And what's kind of failure detector does xcwatcher have?
Theoretically, is it eventually strong?  eventually perfect?

(3)As a possibility, I think that  Postgres-XC components divide into
some parts due to the network failure etc. Is it correct?


----
I cannot go to the conference on Friday.
Please teach if there is time.

Regards,

On Mon, 2011-02-21 at 17:07 +0900, Koichi Suzuki wrote:
> Does postgresql.org help to create mailing list for such very
> specific 
> project? 

Yeah. As I said, we moved psycopg2 lists to there 3-4 weeks before. If
noone objects, I can ask website team for assistance.

Regards,
-- 
Devrim GÜNDÜZ
EnterpriseDB: http://www.enterprisedb.com
PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer
Community: devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
http://www.gunduz.org  Twitter: http://twitter.com/devrimgunduz

Hi,

Nice to hear from you;

Does postgresql.org help to create mailing list for such very specific 
project?

(2011年02月21日 17:02), Devrim GÜNDÜZ wrote:
> On Mon, 2011-02-21 at 09:52 +0900, Michael Paquier wrote:
>>
>> I found another mailing list system.
>> It looks that they do not intrusively insert advert messages at the
>> end of
>> each email.
>> http://lists.freebsd.org/mailman/create
>>
>> So, why not moving to this service?
>
> Why don't we move to postgresql.org ? We recently moved psycopg2 mailing
> lists to there. It would be better than using an external service.
>
> Regards,
>
>
>
> ------------------------------------------------------------------------------
> The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
> Pinpoint memory and threading errors before they happen.
> Find and fix more than 250 security defects in the development cycle.
> Locate bottlenecks in serial and parallel code that limit performance.
> http://p.sf.net/sfu/intel-dev2devfeb
>
>
>
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers

On Mon, 2011-02-21 at 11:11 +0900, Koichi Suzuki wrote:
> Should we consider to have postgres-xc.org? 

You know I have it :)
-- 
Devrim GÜNDÜZ
EnterpriseDB: http://www.enterprisedb.com
PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer
Community: devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
http://www.gunduz.org  Twitter: http://twitter.com/devrimgunduz

On Mon, 2011-02-21 at 09:52 +0900, Michael Paquier wrote:
> 
> I found another mailing list system.
> It looks that they do not intrusively insert advert messages at the
> end of
> each email.
> http://lists.freebsd.org/mailman/create
> 
> So, why not moving to this service? 

Why don't we move to postgresql.org ? We recently moved psycopg2 mailing
lists to there. It would be better than using an external service.

Regards,
-- 
Devrim GÜNDÜZ
EnterpriseDB: http://www.enterprisedb.com
PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer
Community: devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
http://www.gunduz.org  Twitter: http://twitter.com/devrimgunduz

Hi,

XCM module is added to Postgres-XC (ha_support branch so far).   I
added the following file in sourceforge development web-site.

XCM_Module_Document_20110221.pdf

You can download this from the page
https://sourceforge.net/projects/postgres-xc/files/misc/

Misc is created to store temporary materials intended to be a part of
further releases.

Good luck.
----------
Koichi Suzuki

Hi, Mason;
----------
Koichi Suzuki



2011/2/18 Mason <ma...@us...>:
> ---------- Forwarded message ----------
> From: Michael Paquier <mic...@us...>
> Date: Thu, Feb 17, 2011 at 11:06 PM
> Subject: [Postgres-xc-committers] Postgres-XC branch, ha_support,
> updated. v0.9.3-55-gd73ae51
> To: pos...@li...
>
>
> Project "Postgres-XC".
>
> The branch, ha_support has been updated
>       via  d73ae5182149b08e0728edb96eee339e0c0498b7 (commit)
>      from  f42b489b49f366c78d816708d47b380f9db640d9 (commit)
>
>
> - Log -----------------------------------------------------------------
> commit d73ae5182149b08e0728edb96eee339e0c0498b7
> Author: Michael P <mic...@us...>
> Date:   Fri Feb 18 15:49:39 2011 +0900
>
>    Mirroring and XCM (XC Cluster Manager) implementation
>
>
> Hi Michael,
>
> I will try and take a closer look when I have time.
>
> Just wondering, have you guys done any measurements to show the
> performance impact on DBT-1?

Not yet.   Because we have to double-write in the case of updating
transaction, and because of involved 2PC, this will have some impact
to the performance.    Mirroring is for the applications which needs
very high availablity and no transaction loss is allowed even if
datanode fails.

People may think why we do this first and why we don't use streaming
replication.    We were waiting for synchronous mode streaming
replication, which means that MASTER guarantees all the WAL records
are shipped to the SLAVE and no committed transaction will be lost.
I think this is important to maintain database integrity in the whole
cluster.  Unfortunately, I'm afraid it might not be in PG 9.1.
Anyway, I'd like to include streaming replication someday in this year
so that we can maintain whole performance with reasonable
availability.

Regards;
---
Koichi

>
> Thanks,
>
> Mason
>
> ------------------------------------------------------------------------------
> The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
> Pinpoint memory and threading errors before they happen.
> Find and fix more than 250 security defects in the development cycle.
> Locate bottlenecks in serial and parallel code that limit performance.
> http://p.sf.net/sfu/intel-dev2devfeb
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>

Are there other candidate hosts where we can move to?   Should we
consider to have postgres-xc.org?
----------
Koichi Suzuki



2011/2/21 Michael Paquier <mic...@gm...>:
> Hi all,
>
> Perhaps you noticed for the last couple of weeks that some nice (irony?!)
> advertisement
> messages are introduced at the end of each message we send on the mailing
> list.
>
> Having some advertisement about for instance Oracle RAC (close source,
> private owner) in the mailing list of an open source project
> dealing about database clustering is perhaps not the best situation for
> users of Postgres-XC.
>
> I found another mailing list system.
> It looks that they do not intrusively insert advert messages at the end of
> each email.
> http://lists.freebsd.org/mailman/create
>
> So, why not moving to this service?
> --
> Michael Paquier
> http://michaelpq.users.sourceforge.net
>
>
> ------------------------------------------------------------------------------
> The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
> Pinpoint memory and threading errors before they happen.
> Find and fix more than 250 security defects in the development cycle.
> Locate bottlenecks in serial and parallel code that limit performance.
> http://p.sf.net/sfu/intel-dev2devfeb
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>

Hi all,

Perhaps you noticed for the last couple of weeks that some nice (irony?!)
advertisement
messages are introduced at the end of each message we send on the mailing
list.

Having some advertisement about for instance Oracle RAC (close source,
private owner) in the mailing list of an open source project
dealing about database clustering is perhaps not the best situation for
users of Postgres-XC.

I found another mailing list system.
It looks that they do not intrusively insert advert messages at the end of
each email.
http://lists.freebsd.org/mailman/create

So, why not moving to this service?
-- 
Michael Paquier
http://michaelpq.users.sourceforge.net

Hi,

On Sun, 2011-02-20 at 21:16 +0900, Michael Paquier wrote:

> Nice to hear from you.
> Yes, we are planning to do the release 0.9.4 by the end of March.
> I know that the website of the project has not been updated
> lately...[?]
> 
> But this release will be done with all the documentation necessary.

Perfect. Thanks for the update. 

Regards,
-- 
Devrim GÜNDÜZ
EnterpriseDB: http://www.enterprisedb.com
PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer
Community: devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
http://www.gunduz.org  Twitter: http://twitter.com/devrimgunduz

Hi,

There have been lots of commits since 0.9.3. Do you think that it is a
good time for 0.9.4?

Regards,
-- 
Devrim GÜNDÜZ
EnterpriseDB: http://www.enterprisedb.com
PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer
Community: devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
http://www.gunduz.org  Twitter: http://twitter.com/devrimgunduz

---------- Forwarded message ----------
From: Michael Paquier <mic...@us...>
Date: Thu, Feb 17, 2011 at 11:06 PM
Subject: [Postgres-xc-committers] Postgres-XC branch, ha_support,
updated. v0.9.3-55-gd73ae51
To: pos...@li...


Project "Postgres-XC".

The branch, ha_support has been updated
      via  d73ae5182149b08e0728edb96eee339e0c0498b7 (commit)
     from  f42b489b49f366c78d816708d47b380f9db640d9 (commit)


- Log -----------------------------------------------------------------
commit d73ae5182149b08e0728edb96eee339e0c0498b7
Author: Michael P <mic...@us...>
Date:   Fri Feb 18 15:49:39 2011 +0900

   Mirroring and XCM (XC Cluster Manager) implementation


Hi Michael,

I will try and take a closer look when I have time.

Just wondering, have you guys done any measurements to show the
performance impact on DBT-1?

Thanks,

Mason

I took a look at the patch.   Basic idea is to use "name" rather than "oid".

Mason, I'd like to have your idea.   Because "OID" appears in so many
places, I think we should review if using "name" works in many cases
in the catalog search, I mean, general statements.    We've not
discussed much on this, especially integration of coordinator and
datanode catalog.   They're a little different, for example,
definition of aggregates.   I think we should consider this as well.

Because Michael is restoring "EXECUTE DIRECT", for the time being, we
may thing to allow this to more general use.   So far, EXECUTE DIRECT
is intended to be used by dedicated housekeeping, such as recovering
outstanding 2PCs when a coordinator crashes.

Any opinion/idea/comment is welcome.

Best Regards;
----------
Koichi Suzuki



2011/2/18 rahua1 <ra...@16...>:
> hi, koichi & mason:
>
> In postgresql regression test, only this case fails due to OID.
>
> Suggestion, given by mason, that use name instead of OID sounds simple &
> doable to fix this bug. So I try to do follow this way.
>
> Attachment is the code patch, any reply will be great appreciated.
>
>
>
>
> Best Regards!
>
> leo messi
>
> 2011-02-18
> ________________________________
> rahua1
> ________________________________
> 发件人: Mason <ma...@us...>
> 发送时间: 2011-02-14 10:47
> 主 题: Re: [Postgres-xc-developers] any good ways to resolve different oid
> between coordinators and datanodes?
> 收件人: koi...@gm...
> 抄 送: rahua1 <ra...@16...>, "pos...@li..."
> <pos...@li...>
>
>
>
> On Sun, Feb 13, 2011 at 4:52 PM, Koichi Suzuki <koi...@gm...> wrote:
>> Good point.
>>
>> Current XC does not enforce consistent OID among catalogs in
>> coordinators/datanodes.
>>
>> We have several candidate to improve it.
>>
>> 1. To enforce consistent OIDs among all the local catalogs.   This
>> will require to have separate OID system from user tables to reduce
>> coordinator <--> GTM communication for CREATE ... statement.   When a
>> coordinator proxy DDL statement to other coordinators/datanodes,
>> CREATE statement should be associated with such OID.
>
> Something like CREATE TABLE tab1 OID 1001 (col1 OID 1002  int, col2
> OID 1003...)? Basically, make sure every single item that creates a
> new row somewhere in pg_catalog has an OID explicitly assigned to it.
>
>>
>> 2. To have dedicated housekeeping psql interface such as "EXECUTE
>> DIRECT" which targets to specific coordinator/datanode.
>
> We may have talked about this a long time ago. I think one idea
> (yours?) was to convert every DDL statement to pg_catalog DML and
> execute on the other nodes. Maybe there is a little more work that
> would have to be done to make sure certain caches are invalidated if
> going that route.
>
> Either of these sound doable, but just would take some time.
>
>>
>> 1. will need some amount of code work (need to modify
>> coordinator/datanode DDL handling).   I'd like to know how serious
>> this limitation would be.
>
> Yes, this is difficult. Are there any other cases from pg_regress (or
> other cases encountered) that can be traced to OID issues, to get a
> better idea of the size of the problem? If there are not that many,
> you could consider just handling the special cases, like change how it
> is cached and use the name instead of the OID.  OTOH, perhaps there
> are more cases that come up in the future where it is useful to have
> these consistent.
>
> Regards,
>
> Mason
>
>
>>
>> Regards;
>> ----------
>> Koichi Suzuki
>>
>>
>>
>> 2011/2/12 rahua1 <ra...@16...>:
>>> when fixing bug# 3143333,
>>> BUG DETIALS:
>>>
>>> step:
>>> 1.create table with oids:
>>> create table test (x int, y int)with oids;
>>> 2.insert data;
>>> insert into test values(1,2);
>>> 3.create type:
>>> create type int_type as(a int, b int );
>>> 4.create table:
>>> create table int_test(z int, g int_type);
>>> 5.insert data:
>>> insert into int_test values(3,(4,5));
>>> 6.select statement :
>>> select * from int_test;
>>> 7.executed results:
>>> ERROR:  cache lookup failed for type 16391
>>>
>>>
>>>
>>> when created one data type after inserted one record into table with oids,
>>> the oid of self-defined data type is different between coordinator and
>>> datanode. PG-XC uses data type oid received from datanode to do cache
>>> lookup, so
>>> result descripted as bug#3143333 comes up.
>>> Any good ways to fix this bug?
>>> 2011-02-11
>>> ________________________________
>>> rahua1
>>> ------------------------------------------------------------------------------
>>> The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
>>> Pinpoint memory and threading errors before they happen.
>>> Find and fix more than 250 security defects in the development cycle.
>>> Locate bottlenecks in serial and parallel code that limit performance.
>>> http://p.sf.net/sfu/intel-dev2devfeb
>>> _______________________________________________
>>> Postgres-xc-developers mailing list
>>> Pos...@li...
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>>
>>>
>>
>> ------------------------------------------------------------------------------
>> The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
>> Pinpoint memory and threading errors before they happen.
>> Find and fix more than 250 security defects in the development cycle.
>> Locate bottlenecks in serial and parallel code that limit performance.
>> http://p.sf.net/sfu/intel-dev2devfeb
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>

Hi,

ReceiveCopyBegin and SendCopyBegin are functions used by \copy command
when you try to do a copy from a remote psql terminal or application.
It sends copy start and stop messages from the frontend, your psql terminal
or your application
to the postgres server.

COPY_BUFFER is used for COPY TO when you want to print out the results of a
SELECT query with COPY.
You basically tale the results of the SELECT query taken from the nodes and
store them in a COPY buffer,
then you return the results to the backend of you them print out.

I hope this helps.

Trying to support Binary format for COPY in XC is not an easy task.
For COPY FROM (when you want to insert the data in tables), you may have to
materialize the results on Coordinator,
and target the correct nodes depending on the table targeted.
By materializing data, I mean look at the binary data received from
frontend, transform it into queries that will be sent to Datanode backends.

For COPY TO (when you want to print out result in binary format), I would
think that you have to take the data from datanodes, materialize it on
Coordinator,
modify the data format to binary and then send back to frontend the modified
data.

On Wed, Feb 16, 2011 at 6:57 PM, xabc1000 <xab...@16...> wrote:

>  hello,
>
>       When I analyzed the Bug#3151626(copy binary error), I encounterd a problem.
>
> There was a function 'ReceiveCopyBegin' in the file copy.c, which was almost the same
>
> with the function 'SendCopyBegin'. I don't know why and what it's function. In addition,
>
> there was a condition 'COPY_BUFFER' in the function 'CopyGetData'. What's the use for
> the condition 'COPY_BUFFER'? If someone can help me, I would be very grateful.
>
>
>        Yours
>        xcix
>
> 2011-02-16
> ------------------------------
> xabc1000
>

Regards,
-- 
Michael Paquier
http://michaelpq.users.sourceforge.net

hello,
      When I analyzed the Bug#3151626(copy binary error), I encounterd a problem. 
There was a function 'ReceiveCopyBegin' in the file copy.c, which was almost the same 
with the function 'SendCopyBegin'. I don't know why and what it's function. In addition, 
there was a condition 'COPY_BUFFER' in the function 'CopyGetData'. What's the use for 
the condition 'COPY_BUFFER'? If someone can help me, I would be very grateful. 

       Yours
       xcix

2011-02-16 



xabc1000 

Mason;

Thanks a lot for your input.   We may think when we do these.   It's
straightforward technically but may take some time.

Any volunteers?

Please note that the license will be changed to PostgreSQL's one to
eliminate any difficulty to bring XC code into PG core and to
encourage more people to join.
----------
Koichi Suzuki



2011/2/14 Mason <ma...@us...>:
> On Sun, Feb 13, 2011 at 4:52 PM, Koichi Suzuki <koi...@gm...> wrote:
>> Good point.
>>
>> Current XC does not enforce consistent OID among catalogs in
>> coordinators/datanodes.
>>
>> We have several candidate to improve it.
>>
>> 1. To enforce consistent OIDs among all the local catalogs.   This
>> will require to have separate OID system from user tables to reduce
>> coordinator <--> GTM communication for CREATE ... statement.   When a
>> coordinator proxy DDL statement to other coordinators/datanodes,
>> CREATE statement should be associated with such OID.
>
> Something like CREATE TABLE tab1 OID 1001 (col1 OID 1002  int, col2
> OID 1003...)? Basically, make sure every single item that creates a
> new row somewhere in pg_catalog has an OID explicitly assigned to it.
>
>>
>> 2. To have dedicated housekeeping psql interface such as "EXECUTE
>> DIRECT" which targets to specific coordinator/datanode.
>
> We may have talked about this a long time ago. I think one idea
> (yours?) was to convert every DDL statement to pg_catalog DML and
> execute on the other nodes. Maybe there is a little more work that
> would have to be done to make sure certain caches are invalidated if
> going that route.
>
> Either of these sound doable, but just would take some time.
>
>>
>> 1. will need some amount of code work (need to modify
>> coordinator/datanode DDL handling).   I'd like to know how serious
>> this limitation would be.
>
> Yes, this is difficult. Are there any other cases from pg_regress (or
> other cases encountered) that can be traced to OID issues, to get a
> better idea of the size of the problem? If there are not that many,
> you could consider just handling the special cases, like change how it
> is cached and use the name instead of the OID.  OTOH, perhaps there
> are more cases that come up in the future where it is useful to have
> these consistent.
>
> Regards,
>
> Mason
>
>
>>
>> Regards;
>> ----------
>> Koichi Suzuki
>>
>>
>>
>> 2011/2/12 rahua1 <ra...@16...>:
>>> when fixing bug# 3143333,
>>> BUG DETIALS:
>>>
>>> step:
>>> 1.create table with oids:
>>> create table test (x int, y int)with oids;
>>> 2.insert data;
>>> insert into test values(1,2);
>>> 3.create type:
>>> create type int_type as(a int, b int );
>>> 4.create table:
>>> create table int_test(z int, g int_type);
>>> 5.insert data:
>>> insert into int_test values(3,(4,5));
>>> 6.select statement :
>>> select * from int_test;
>>> 7.executed results:
>>> ERROR:  cache lookup failed for type 16391
>>>
>>>
>>>
>>> when created one data type after inserted one record into table with oids,
>>> the oid of self-defined data type is different between coordinator and
>>> datanode. PG-XC uses data type oid received from datanode to do cache
>>> lookup, so
>>> result descripted as bug#3143333 comes up.
>>> Any good ways to fix this bug?
>>> 2011-02-11
>>> ________________________________
>>> rahua1
>>> ------------------------------------------------------------------------------
>>> The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
>>> Pinpoint memory and threading errors before they happen.
>>> Find and fix more than 250 security defects in the development cycle.
>>> Locate bottlenecks in serial and parallel code that limit performance.
>>> http://p.sf.net/sfu/intel-dev2devfeb
>>> _______________________________________________
>>> Postgres-xc-developers mailing list
>>> Pos...@li...
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>>
>>>
>>
>> ------------------------------------------------------------------------------
>> The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
>> Pinpoint memory and threading errors before they happen.
>> Find and fix more than 250 security defects in the development cycle.
>> Locate bottlenecks in serial and parallel code that limit performance.
>> http://p.sf.net/sfu/intel-dev2devfeb
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>

On Sun, Feb 13, 2011 at 4:52 PM, Koichi Suzuki <koi...@gm...> wrote:
> Good point.
>
> Current XC does not enforce consistent OID among catalogs in
> coordinators/datanodes.
>
> We have several candidate to improve it.
>
> 1. To enforce consistent OIDs among all the local catalogs.   This
> will require to have separate OID system from user tables to reduce
> coordinator <--> GTM communication for CREATE ... statement.   When a
> coordinator proxy DDL statement to other coordinators/datanodes,
> CREATE statement should be associated with such OID.

Something like CREATE TABLE tab1 OID 1001 (col1 OID 1002  int, col2
OID 1003...)? Basically, make sure every single item that creates a
new row somewhere in pg_catalog has an OID explicitly assigned to it.

>
> 2. To have dedicated housekeeping psql interface such as "EXECUTE
> DIRECT" which targets to specific coordinator/datanode.

We may have talked about this a long time ago. I think one idea
(yours?) was to convert every DDL statement to pg_catalog DML and
execute on the other nodes. Maybe there is a little more work that
would have to be done to make sure certain caches are invalidated if
going that route.

Either of these sound doable, but just would take some time.

>
> 1. will need some amount of code work (need to modify
> coordinator/datanode DDL handling).   I'd like to know how serious
> this limitation would be.

Yes, this is difficult. Are there any other cases from pg_regress (or
other cases encountered) that can be traced to OID issues, to get a
better idea of the size of the problem? If there are not that many,
you could consider just handling the special cases, like change how it
is cached and use the name instead of the OID.  OTOH, perhaps there
are more cases that come up in the future where it is useful to have
these consistent.

Regards,

Mason


>
> Regards;
> ----------
> Koichi Suzuki
>
>
>
> 2011/2/12 rahua1 <ra...@16...>:
>> when fixing bug# 3143333,
>> BUG DETIALS:
>>
>> step:
>> 1.create table with oids:
>> create table test (x int, y int)with oids;
>> 2.insert data;
>> insert into test values(1,2);
>> 3.create type:
>> create type int_type as(a int, b int );
>> 4.create table:
>> create table int_test(z int, g int_type);
>> 5.insert data:
>> insert into int_test values(3,(4,5));
>> 6.select statement :
>> select * from int_test;
>> 7.executed results:
>> ERROR:  cache lookup failed for type 16391
>>
>>
>>
>> when created one data type after inserted one record into table with oids,
>> the oid of self-defined data type is different between coordinator and
>> datanode. PG-XC uses data type oid received from datanode to do cache
>> lookup, so
>> result descripted as bug#3143333 comes up.
>> Any good ways to fix this bug?
>> 2011-02-11
>> ________________________________
>> rahua1
>> ------------------------------------------------------------------------------
>> The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
>> Pinpoint memory and threading errors before they happen.
>> Find and fix more than 250 security defects in the development cycle.
>> Locate bottlenecks in serial and parallel code that limit performance.
>> http://p.sf.net/sfu/intel-dev2devfeb
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>>
>
> ------------------------------------------------------------------------------
> The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
> Pinpoint memory and threading errors before they happen.
> Find and fix more than 250 security defects in the development cycle.
> Locate bottlenecks in serial and parallel code that limit performance.
> http://p.sf.net/sfu/intel-dev2devfeb
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>

Good point.

Current XC does not enforce consistent OID among catalogs in
coordinators/datanodes.

We have several candidate to improve it.

1. To enforce consistent OIDs among all the local catalogs.   This
will require to have separate OID system from user tables to reduce
coordinator <--> GTM communication for CREATE ... statement.   When a
coordinator proxy DDL statement to other coordinators/datanodes,
CREATE statement should be associated with such OID.

2. To have dedicated housekeeping psql interface such as "EXECUTE
DIRECT" which targets to specific coordinator/datanode.

1. will need some amount of code work (need to modify
coordinator/datanode DDL handling).   I'd like to know how serious
this limitation would be.

Regards;
----------
Koichi Suzuki



2011/2/12 rahua1 <ra...@16...>:
> when fixing bug# 3143333,
> BUG DETIALS:
>
> step:
> 1.create table with oids:
> create table test (x int, y int)with oids;
> 2.insert data;
> insert into test values(1,2);
> 3.create type:
> create type int_type as(a int, b int );
> 4.create table:
> create table int_test(z int, g int_type);
> 5.insert data:
> insert into int_test values(3,(4,5));
> 6.select statement :
> select * from int_test;
> 7.executed results:
> ERROR:  cache lookup failed for type 16391
>
>
>
> when created one data type after inserted one record into table with oids,
> the oid of self-defined data type is different between coordinator and
> datanode. PG-XC uses data type oid received from datanode to do cache
> lookup, so
> result descripted as bug#3143333 comes up.
> Any good ways to fix this bug?
> 2011-02-11
> ________________________________
> rahua1
> ------------------------------------------------------------------------------
> The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
> Pinpoint memory and threading errors before they happen.
> Find and fix more than 250 security defects in the development cycle.
> Locate bottlenecks in serial and parallel code that limit performance.
> http://p.sf.net/sfu/intel-dev2devfeb
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>

OK I see.
So for the time being it may be better to let xc-watcher manage that.

Let's imagine that the node information is managed by catalog tables.
And let's also imagine that there is a way to inform in a short period time
all the pooler processes
that a node is down.

Could it be possible to do this management in catalog table instead of a
malloc'ed space?

> Memory context is a malloced memory. The only difference is that its
> managed by postgres. If your requirement is that modifications done by
> one process should be visible to other processes, then you must use
> shared memory (like what Suzuki-san is doing for xc-watcher).

On Thu, Feb 10, 2011 at 7:04 PM, Pavan Deolasee <pav...@gm...>wrote:

> Without reading very carefully what you are doing or want to do, here
> is a simple rule:
>
> A backend memory, irrespective of the context, is always local to the
> process. So any changes made by it will not be visible to other
> backends. If you want changes to be visible across backends, you must
> use shared memory.
>
> The memory context makes sense only for that backend. The transaction
> context is freed at the end of a transaction where as top memory
> context stays forever until the process exits or the memory is freed
> explicitly.
>


-- 
Michael Paquier
http://michaelpq.users.sourceforge.net

when fixing bug# 3143333, 
BUG DETIALS: 
step:
1.create table with oids:
create table test (x int, y int)with oids;
2.insert data;
insert into test values(1,2);
3.create type:
create type int_type as(a int, b int );
4.create table:
create table int_test(z int, g int_type);
5.insert data:
insert into int_test values(3,(4,5));
6.select statement :
select * from int_test;
7.executed results:
ERROR:  cache lookup failed for type 16391

when created one data type after inserted one record into table with oids, 
the oid of self-defined data type is different between coordinator and
datanode. PG-XC uses data type oid received from datanode to do cache lookup, so 
result descripted as bug#3143333 comes up.

Any good ways to fix this bug?

2011-02-11



rahua1

Without reading very carefully what you are doing or want to do, here
is a simple rule:

A backend memory, irrespective of the context, is always local to the
process. So any changes made by it will not be visible to other
backends. If you want changes to be visible across backends, you must
use shared memory.

The memory context makes sense only for that backend. The transaction
context is freed at the end of a transaction where as top memory
context stays forever until the process exits or the memory is freed
explicitly.

Thanks,
Pavan

On Thu, Feb 10, 2011 at 3:12 PM, Michael Paquier
<mic...@gm...> wrote:
> Hi all,
>
> A simple question...
> I have an array that has been palloc'ed by the postmaster in
> TopMemoryContext when node started.
> And I would like to modify the content of this array directly from Pooler
> process
> and to make it visible to other postmaster childs (transaction context).
>
> Pooler memory context has been created from TopMemoryContext.
> I tried to change my array contents from pooler by switching memory context
> in pooler
> from PoolerMemoryContext to TopMemoryContext.
> But the array modified by pooler is still seen by my backends as when
> postmaster initialized it.
>
> I checked with gdb and the memory area modified by pooler is the same as the
> one read by my backends.
> Is it necessary to move the memory context of my backends to
> TopMemoryContect to make the results modified by pooler visible to the
> backends?
>
> This modification is a part of mirroring, where backends notifies to a
> postmaster array that some datanode mirrors are offline.
> Offline reports are saved in TopMemoryContext to make it available to my
> backends.
>
> Thanks in advance,
> --
> Michael Paquier
> http://michaelpq.users.sourceforge.net
>
>
> ------------------------------------------------------------------------------
> The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
> Pinpoint memory and threading errors before they happen.
> Find and fix more than 250 security defects in the development cycle.
> Locate bottlenecks in serial and parallel code that limit performance.
> http://p.sf.net/sfu/intel-dev2devfeb
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>



-- 
Pavan Deolasee
EnterpriseDB     http://www.enterprisedb.com

Hi all,

A simple question...
I have an array that has been palloc'ed by the postmaster in
TopMemoryContext when node started.
And I would like to modify the content of this array directly from Pooler
process
and to make it visible to other postmaster childs (transaction context).

Pooler memory context has been created from TopMemoryContext.
I tried to change my array contents from pooler by switching memory context
in pooler
from PoolerMemoryContext to TopMemoryContext.
But the array modified by pooler is still seen by my backends as when
postmaster initialized it.

I checked with gdb and the memory area modified by pooler is the same as the
one read by my backends.
Is it necessary to move the memory context of my backends to
TopMemoryContect to make the results modified by pooler visible to the
backends?

This modification is a part of mirroring, where backends notifies to a
postmaster array that some datanode mirrors are offline.
Offline reports are saved in TopMemoryContext to make it available to my
backends.

Thanks in advance,
-- 
Michael Paquier
http://michaelpq.users.sourceforge.net

Michael,

2011/2/10 Michael Paquier <mic...@gm...>

> Thanks for your answers.
>
> On Thu, Feb 10, 2011 at 3:42 PM, Andrei Martsinchyk <
> and...@gm...> wrote:
>
>> Michael,
>>
>> The purpose of this numbers is to made hash values distributed more
>> evenly.
>> I think Postgres built-in hash function should be used here. It would
>> allow to support more data types as a distribution key.
>>
> OK understood. I was kind of thinking that we should support soon new types
> of distribution keys.
> The functionalities are kind of limited with current XC if users can just
> use integers.
>
> Even more generic, any function returning integer may be allowed to be a
>> distribution function. We would need to store Oid in the catalog.
>> The target node will be calculated as
>> distribution_function(distribution_key)%NumDataNodes. The InvalidOid here
>> may mean MODULO distribution.
>>
> If I understood well, this will make the distribution functions even going
> deeper with Postgres code, what is not a bad thing.
> Are distributed functions stored procedures?
>

A kind of. Please see pg_proc catalog table and pg_proc.c and pg_proc.h in
the code for details.


> When you say store the Oid, you mean to store the Oid of the distribution
> key type in pgxc_catalog?
>
>
I mean store Oid of the function. When table is being created with hash
distribution it can be looked up by the data type of the distribution key.
We may allow in addition CUSTOM distribution with arbitrary function.


>
> --
> Michael Paquier
> http://michaelpq.users.sourceforge.net
>
>


-- 
Best regards,
Andrei Martsinchyk                   mailto:and...@gm...

Thanks for your answers.

On Thu, Feb 10, 2011 at 3:42 PM, Andrei Martsinchyk <
and...@gm...> wrote:

> Michael,
>
> The purpose of this numbers is to made hash values distributed more evenly.
> I think Postgres built-in hash function should be used here. It would allow
> to support more data types as a distribution key.
>
OK understood. I was kind of thinking that we should support soon new types
of distribution keys.
The functionalities are kind of limited with current XC if users can just
use integers.

Even more generic, any function returning integer may be allowed to be a
> distribution function. We would need to store Oid in the catalog.
> The target node will be calculated as
> distribution_function(distribution_key)%NumDataNodes. The InvalidOid here
> may mean MODULO distribution.
>
If I understood well, this will make the distribution functions even going
deeper with Postgres code, what is not a bad thing.
Are distributed functions stored procedures?
When you say store the Oid, you mean to store the Oid of the distribution
key type in pgxc_catalog?

-- 
Michael Paquier
http://michaelpq.users.sourceforge.net

Michael,

The purpose of this numbers is to made hash values distributed more evenly.
I think Postgres built-in hash function should be used here. It would allow
to support more data types as a distribution key.
Even more generic, any function returning integer may be allowed to be a
distribution function. We would need to store Oid in the catalog.
The target node will be calculated as
distribution_function(distribution_key)%NumDataNodes. The InvalidOid here
may mean MODULO distribution.

2011/2/10 Michael Paquier <mic...@gm...>

> Hi,
>
> I have a question to the Grid-SQL developers (I am not really targeting
> people especially but...).
> In locator.c, PGXC is using a hash range function taken from Grid-SQL.
>
> There are a couple of magic numbers in this function hash_range,
> and I wanted to know if there were any meaning to it.
>
> [code]
>     value = 0x238F13AF *
> length;
>
>
>
>     for (i = 0; i < length;
> i++)
>
>         value = value + ((key[i] << i * 5 % 24) &
> 0x7fffffff);
>
>
>
>     return (1103515243 * value + 12345) % 65537 & HASH_MASK;
> [/code]
>
> Regards,
> --
> Michael Paquier
> http://michaelpq.users.sourceforge.net
>
>
>
> ------------------------------------------------------------------------------
> The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
> Pinpoint memory and threading errors before they happen.
> Find and fix more than 250 security defects in the development cycle.
> Locate bottlenecks in serial and parallel code that limit performance.
> http://p.sf.net/sfu/intel-dev2devfeb
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>


-- 
Best regards,
Andrei Martsinchyk                   mailto:and...@gm...