postgres-xc-general Mailing List for Postgres-XC

Hi all ...

After running some some preliminary testing on our postgres-xc dev 
sandbox, I've come across some major concerns with High Availability and 
Failover.  I am hoping someone will be able to provide some insight on 
how to set up seamless high-available failover in a postgres-xc environment.


Here are the details:
--------------------

In the postgres-xc dev sandbox, a test table with the default 
round-robin distribution was created to test simultaneous DML SQL 
running on both db VMs.

With the DML SQL running, the coordinator and datanode on one of the db 
VMs was brought down (NB the host was not shut down, just the 
postgres-xc processes on the VM).  After a very short time (lets say max 
of 2 minutes), the DML SQL script running on the current active db VM 
started failing with the following error :
         ERROR:  Failed to get pooled connections

Our application requires 0 single points of failure.  If one of the db 
VMs goes down, the application needs the ability to automatically detect 
this outage and use the remaining active db VMs in the postgres-xc cluster.

Basically, I need to know how to set up the postgres-xc cluster to 
enable the applications to continue to run DML SQL when one of the db 
VMs goes down.  If this is not supported, then what would need to be put 
in place to allow for seamless high-available failover for a postgres-xc 
cluster.

Thanks in advance,
Phil.

Thanks Phil.

On Wed, Apr 25, 2012 at 3:43 AM, Phil Somers <phi...@ne...> wrote:
> Ashutosh,
>
> We have a standalone postgreSQL db host (v9.0.6) we are currently using for
> QA functional testing.  I used the databases on this host as the source
> db's.  I used pg_dump to dump each QA database to their respective SQL
> files.  I then copied these SQL files over to db_VM1 of the postgres-xc db
> host cluster.  I then used psql to load the SQL files into the test db's I
> had already created on db_VM1.  I then logged onto db_VM2 to verify the
> pg_dump SQLs had been applied to this db node and everything looked good.
>
> However, I subsequently noticed that all the unique constraints on the QA db
> tables did not get applied to the postgres-xc db tables.  I am currently
> looking into fixing this issue.  This appears to be the only issue
> encountered so far since fixing the connection pooling issue.
>
> Thanks,
> Phil.
>
>
> On 4/23/12 3:32 AM, Ashutosh Bapat wrote:
>
>
>>>
>>> I have subsequently created 3 test db's, numerous login roles, numerous
>>> group roles and loaded pg_dump backups into each test db on one of the db
>>> VMs.  The second db VM was then accessed and all db objects that were
>>> created on the remote db VM had successfully been created on the db VM I was
>>> accessing.  As well, all the data on the remote db VM had successfully been
>>> loaded into the db VM I was accessing.
>
>
> Can you please enlist the steps you took to load the databases? From where
> did you dump the data? in which format? How did you load it (depends upon
> format). I think, we can use this as a example case, for those who want to
> move from PostgreSQL to Postgres-XC.
>
>>>
>>>
>>> Postgres-XC is working great so far!!
>>
>> Thanks for your feedback. It is appreciated too.
>> Don't forget that we released 1.0beta1, we have more features in so you
>> might try it.
>> Regards,
>> --
>> Michael Paquier
>> http://michael.otacoo.com
>>
>>
>> ------------------------------------------------------------------------------
>> For Developers, A Lot Can Happen In A Second.
>> Boundary is the first to Know...and Tell You.
>> Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
>> http://p.sf.net/sfu/Boundary-d2dvs2
>> _______________________________________________
>> Postgres-xc-general mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>
>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Enterprise Postgres Company
>



-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

Ashutosh,

We have a standalone postgreSQL db host (v9.0.6) we are currently using 
for QA functional testing.  I used the databases on this host as the 
source db's.  I used pg_dump to dump each QA database to their 
respective SQL files.  I then copied these SQL files over to db_VM1 of 
the postgres-xc db host cluster.  I then used psql to load the SQL files 
into the test db's I had already created on db_VM1.  I then logged onto 
db_VM2 to verify the pg_dump SQLs had been applied to this db node and 
everything looked good.

However, I subsequently noticed that all the unique constraints on the 
QA db tables did not get applied to the postgres-xc db tables.  I am 
currently looking into fixing this issue.  This appears to be the only 
issue encountered so far since fixing the connection pooling issue.

Thanks,
Phil.

On 4/23/12 3:32 AM, Ashutosh Bapat wrote:
>
>
>         I have subsequently created 3 test db's, numerous login roles,
>         numerous group roles and loaded pg_dump backups into each test
>         db on one of the db VMs.  The second db VM was then accessed
>         and all db objects that were created on the remote db VM had
>         successfully been created on the db VM I was accessing.  As
>         well, all the data on the remote db VM had successfully been
>         loaded into the db VM I was accessing.
>
>
> Can you please enlist the steps you took to load the databases? From 
> where did you dump the data? in which format? How did you load it 
> (depends upon format). I think, we can use this as a example case, for 
> those who want to move from PostgreSQL to Postgres-XC.
>
>
>         Postgres-XC is working great so far!!
>
>     Thanks for your feedback. It is appreciated too.
>     Don't forget that we released 1.0beta1, we have more features in
>     so you might try it.
>     Regards,
>     -- 
>     Michael Paquier
>     http://michael.otacoo.com
>
>     ------------------------------------------------------------------------------
>     For Developers, A Lot Can Happen In A Second.
>     Boundary is the first to Know...and Tell You.
>     Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
>     http://p.sf.net/sfu/Boundary-d2dvs2
>     _______________________________________________
>     Postgres-xc-general mailing list
>     Pos...@li...
>     <mailto:Pos...@li...>
>     https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
>
>
>
> -- 
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Enterprise Postgres Company
>

>
>> I have subsequently created 3 test db's, numerous login roles, numerous
>> group roles and loaded pg_dump backups into each test db on one of the db
>> VMs.  The second db VM was then accessed and all db objects that were
>> created on the remote db VM had successfully been created on the db VM I
>> was accessing.  As well, all the data on the remote db VM had successfully
>> been loaded into the db VM I was accessing.
>>
>
Can you please enlist the steps you took to load the databases? From where
did you dump the data? in which format? How did you load it (depends upon
format). I think, we can use this as a example case, for those who want to
move from PostgreSQL to Postgres-XC.


>
>> Postgres-XC is working great so far!!
>>
> Thanks for your feedback. It is appreciated too.
> Don't forget that we released 1.0beta1, we have more features in so you
> might try it.
> Regards,
> --
> Michael Paquier
> http://michael.otacoo.com
>
>
> ------------------------------------------------------------------------------
> For Developers, A Lot Can Happen In A Second.
> Boundary is the first to Know...and Tell You.
> Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
> http://p.sf.net/sfu/Boundary-d2dvs2
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
>


-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

On Sat, Apr 21, 2012 at 3:19 AM, Phil Somers <phi...@ne...> wrote:

>  I turned up the logging on both coord/datanode VMs as per your suggestion
> but nothing of any significance was noted.
>
> In attempts to troubleshoot the issue, I then updated the node
> initialization removing the remote nodes on each VM.  Once I did this, the
> CREATE DATABASE sql would run clean but only get created on the local node,
> which made sense, seeing only the local datanode was configured in the
> pgxc_node catalog.
>
> Seeing it was probably network related, one of my SA co-workers took a
> thorough look at the network configs on each VM and found that one of the
> VMs had an additional NIC configured called virbr0.  This apparently
> occured when VMTools was installed on this host (another SA was testing out
> the VMTool suite a while back and used this VM to run his tests).
>
> I disabled and removed the virbr0 NIC.  I then had to edit the pg_hba.conf
> hosts on coord1, data1, coord2 and data2 as follows:
>
>     host all all 192.168.38.0/24 trust
>
> I had originally added it as 'host all all 192.168.38.0*/32* trust' but
> the /32 was causing issues where postgres-xc couldn't find the hosts
> properly.
>
OK. Good to know you found the origin of the problem.

Once I did all this and updated the node initialization to include the
> remote hosts on each VM, the CREATE DATABASE SQL ran clean.
>
> I have subsequently created 3 test db's, numerous login roles, numerous
> group roles and loaded pg_dump backups into each test db on one of the db
> VMs.  The second db VM was then accessed and all db objects that were
> created on the remote db VM had successfully been created on the db VM I
> was accessing.  As well, all the data on the remote db VM had successfully
> been loaded into the db VM I was accessing.
>
> Postgres-XC is working great so far!!
>
Thanks for your feedback. It is appreciated too.
Don't forget that we released 1.0beta1, we have more features in so you
might try it.
Regards,
-- 
Michael Paquier
http://michael.otacoo.com

Michel,

Just a quick note to let you know I resolved the connection pool issue.

It wasn't a permissions or firewall issue as no firewalls are running 
between any of the three VMs.

I turned up the logging on both coord/datanode VMs as per your 
suggestion but nothing of any significance was noted.

In attempts to troubleshoot the issue, I then updated the node 
initialization removing the remote nodes on each VM.  Once I did this, 
the CREATE DATABASE sql would run clean but only get created on the 
local node, which made sense, seeing only the local datanode was 
configured in the pgxc_node catalog.

Seeing it was probably network related, one of my SA co-workers took a 
thorough look at the network configs on each VM and found that one of 
the VMs had an additional NIC configured called virbr0.  This apparently 
occured when VMTools was installed on this host (another SA was testing 
out the VMTool suite a while back and used this VM to run his tests).

I disabled and removed the virbr0 NIC.  I then had to edit the 
pg_hba.conf hosts on coord1, data1, coord2 and data2 as follows:

     host all all 192.168.38.0/24 trust

I had originally added it as 'host all all 192.168.38.0*/32* trust' but 
the /32 was causing issues where postgres-xc couldn't find the hosts 
properly.


Once I did all this and updated the node initialization to include the 
remote hosts on each VM, the CREATE DATABASE SQL ran clean.

I have subsequently created 3 test db's, numerous login roles, numerous 
group roles and loaded pg_dump backups into each test db on one of the 
db VMs.  The second db VM was then accessed and all db objects that were 
created on the remote db VM had successfully been created on the db VM I 
was accessing.  As well, all the data on the remote db VM had 
successfully been loaded into the db VM I was accessing.

Postgres-XC is working great so far!!


Thanks very much for taking the time to respond to and halp with this 
issue.  It was greatly appreciated.

Phil.



On 4/16/12 7:24 PM, Michael Paquier wrote:
>
>
> On Mon, Apr 16, 2012 at 11:34 PM, Phil Somers <phi...@ne... 
> <mailto:phi...@ne...>> wrote:
>
>     Michel,
>
>     Thanks very much for your speedy reply.
>
>     After reading your email, I realized I had forgotten to set the
>     listen_addresses parameter to '*' on the coordinators and
>     datanodes.  I shut down the whole postgres-xc environment, updated
>     the postgresql.conf files on the coordinators and datanodes and
>     fired everything back up without issue, using the -o "-i" option
>     for the coordinator/datanodes startup.
>
>     After doing all this, I am still getting connection issues with
>     datanode 0 when trying to create a database on coordinator1.
>
>     I provided the output of the pgxc_node table as I am thinking
>     maybe there are problems with the entries in this table?
>
>     Any other suggestions you may have with this issue would be
>     greatly appreciated.
>
>     Phil
>
>
>
>     Here are the details:
>
>     -----------------------------------------------------------------------------------------------------------------
>     -----------------------------------------------------------------------------------------------------------------
>     -----------------------------------------------------------------------------------------------------------------
>
>
>     shutdown gtm, coord1/data1, coord2/data2
>     ----------------------------------------
>
>     updated postgresql.conf on dbhost1 for both coord1 and data1:
>     ------------------------------------------------------------
>     - listen_addresses = '*'
>
>     updated postgresql.conf on dbhost2 for both coord2 and data2:
>     ------------------------------------------------------------
>     - listen_addresses = '*'
>
>
>     started coordinator and datanodes on each db host adding the -o
>     "-i" as per your recommendation:
>     -----------------------------------------------------------------------------------------------
>     dbhost1:
>         pg_ctl start -D /home/postgres/datanode/data -l
>     /home/postgres/datanode/log/datanode1.log -o "-i -p 15432" -Z datanode
>         pg_ctl start -D /home/postgres/coordinator/data -l
>     /home/postgres/coordinator/log/coordinator1.log -o "-i" -Z coordinator
>
>     dbhost2:
>         pg_ctl start -D /home/postgres/datanode/data -l
>     /home/postgres/datanode/log/datanode2.log -o "-i -p 15432" -Z datanode
>         pg_ctl start -D /home/postgres/coordinator/data -l
>     /home/postgres/coordinator/log/coordinator2.log -o "-i" -Z coordinator
>
>
>     checked to make sure postgres was running on dbhost1:
>     -----------------------------------------------------
>     $ ps -ef | grep postgres
>
>     root      1005   587  0 10:22 ?        00:00:00 sshd: postgres [priv]
>     1058      1083  1005  0 10:22 ?        00:00:01 sshd: postgres@pts/0
>     1058      1222     1  0 10:47 pts/0    00:00:00
>     /home/postgres/db_home/bin/postgres -X -D
>     /home/postgres/datanode/data -i -p 15432
>     1058      1224  1222  0 10:47 ?        00:00:00 postgres: writer
>     process
>     1058      1225  1222  0 10:47 ?        00:00:00 postgres: wal
>     writer process
>     1058      1226  1222  0 10:47 ?        00:00:00 postgres:
>     autovacuum launcher process
>     1058      1227  1222  0 10:47 ?        00:00:00 postgres: stats
>     collector process
>     1058      1234     1  0 10:48 pts/0    00:00:00
>     /home/postgres/db_home/bin/postgres -C -D
>     /home/postgres/coordinator/data -i
>     1058      1236  1234  0 10:48 ?        00:00:00 postgres: pooler
>     process
>     1058      1237  1234  0 10:48 ?        00:00:00 postgres: writer
>     process
>     1058      1238  1234  0 10:48 ?        00:00:00 postgres: wal
>     writer process
>     1058      1239  1234  0 10:48 ?        00:00:00 postgres:
>     autovacuum launcher process
>     1058      1240  1234  0 10:48 ?        00:00:00 postgres: stats
>     collector process
>     1058      1401  1084  0 10:53 pts/0    00:00:00 grep --color=auto
>     postgres
>
>
>     psql onto dbhost1 coord1 and ran 'create database TEST';
>     ------------------------------------------------------
>
>
>     psql -U postgres -d postgres
>
>     Password for user postgresql:
>     psql (9.1.2)
>     Type "help" for help.
>
>     postgres=# create database TEST;
>
>     ERROR:  Failed to get pooled connections
>
>
>
>     coord1 log output:
>     -----------------
>
>     LOG:  database system was shut down at 2012-04-16 11:00:24 ADT
>     LOG:  database system is ready to accept connections
>     LOG:  autovacuum launcher started
>
>     LOG:  failed to connect to data node
>     WARNING:  can not connect to datanode 0
>     LOG:  failed to acquire connections
>     STATEMENT:  create database TEST;
>     ERROR:  Failed to get pooled connections
>     STATEMENT:  create database TEST;
>
>
>
>     psql output for pgxc_node (output is exactly same on both coord1
>     and coord2):
>     ----------------------------------------------------------------------------
>
>     postgres=# select * from pgxc_node;
>      node_name | node_type | node_port |   node_host    |
>     nodeis_primary | nodeis_preferred
>     -----------+-----------+-----------+----------------+----------------+------------------
>      coord2    | C         |      5432 | 192.168.38.100 |
>     f              | f
>      data2     | D         |     15432 | 192.168.38.100 |
>     f              | f
>      coord1    | C         |      5432 | 192.168.38.101 |
>     f              | f
>      data1     | D         |     15432 | 192.168.38.101 |
>     f              | f
>     (4 rows)
>
> Just by looking at that, I can't really get the issue you have, but I 
> am pretty sure it is a permission problem.
> Could it be a firewall issue? You may try to shut it down on node 2 
> once and see what happens... Are you able to create a database correctly?
> If this doesn't work, you should turn on log_connections on 
> postgresql.conf of each node and have a look at the logs. It will for 
> sure help to spot your problem.
> -- 
> Michael Paquier
> http://michael.otacoo.com

I don't really know what you are looking for, but based on your example,
here simplified:
create table org (id int primary key, name varchar(50) not null);
create table poc (id int primary key, firstname varchar(50) not null,
lastname varchar(50) not null);
create table link (org_id int, poc_id int, function varchar(2), primary key
(org_id,poc_id));

Depending on the distribution type of each table, the key to performance is
to get queries that are completely shipped to Datanodes (remote nodes)
without having to materialize data from multiple Datanodes on Coordinators
(where application connects).

Your query is this one, I just changed the primary keys to integers:
select poc.firstname, poc.lastname, org.id, link.function from poc,org,
link where org.name = 'whatever' and org.id = link.org_id and link.poc_id =
poc.id;

For example, if I create all the tables below as replicated:
create table org (id int primary key, name varchar(50) not null) distribute
by replication;
create table poc (id int primary key, firstname varchar(50) not null,
lastname varchar(50) not null) distribute by replication;
create table link (org_id int, poc_id int, function varchar(2), primary key
(org_id,poc_id)) distribute by replication;
Your SELECT query becomes like this:
postgres=# explain select poc.firstname, poc.lastname, org.id,
link.function from poc,org, link where org.name = 'whatever' and org.id =
link.org_id and link.poc_id = poc.id;
                                 QUERY PLAN
----------------------------------------------------------------------------
 Data Node Scan on "__REMOTE_FQS_QUERY__"  (cost=0.00..0.00 rows=0 width=0)
   Node/s: dn1
(2 rows)
This means that the query is completely shipped to a single node, normal as
the data of all the tables are replicated on all the nodes.

If your tables become distributed:
create table org (id int primary key, name varchar(50) not null) distribute
by hash(id);
create table poc (id int primary key, firstname varchar(50) not null,
lastname varchar(50) not null) distribute by hash(id);
create table link (org_id int, poc_id int, function varchar(2), primary key
(org_id,poc_id)) distribute by hash(org_id);
postgres=# explain select poc.firstname, poc.lastname, org.id,
link.function from poc,org, link where org.name = 'whatever' and org.id =
link.org_id and link.poc_id = poc.id;
                                QUERY PLAN
--------------------------------------------------------------------------
 Nested Loop  (cost=0.00..0.03 rows=1 width=252)
   Join Filter: (link.org_id = org.id)
   ->  Nested Loop  (cost=0.00..0.01 rows=1 width=252)
         Join Filter: (poc.id = link.poc_id)
         ->  Data Node Scan on poc  (cost=0.00..0.00 rows=1000 width=240)
               Node/s: dn1, dn2
         ->  Data Node Scan on link  (cost=0.00..0.00 rows=1000 width=20)
               Node/s: dn1, dn2
   ->  Data Node Scan on org  (cost=0.00..0.00 rows=1000 width=4)
         Node/s: dn1, dn2
(10 rows)
You need to use a nested loop, meaning that you need first to gather the
data from remote nodes, materialize it on Coordinator, then return the
results to client. This is extremely bad for performance.
So your goal to performance is to find the correct combination of table
distribution to have a maximum number of queries being shipped to remote
nodes.

I do not say that it is necessary to replicate all the tables. By doing so,
of course your application will be good in read but write operations will
have to run on all the nodes where table data is located, lowering
performance if your application does a lot of read, so the secret is to
have a good balance.

In XC, you can also located data only on a portion of nodes. This can also
help tune your applications.
http://postgres-xc.sourceforge.net/docs/1_0/sql-createtable.html
CREATE TABLE uses an extension called TO GROUP/NODE to locate data of a
table only on a portion of nodes.
Regards,

On Thu, Apr 19, 2012 at 9:30 PM, Michael Vitale <mic...@ar...> wrote:

>  Thank you for your response.  Can I just give a simple schema example
> and query resulting from it and see if it would suffer in a cluster
> solution using the primary keys, which are system generated names (GUIDs)?
>
> Table ORG (ORG_HANDLE VARCHAR(50) NOT NULL, ORG_NAME VARCHAR2(150) NOT
> NULL);
> primary key: ORG_HANDLE
>
> TABLE POC (POC_HANDLE VARCHAR(50), FIRST_NAME(VARCHAR50) NOT NULL,
> LAST_NAME VARCHAR(50) NOT NULL);
> primary key: POC_HANDLE
>
> Table ORG_POC_LINK (ORG_HANDLE VARCHAR(50) NOT NULL, POC_FUNCTION
> VARCHAR(2) NOT NULL, POC_HANDLE VARCHAR(50) NOT NULL);
> primary key: ORG_HANDLE,POC_FUNCTION,POC_HANDLE
>
>
> Query:
> select POC.FIRST_NAME, POC.LAST_NAME, ORG.HANDLE, OPL.POC_FUNCTION FROM
> POC POC, ORG ORG, ORG_POC_LINK OPL
> WHERE ORG.ORG_NAME = 'whatever' and ORG.ORG_HANDLE = OPL.ORG_HANDLE and
> OPL.POC_HANDLE = POC.POC_HANDLE
>  ------------------------------
> *From:* Ashutosh Bapat [ash...@en...]
> *Sent:* Thursday, April 19, 2012 8:05 AM
> *To:* Michael Vitale
> *Cc:* pos...@li...;
> pos...@li...
> *Subject:* Re: [Postgres-xc-general] the cluster cost for normalized
> tables
>
>  HI Michael,
> The distribution of data depends upon the distribution strategy used. In
> Postgres-XC, we distribute data based on the hash/modulo of the given
> column. It's usually advisable to choose the same distribution for the
> tables which have equi-joins on their distribution columns.
>
> Choosing the right distribution for the tables involved is an art. We need
> the knowledge of table definitions and set of queries to decide the exact
> distribution. If the queries are such that they join on collocated data,
> the performance is greatly improved.
>
> On Thu, Apr 19, 2012 at 4:56 PM, Michael Vitale <mic...@ar...> wrote:
>
>>  Hi you most honorable cluster folks!
>>
>> Our company is moving from Oracle to PostgreSQL.  We initially thought we
>> would be moving to MySQL Cluster, but an investigation of how clustering
>> works in MySQL Cluster revealed that performance would suffer substantially
>> since it is predicated on keys that segregate SQL-requested data to
>> specific nodes and not to all or most of the nodes.  A highly normalized
>> database would suffer in this situation where a result set would normally
>> consist of rows gathered from most, if not all, of the back-end nodes.
>>
>> Do you all have the same problem with Clustered PostgreSQL (Postgres-XC)?
>>
>> Respectfully Yours,
>>
>> Michael Vitale
>> ARIN DBA
>> mic...@ar...
>> 703-227-9885
>>
>>
>>
>> ------------------------------------------------------------------------------
>> For Developers, A Lot Can Happen In A Second.
>> Boundary is the first to Know...and Tell You.
>> Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
>> http://p.sf.net/sfu/Boundary-d2dvs2
>> _______________________________________________
>> Postgres-xc-general mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>
>>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Enterprise Postgres Company
>
>
>
> ------------------------------------------------------------------------------
> For Developers, A Lot Can Happen In A Second.
> Boundary is the first to Know...and Tell You.
> Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
> http://p.sf.net/sfu/Boundary-d2dvs2
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
>


-- 
Michael Paquier
http://michael.otacoo.com

On Thu, Apr 19, 2012 at 9:05 PM, Ashutosh Bapat <
ash...@en...> wrote:

> HI Michael,
> The distribution of data depends upon the distribution strategy used. In
> Postgres-XC, we distribute data based on the hash/modulo of the given
> column. It's usually advisable to choose the same distribution for the
> tables which have equi-joins on their distribution columns.
>
Some addition on that. in XC it is possible to choose the distribution
strategy at table creation by extending the DDL CREATE TABLE. Now XC
supports replication, round robin and hash on a key (useful for a primary
key).
Distribution is tuple-based, meaning that for a given strategy all the
entire tuple is distributed, or replicated.

As a global guideline, we generally recommend to replicate the tables that
are not much updated but used a lot in joins like an item table in an
amazon-type application, and distribute tables that are heavily updates,
let's say a user table.


> Choosing the right distribution for the tables involved is an art. We need
> the knowledge of table definitions and set of queries to decide the exact
> distribution. If the queries are such that they join on collocated data,
> the performance is greatly improved.
>
True. Design phase is the key of performance.


>
> On Thu, Apr 19, 2012 at 4:56 PM, Michael Vitale <mic...@ar...> wrote:
>
>>  Hi you most honorable cluster folks!
>>
>> Our company is moving from Oracle to PostgreSQL.  We initially thought we
>> would be moving to MySQL Cluster, but an investigation of how clustering
>> works in MySQL Cluster revealed that performance would suffer substantially
>> since it is predicated on keys that segregate SQL-requested data to
>> specific nodes and not to all or most of the nodes.  A highly normalized
>> database would suffer in this situation where a result set would normally
>> consist of rows gathered from most, if not all, of the back-end nodes.
>>
>> Do you all have the same problem with Clustered PostgreSQL (Postgres-XC)?
>>
>> Respectfully Yours,
>>
>> Michael Vitale
>> ARIN DBA
>> mic...@ar...
>> 703-227-9885
>>
>>
>>
>> ------------------------------------------------------------------------------
>> For Developers, A Lot Can Happen In A Second.
>> Boundary is the first to Know...and Tell You.
>> Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
>> http://p.sf.net/sfu/Boundary-d2dvs2
>> _______________________________________________
>> Postgres-xc-general mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>
>>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Enterprise Postgres Company
>
>
>
> ------------------------------------------------------------------------------
> For Developers, A Lot Can Happen In A Second.
> Boundary is the first to Know...and Tell You.
> Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
> http://p.sf.net/sfu/Boundary-d2dvs2
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
>


-- 
Michael Paquier
http://michael.otacoo.com

Thank you for your response.  Can I just give a simple schema example and query resulting from it and see if it would suffer in a cluster solution using the primary keys, which are system generated names (GUIDs)?

Table ORG (ORG_HANDLE VARCHAR(50) NOT NULL, ORG_NAME VARCHAR2(150) NOT NULL);
primary key: ORG_HANDLE

TABLE POC (POC_HANDLE VARCHAR(50), FIRST_NAME(VARCHAR50) NOT NULL, LAST_NAME VARCHAR(50) NOT NULL);
primary key: POC_HANDLE

Table ORG_POC_LINK (ORG_HANDLE VARCHAR(50) NOT NULL, POC_FUNCTION VARCHAR(2) NOT NULL, POC_HANDLE VARCHAR(50) NOT NULL);
primary key: ORG_HANDLE,POC_FUNCTION,POC_HANDLE


Query:
select POC.FIRST_NAME, POC.LAST_NAME, ORG.HANDLE, OPL.POC_FUNCTION FROM POC POC, ORG ORG, ORG_POC_LINK OPL
WHERE ORG.ORG_NAME = 'whatever' and ORG.ORG_HANDLE = OPL.ORG_HANDLE and OPL.POC_HANDLE = POC.POC_HANDLE
________________________________
From: Ashutosh Bapat [ash...@en...]
Sent: Thursday, April 19, 2012 8:05 AM
To: Michael Vitale
Cc: pos...@li...; pos...@li...
Subject: Re: [Postgres-xc-general] the cluster cost for normalized tables

HI Michael,
The distribution of data depends upon the distribution strategy used. In Postgres-XC, we distribute data based on the hash/modulo of the given column. It's usually advisable to choose the same distribution for the tables which have equi-joins on their distribution columns.

Choosing the right distribution for the tables involved is an art. We need the knowledge of table definitions and set of queries to decide the exact distribution. If the queries are such that they join on collocated data, the performance is greatly improved.

On Thu, Apr 19, 2012 at 4:56 PM, Michael Vitale <mic...@ar...<mailto:mic...@ar...>> wrote:
Hi you most honorable cluster folks!

Our company is moving from Oracle to PostgreSQL.  We initially thought we would be moving to MySQL Cluster, but an investigation of how clustering works in MySQL Cluster revealed that performance would suffer substantially since it is predicated on keys that segregate SQL-requested data to specific nodes and not to all or most of the nodes.  A highly normalized database would suffer in this situation where a result set would normally consist of rows gathered from most, if not all, of the back-end nodes.

Do you all have the same problem with Clustered PostgreSQL (Postgres-XC)?

Respectfully Yours,

Michael Vitale
ARIN DBA
mic...@ar...<mailto:mic...@ar...>
703-227-9885


------------------------------------------------------------------------------
For Developers, A Lot Can Happen In A Second.
Boundary is the first to Know...and Tell You.
Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
http://p.sf.net/sfu/Boundary-d2dvs2
_______________________________________________
Postgres-xc-general mailing list
Pos...@li...<mailto:Pos...@li...>
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general




--
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

HI Michael,
The distribution of data depends upon the distribution strategy used. In
Postgres-XC, we distribute data based on the hash/modulo of the given
column. It's usually advisable to choose the same distribution for the
tables which have equi-joins on their distribution columns.

Choosing the right distribution for the tables involved is an art. We need
the knowledge of table definitions and set of queries to decide the exact
distribution. If the queries are such that they join on collocated data,
the performance is greatly improved.

On Thu, Apr 19, 2012 at 4:56 PM, Michael Vitale <mic...@ar...> wrote:

>  Hi you most honorable cluster folks!
>
> Our company is moving from Oracle to PostgreSQL.  We initially thought we
> would be moving to MySQL Cluster, but an investigation of how clustering
> works in MySQL Cluster revealed that performance would suffer substantially
> since it is predicated on keys that segregate SQL-requested data to
> specific nodes and not to all or most of the nodes.  A highly normalized
> database would suffer in this situation where a result set would normally
> consist of rows gathered from most, if not all, of the back-end nodes.
>
> Do you all have the same problem with Clustered PostgreSQL (Postgres-XC)?
>
> Respectfully Yours,
>
> Michael Vitale
> ARIN DBA
> mic...@ar...
> 703-227-9885
>
>
>
> ------------------------------------------------------------------------------
> For Developers, A Lot Can Happen In A Second.
> Boundary is the first to Know...and Tell You.
> Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
> http://p.sf.net/sfu/Boundary-d2dvs2
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
>


-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

Hi you most honorable cluster folks!

Our company is moving from Oracle to PostgreSQL.  We initially thought we would be moving to MySQL Cluster, but an investigation of how clustering works in MySQL Cluster revealed that performance would suffer substantially since it is predicated on keys that segregate SQL-requested data to specific nodes and not to all or most of the nodes.  A highly normalized database would suffer in this situation where a result set would normally consist of rows gathered from most, if not all, of the back-end nodes.

Do you all have the same problem with Clustered PostgreSQL (Postgres-XC)?

Respectfully Yours,

Michael Vitale
ARIN DBA
mic...@ar...
703-227-9885

On Mon, Apr 16, 2012 at 11:34 PM, Phil Somers <phi...@ne...>wrote:

>  Michel,
>
> Thanks very much for your speedy reply.
>
> After reading your email, I realized I had forgotten to set the
> listen_addresses parameter to '*' on the coordinators and datanodes.  I
> shut down the whole postgres-xc environment, updated the postgresql.conf
> files on the coordinators and datanodes and fired everything back up
> without issue, using the -o "-i" option for the coordinator/datanodes
> startup.
>
> After doing all this, I am still getting connection issues with datanode 0
> when trying to create a database on coordinator1.
>
> I provided the output of the pgxc_node table as I am thinking maybe there
> are problems with the entries in this table?
>
> Any other suggestions you may have with this issue would be greatly
> appreciated.
>
> Phil
>
>
>
> Here are the details:
>
>
> -----------------------------------------------------------------------------------------------------------------
>
> -----------------------------------------------------------------------------------------------------------------
>
> -----------------------------------------------------------------------------------------------------------------
>
>
> shutdown gtm, coord1/data1, coord2/data2
> ----------------------------------------
>
> updated postgresql.conf on dbhost1 for both coord1 and data1:
> ------------------------------------------------------------
> - listen_addresses = '*'
>
> updated postgresql.conf on dbhost2 for both coord2 and data2:
> ------------------------------------------------------------
> - listen_addresses = '*'
>
>
> started coordinator and datanodes on each db host adding the -o "-i" as
> per your recommendation:
>
> -----------------------------------------------------------------------------------------------
> dbhost1:
>     pg_ctl start -D /home/postgres/datanode/data -l
> /home/postgres/datanode/log/datanode1.log -o "-i -p 15432" -Z datanode
>     pg_ctl start -D /home/postgres/coordinator/data -l
> /home/postgres/coordinator/log/coordinator1.log -o "-i" -Z coordinator
>
> dbhost2:
>     pg_ctl start -D /home/postgres/datanode/data -l
> /home/postgres/datanode/log/datanode2.log -o "-i -p 15432" -Z datanode
>     pg_ctl start -D /home/postgres/coordinator/data -l
> /home/postgres/coordinator/log/coordinator2.log -o "-i" -Z coordinator
>
>
> checked to make sure postgres was running on dbhost1:
> -----------------------------------------------------
> $ ps -ef | grep postgres
>
> root      1005   587  0 10:22 ?        00:00:00 sshd: postgres [priv]
> 1058      1083  1005  0 10:22 ?        00:00:01 sshd: postgres@pts/0
> 1058      1222     1  0 10:47 pts/0    00:00:00
> /home/postgres/db_home/bin/postgres -X -D /home/postgres/datanode/data -i
> -p 15432
> 1058      1224  1222  0 10:47 ?        00:00:00 postgres: writer process
> 1058      1225  1222  0 10:47 ?        00:00:00 postgres: wal writer
> process
> 1058      1226  1222  0 10:47 ?        00:00:00 postgres: autovacuum
> launcher process
> 1058      1227  1222  0 10:47 ?        00:00:00 postgres: stats collector
> process
> 1058      1234     1  0 10:48 pts/0    00:00:00
> /home/postgres/db_home/bin/postgres -C -D /home/postgres/coordinator/data -i
> 1058      1236  1234  0 10:48 ?        00:00:00 postgres: pooler process
> 1058      1237  1234  0 10:48 ?        00:00:00 postgres: writer process
> 1058      1238  1234  0 10:48 ?        00:00:00 postgres: wal writer
> process
> 1058      1239  1234  0 10:48 ?        00:00:00 postgres: autovacuum
> launcher process
> 1058      1240  1234  0 10:48 ?        00:00:00 postgres: stats collector
> process
> 1058      1401  1084  0 10:53 pts/0    00:00:00 grep --color=auto postgres
>
>
> psql onto dbhost1 coord1 and ran 'create database TEST';
> ------------------------------------------------------
>
>
> psql -U postgres -d postgres
>
> Password for user postgresql:
> psql (9.1.2)
> Type "help" for help.
>
> postgres=# create database TEST;
>
> ERROR:  Failed to get pooled connections
>
>
>
> coord1 log output:
> -----------------
>
> LOG:  database system was shut down at 2012-04-16 11:00:24 ADT
> LOG:  database system is ready to accept connections
> LOG:  autovacuum launcher started
>
> LOG:  failed to connect to data node
> WARNING:  can not connect to datanode 0
> LOG:  failed to acquire connections
> STATEMENT:  create database TEST;
> ERROR:  Failed to get pooled connections
> STATEMENT:  create database TEST;
>
>
>
> psql output for pgxc_node (output is exactly same on both coord1 and
> coord2):
>
> ----------------------------------------------------------------------------
>
> postgres=# select * from pgxc_node;
>  node_name | node_type | node_port |   node_host    | nodeis_primary |
> nodeis_preferred
>
> -----------+-----------+-----------+----------------+----------------+------------------
>  coord2    | C         |      5432 | 192.168.38.100 | f              | f
>  data2     | D         |     15432 | 192.168.38.100 | f              | f
>  coord1    | C         |      5432 | 192.168.38.101 | f              | f
>  data1     | D         |     15432 | 192.168.38.101 | f              | f
> (4 rows)
>
Just by looking at that, I can't really get the issue you have, but I am
pretty sure it is a permission problem.
Could it be a firewall issue? You may try to shut it down on node 2 once
and see what happens... Are you able to create a database correctly?
If this doesn't work, you should turn on log_connections on postgresql.conf
of each node and have a look at the logs. It will for sure help to spot
your problem.
-- 
Michael Paquier
http://michael.otacoo.com

Michel,

Thanks very much for your speedy reply.

After reading your email, I realized I had forgotten to set the 
listen_addresses parameter to '*' on the coordinators and datanodes.  I 
shut down the whole postgres-xc environment, updated the postgresql.conf 
files on the coordinators and datanodes and fired everything back up 
without issue, using the -o "-i" option for the coordinator/datanodes 
startup.

After doing all this, I am still getting connection issues with datanode 
0 when trying to create a database on coordinator1.

I provided the output of the pgxc_node table as I am thinking maybe 
there are problems with the entries in this table?

Any other suggestions you may have with this issue would be greatly 
appreciated.

Phil



Here are the details:

-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------


shutdown gtm, coord1/data1, coord2/data2
----------------------------------------

updated postgresql.conf on dbhost1 for both coord1 and data1:
------------------------------------------------------------
- listen_addresses = '*'

updated postgresql.conf on dbhost2 for both coord2 and data2:
------------------------------------------------------------
- listen_addresses = '*'


started coordinator and datanodes on each db host adding the -o "-i" as 
per your recommendation:
-----------------------------------------------------------------------------------------------
dbhost1:
     pg_ctl start -D /home/postgres/datanode/data -l 
/home/postgres/datanode/log/datanode1.log -o "-i -p 15432" -Z datanode
     pg_ctl start -D /home/postgres/coordinator/data -l 
/home/postgres/coordinator/log/coordinator1.log -o "-i" -Z coordinator

dbhost2:
     pg_ctl start -D /home/postgres/datanode/data -l 
/home/postgres/datanode/log/datanode2.log -o "-i -p 15432" -Z datanode
     pg_ctl start -D /home/postgres/coordinator/data -l 
/home/postgres/coordinator/log/coordinator2.log -o "-i" -Z coordinator


checked to make sure postgres was running on dbhost1:
-----------------------------------------------------
$ ps -ef | grep postgres

root      1005   587  0 10:22 ?        00:00:00 sshd: postgres [priv]
1058      1083  1005  0 10:22 ?        00:00:01 sshd: postgres@pts/0
1058      1222     1  0 10:47 pts/0    00:00:00 
/home/postgres/db_home/bin/postgres -X -D /home/postgres/datanode/data 
-i -p 15432
1058      1224  1222  0 10:47 ?        00:00:00 postgres: writer process
1058      1225  1222  0 10:47 ?        00:00:00 postgres: wal writer process
1058      1226  1222  0 10:47 ?        00:00:00 postgres: autovacuum 
launcher process
1058      1227  1222  0 10:47 ?        00:00:00 postgres: stats 
collector process
1058      1234     1  0 10:48 pts/0    00:00:00 
/home/postgres/db_home/bin/postgres -C -D /home/postgres/coordinator/data -i
1058      1236  1234  0 10:48 ?        00:00:00 postgres: pooler process
1058      1237  1234  0 10:48 ?        00:00:00 postgres: writer process
1058      1238  1234  0 10:48 ?        00:00:00 postgres: wal writer process
1058      1239  1234  0 10:48 ?        00:00:00 postgres: autovacuum 
launcher process
1058      1240  1234  0 10:48 ?        00:00:00 postgres: stats 
collector process
1058      1401  1084  0 10:53 pts/0    00:00:00 grep --color=auto postgres


psql onto dbhost1 coord1 and ran 'create database TEST';
------------------------------------------------------

psql -U postgres -d postgres

Password for user postgresql:
psql (9.1.2)
Type "help" for help.

postgres=# create database TEST;
ERROR:  Failed to get pooled connections



coord1 log output:
-----------------

LOG:  database system was shut down at 2012-04-16 11:00:24 ADT
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started
LOG:  failed to connect to data node
WARNING:  can not connect to datanode 0
LOG:  failed to acquire connections
STATEMENT:  create database TEST;
ERROR:  Failed to get pooled connections
STATEMENT:  create database TEST;



psql output for pgxc_node (output is exactly same on both coord1 and 
coord2):
----------------------------------------------------------------------------

postgres=# select * from pgxc_node;
  node_name | node_type | node_port |   node_host    | nodeis_primary | 
nodeis_preferred
-----------+-----------+-----------+----------------+----------------+------------------
  coord2    | C         |      5432 | 192.168.38.100 | f              | f
  data2     | D         |     15432 | 192.168.38.100 | f              | f
  coord1    | C         |      5432 | 192.168.38.101 | f              | f
  data1     | D         |     15432 | 192.168.38.101 | f              | f
(4 rows)




On 4/13/12 10:54 PM, Michael Paquier wrote:
> Hi,
>
> I had a look at your settings. Please see my comments in-line. It 
> looks that your remote nodes are refusing connections as pooler cannot 
> get a connection.
>
>     - started the coordinator on each node without issue using the
>     folowing command:
>
>         pg_ctl start -D /home/postgres/coordinator/data -l
>     /home/postgresql/coordinator/log/coordinator1.log -Z coordinator 
>     ---> log is coordinator2 on 2nd db host
>
> When starting a nodes that are on separate servers, you also need to 
> use option "-i" or the node will only accept local connections.
> This option basically allows remote clients to connect with TCP-IP.
> So you startup option becomes:
> pg_ctl start -D /home/postgres/coordinator/data -l 
> /home/postgresql/coordinator/log/coordinator1.log -Z coordinator -o "-i"
>
>
>     - started the datanode on each node without issue using the
>     folowing command:
>
>         pg_ctl start -D /home/postgres/datanode/data -l
>     /home/postgresql/coordinator/log/datanode1.log -o "-p 15432" -Z
>     datanode  ---> log is datanode2 on 2nd db host
>
> Same here, you need to allow Datanode to accept remote TCP-IP connections.
> Even if you set up correctly pg_hba.conf, this is mandatory.
> pg_ctl start -D /home/postgres/datanode/data -l 
> /home/postgresql/coordinator/log/datanode1.log -o "-i -p 15432" -Z 
> datanode
>
> Please note that this is equivalent to setting listen_addresses to "*" 
> on postgresql.conf of each node.
> Regards,
> -- 
> Michael Paquier
> http://michael.otacoo.com

Hi,

I had a look at your settings. Please see my comments in-line. It looks
that your remote nodes are refusing connections as pooler cannot get a
connection.

- started the coordinator on each node without issue using the folowing
> command:
>
>     pg_ctl start -D /home/postgres/coordinator/**data -l
> /home/postgresql/coordinator/**log/coordinator1.log -Z coordinator  --->
> log is coordinator2 on 2nd db host
>
When starting a nodes that are on separate servers, you also need to use
option "-i" or the node will only accept local connections.
This option basically allows remote clients to connect with TCP-IP.
So you startup option becomes:
pg_ctl start -D /home/postgres/coordinator/**data -l
/home/postgresql/coordinator/**log/coordinator1.log -Z coordinator -o "-i"


> - started the datanode on each node without issue using the folowing
> command:
>
>     pg_ctl start -D /home/postgres/datanode/data -l
> /home/postgresql/coordinator/**log/datanode1.log -o "-p 15432" -Z
> datanode  ---> log is datanode2 on 2nd db host
>
Same here, you need to allow Datanode to accept remote TCP-IP connections.
Even if you set up correctly pg_hba.conf, this is mandatory.
pg_ctl start -D /home/postgres/datanode/data -l
/home/postgresql/coordinator/**log/datanode1.log -o "-i -p 15432" -Z
datanode

Please note that this is equivalent to setting listen_addresses to "*" on
postgresql.conf of each node.
Regards,
-- 
Michael Paquier
http://michael.otacoo.com

Hello all ...

I am in the process of setting up and running postgres-XC 0.9.7 on 
Ubuntu 10.04 LTS 64bit in a development/test sandbox.

I used the install manual, 
http://iweb.dl.sourceforge.net/project/postgres-xc/Version_0.9.7/PG-XC_InstallManual_v0_9_7.pdf 
<http://iweb.dl.sourceforge.net/project/postgres-xc/Version_0.9.7/PG-XC_InstallManual_v0_9_7.pdf>, 
to set up one GTM VM host and two coordinator/datanode VM hosts as this 
is the recommended environment per the manual.

After I initialized the node information in each coordinator using the 
Create Node SQL, I am getting the following error trying to create a 
database on the coord1 node:

     LOG:  failed to connect to data node
     WARNING:  can not connect to datanode 0
     LOG:  failed to acquire connections
     STATEMENT:  create database TEST;
     ERROR:  Failed to get pooled connections
     STATEMENT:  create database TEST;

Was wondering if someone could suggest what is causing my connection 
issues.  I have searched through all the forums on sourceforge as well 
as the net and haven't been able to find anything useful yet.

Thanks in advance,
Phil.




------------------------------


Here are the details:


GTM host install:
----------------
as root:
- installed all necessary Ubuntu packages required for postgres-xc install
- created the postgres-xc user
- created a postgres-xc build directory
- downloaded pgxc_v0.9.7.tar.gz from sourgeforge.net 
<http://sourgeforge.net> and untarred it in build dir above
- config'ed postgres-xc build pointing to custom db home directory with 
NLS enabled and perl
- ran 'make' and 'make install' without issues
- created custom gtm data and log directories

as postgres-xc:
- copied gtm.conf.sample to gtm data dir
- edited the file as per the install manual:

     nodename = 'one'
     listen_addresses = '*'
     port = 6666
     log_file = '/home/postgres/gtm/log/gtm.log'

- started the GTM without issue using the following cmd:

     gtm_ctl start -S gtm -D /home/postgres/gtm/data -l 
/home/postgres/gtm/log/gtm.log


Coord/Datanode host install:
---------------------------

as root:
- installed all necessary Ubuntu packages required for postgres-xc install
- created the postgres-xc user
- created a postgres-xc build directory
- downloaded pgxc_v0.9.7.tar.gz from sourgeforge.net 
<http://sourgeforge.net> and untarred it in build dir above
- config'ed postgres-xc build pointing to custom db home directory with 
NLS enabled and perl
- ran 'make' and 'make install' without issues
- created custom coordinator and datanode data and log directories


as postgres-xc:
- ran initdb for cooordinator on each of the two db hosts
   - data directory was customized (ie not default)
   - nodename was coord1/coord2 respective of db_host_1/db_host_2

- ran initdb for datanode on each of the two db hosts
   - data directory was customized (ie not default)
   - nodename was data1/data2 respective of db_host_1/db_host_2

- edited coordinators postgres.conf and pg_hba.conf respective of the db 
host
   - postgresql.conf
     - pooler_port=6667            (just commented it out)
     - gmt_host=<IP address of GTM host>    (just commented it out)
     - gmt_post=6666                (just commented it out)
     - pgxc_node_name=coord1         (didn't have to change this)  ---> 
it was set to coord2 on the 2nd db host
     - port=5432                (just commented it out)
   - pg_hba.conf
     - added the following line to the end of the file:
         host    all    all <IP_subnet>.0/32    trust    ---> all hosts 
are on the same IP_subnet


- edited datanodes postgres.conf and pg_hba.conf respective of the db host
   - postgresql.conf
     - gmt_host=<IP address of GTM host>    (just commented it out)
     - gmt_post=6666                (just commented it out)
     - pgxc_node_name=data1             (didn't have to change this)  
---> it was set to data2 on the 2nd db host
     - port=15432                (edited it to change to 15432)
   - pg_hba.conf
     - added the following line to the end of the file:
         host    all    all <IP_subnet>.0/32    trust    ---> all hosts 
are on the same IP_subnet

- started the coordinator on each node without issue using the folowing 
command:

     pg_ctl start -D /home/postgres/coordinator/data -l 
/home/postgresql/coordinator/log/coordinator1.log -Z coordinator  ---> 
log is coordinator2 on 2nd db host


- started the datanode on each node without issue using the folowing 
command:

     pg_ctl start -D /home/postgres/datanode/data -l 
/home/postgresql/coordinator/log/datanode1.log -o "-p 15432" -Z 
datanode  ---> log is datanode2 on 2nd db host



Node inititialization on each coordinator:
-----------------------------------------

logged onto each coordinator using psql and ran the following cmds:

     psql -U postgres -d postgres

     CREATE NODE coord1 WITH (TYPE = 'coordinator', PORT = 5432, HOST = 
'<IP_of_coord1>');
     CREATE NODE coord2 WITH (TYPE = 'coordinator', PORT = 5432, HOST = 
'<IP_of_coord2>');

     CREATE NODE data1 WITH (TYPE = 'datanode', PORT = 15432, HOST = 
'<IP_of_data1>');
     CREATE NODE data2 WITH (TYPE = 'datanode', PORT = 15432, HOST = 
'<IP_of_data2>');

     SELECT pgxc_pool_reload();

I did the same thing and had the following:

============================
[koichi@willey:recovery]$ pwd
/home/common/PGXC/pgxc_head/postgres-xc/src/gtm/recovery
[koichi@willey:recovery]$ touch register_gtm.c
[koichi@willey:recovery]$ git branch
  REL0_9_7_STABLE
* master
[koichi@willey:recovery]$ make
gcc -DPGXC -Wall -Wmissing-prototypes -Wpointer-arith
-Wdeclaration-after-statement -Wendif-labels -Wformat-security
-fno-strict-aliasing -fwrapv -g -fpic -I../../../src/include
-D_GNU_SOURCE  -DSO_MAJOR_VERSION=1  -c -o register_gtm.o
register_gtm.c
gcc -DPGXC -Wall -Wmissing-prototypes -Wpointer-arith
-Wdeclaration-after-statement -Wendif-labels -Wformat-security
-fno-strict-aliasing -fwrapv -g -fpic -shared
-Wl,-soname,libgtmrecovery.so.1 -o libgtmrecovery.so.1.0
register_common.o register_gtm.o replication.o standby_utils.o
-L../../../src/port -lpthread -Wl,--as-needed
-Wl,-rpath,'/home/koichi/pgxc/lib',--enable-new-dtags
rm -f libgtmrecovery.so.1
ln -s libgtmrecovery.so.1.0 libgtmrecovery.so.1
rm -f libgtmrecovery.so
ln -s libgtmrecovery.so.1.0 libgtmrecovery.so
ar crs libgtmrecovery.a register_common.o register_gtm.o replication.o
standby_utils.o
ranlib libgtmrecovery.a
[koichi@willey:recovery]$
============================

I did this using ubuntu 10.4 64bit for X64_64 (AMD64) architecture.
Version of make is GNU Make 3.81.   All the source code was taken from
original GIT repository at sourceforge.

Of course, "make" at this directory does not compile all the GTM, just
local sources to be linked with others.

What environment are you working at?

Regards;
----------
Koichi Suzuki



2012年4月13日3:53 Tejeswar Mupparti <tej...@ho...>:
> Thanks Koichi and Michael, may be I'm doing something incorrect
>
>
> [tmuppart@centos0002 main]$ pwd
>
> /home/tmuppart/Install/pgxc/src/gtm/main
>
> [tmuppart@centos0002 main]$ touch ../recovery/register_gtm.c
>
> [tmuppart@centos0002 main]$ make
>
> make: Nothing to be done for `all'.
>
> I'm expecting here gtm binary to be built.
>
>
> [tmuppart@centos0002 gtm]$ pwd
> /home/tmuppart/Install/pgxc/src/gtm
>
> [tmuppart@centos0002 gtm]$ touch recovery/register_gtm.cpp
> [tmuppart@centos0002 gtm]$ make
>
> ar crs libgtmrecovery.a register_common.o register_gtm.o replication.o
> standby_utils.o
> ranlib libgtmrecovery.a
>
> I see libgtmrecovery built, but not gtm binary, So I added a dependency rule
> in
>
> /home/tmuppart/Install/pgxc/src/gtm/main/Makefile
>
> something like this
>
> gtm:$(OBJS) $(OTHERS)
>
> it started building the gtm binary when ever I modify files
> in /home/tmuppart/Install/pgxc/src/gtm/recovery directory
>
> Regards
> Teja
>
>
>> Date: Thu, 12 Apr 2012 13:45:09 +0900
>> From: ko...@in...
>> To: mic...@gm...
>> CC: tej...@ho...; pos...@li...
>> Subject: Re: [Postgres-xc-general] Minor GTM compile question
>
>>
>> If you mena register_gtm.c, not register_gtm.cpp, then you don't need to
>> modify Makefile unless you're using another module.
>> ---
>> Koichi Suzuki
>>
>> On Thu, 12 Apr 2012 12:43:24 +0900
>> Michael Paquier <mic...@gm...> wrote:
>>
>> > On Thu, Apr 12, 2012 at 9:12 AM, Tejeswar Mupparti
>> > <tej...@ho...>wrote:
>> >
>> > > Hi,
>> > >
>> > > In my recent experiments with GTM, I made some changes to the file
>> > >
>> > > "gtm/recovery/register_gtm.cpp"
>> > >
>> > Neither PostgreSQL nor Postgres-XC have c++ files. I am sure you are
>> > talking about register_gtm.c.
>> >
>> >
>> >
>> > >
>> > > and called
>> > >
>> > > gtm> make
>> > >
>> > Is it a make done from the folder src/gtm, or the folder src/gtm/main.
>> > In src/gtm/main, recovery library is referenced already so it should be
>> > enough with the current structure, or I am missing smth.
>> >
>> >
>> > > to build the "gtm binary" with my latest changes, but later realized
>> > > it's
>> > > not picking up my chnages to any of the files made under
>> > > "gtm/recovery"
>> > > directory. It looks like the depenedency
>> > > link in Makefile is missing
>> > >
>> > > > gtm:$(OBJS) <---- Original
>> > > ---
>> > > < gtm:$(OBJS) $(OTHERS) <-- works
>> > >
>> > >
>> > > p.s. I'm not a Makefile expert :), but just hacked it this way, please
>> > > let
>> > > me know if I'm not setting up my codeline incorrect.
>> > >
>> > I am not sure what you are trying to achieve, if you can provide a patch
>> > though...
>> > --
>> > Michael Paquier
>> > http://michael.otacoo.com
>
> ------------------------------------------------------------------------------
> For Developers, A Lot Can Happen In A Second.
> Boundary is the first to Know...and Tell You.
> Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
> http://p.sf.net/sfu/Boundary-d2dvs2
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>

Thanks Koichi and Michael, may be I'm doing something incorrect

[tmuppart@centos0002 main]$ pwd
/home/tmuppart/Install/pgxc/src/gtm/main
[tmuppart@centos0002 main]$ touch ../recovery/register_gtm.c
[tmuppart@centos0002 main]$ make
make: Nothing to be done for `all'.
I'm expecting here gtm binary to be built.

[tmuppart@centos0002 gtm]$ pwd/home/tmuppart/Install/pgxc/src/gtm
[tmuppart@centos0002 gtm]$ touch recovery/register_gtm.cpp [tmuppart@centos0002 gtm]$ make
ar crs libgtmrecovery.a register_common.o register_gtm.o replication.o standby_utils.oranlib libgtmrecovery.a
I see libgtmrecovery built, but not gtm binary, So I added a dependency rule in
/home/tmuppart/Install/pgxc/src/gtm/main/Makefile
something like this
gtm:$(OBJS) $(OTHERS)
it started building the gtm binary when ever I modify files in /home/tmuppart/Install/pgxc/src/gtm/recovery directory
RegardsTeja

> Date: Thu, 12 Apr 2012 13:45:09 +0900
> From: ko...@in...
> To: mic...@gm...
> CC: tej...@ho...; pos...@li...
> Subject: Re: [Postgres-xc-general] Minor GTM compile question
> 
> If you mena register_gtm.c, not register_gtm.cpp, then you don't need to modify Makefile unless you're using another module.
> ---
> Koichi Suzuki
> 
> On Thu, 12 Apr 2012 12:43:24 +0900
> Michael Paquier <mic...@gm...> wrote:
> 
> > On Thu, Apr 12, 2012 at 9:12 AM, Tejeswar Mupparti <tej...@ho...>wrote:
> > 
> > >  Hi,
> > >
> > > In my recent experiments with GTM, I made some changes to the file
> > >
> > > "gtm/recovery/register_gtm.cpp"
> > >
> > Neither PostgreSQL nor Postgres-XC have c++ files. I am sure you are
> > talking about register_gtm.c.
> > 
> > 
> > 
> > >
> > > and called
> > >
> > > gtm> make
> > >
> > Is it a make done from the folder src/gtm, or the folder src/gtm/main.
> > In src/gtm/main, recovery library is referenced already so it should be
> > enough with the current structure, or I am missing smth.
> > 
> > 
> > >  to build the "gtm binary" with my latest changes, but later realized it's
> > > not picking up my chnages to any of the files made under "gtm/recovery"
> > > directory.  It looks like the depenedency
> > > link in Makefile is missing
> > >
> > > > gtm:$(OBJS)                <---- Original
> > > ---
> > > < gtm:$(OBJS) $(OTHERS)     <-- works
> > >
> > >
> > > p.s. I'm not a Makefile expert :), but just hacked it this way, please let
> > > me know if I'm not setting up my codeline incorrect.
> > >
> > I am not sure what you are trying to achieve, if you can provide a patch
> > though...
> > -- 
> > Michael Paquier
> > http://michael.otacoo.com
 		 	   		  

If you mena register_gtm.c, not register_gtm.cpp, then you don't need to modify Makefile unless you're using another module.
---
Koichi Suzuki

On Thu, 12 Apr 2012 12:43:24 +0900
Michael Paquier <mic...@gm...> wrote:

> On Thu, Apr 12, 2012 at 9:12 AM, Tejeswar Mupparti <tej...@ho...>wrote:
> 
> >  Hi,
> >
> > In my recent experiments with GTM, I made some changes to the file
> >
> > "gtm/recovery/register_gtm.cpp"
> >
> Neither PostgreSQL nor Postgres-XC have c++ files. I am sure you are
> talking about register_gtm.c.
> 
> 
> 
> >
> > and called
> >
> > gtm> make
> >
> Is it a make done from the folder src/gtm, or the folder src/gtm/main.
> In src/gtm/main, recovery library is referenced already so it should be
> enough with the current structure, or I am missing smth.
> 
> 
> >  to build the "gtm binary" with my latest changes, but later realized it's
> > not picking up my chnages to any of the files made under "gtm/recovery"
> > directory.  It looks like the depenedency
> > link in Makefile is missing
> >
> > > gtm:$(OBJS)                <---- Original
> > ---
> > < gtm:$(OBJS) $(OTHERS)     <-- works
> >
> >
> > p.s. I'm not a Makefile expert :), but just hacked it this way, please let
> > me know if I'm not setting up my codeline incorrect.
> >
> I am not sure what you are trying to achieve, if you can provide a patch
> though...
> -- 
> Michael Paquier
> http://michael.otacoo.com

Hi all,

Postgres-XC, write-scalable multi-master symetric cluster based on
PostgreSQL, version XC 1.0beta1 has been released.
This beta version is based on PostgreSQL 9.1beta2, and all the fixes of
PostgreSQL 9.1 stable branch will be backported in Postgres-XC 1.0
stabilized version.

You can download the binary directly from here:
https://sourceforge.net/projects/postgres-xc/files/latest/download
This binary contains all the HTML and man documentation.

A PDF file containing all the references is also available here:
https://sourceforge.net/projects/postgres-xc/files/Version_1.0beta1/PG-XC_ReferenceManual_v1_0beta1.pdf/download

Compared to version 0.9.7, the following features have been added:
- Fast query shipping (FQS), quick identification of expressions in a query
that can be pushed down to remote nodes
- SERIAL types
- TABLESPACE
- Utility to clean up 2PC transactions in cluster (pgxc_clean)
- Utility for initialization of GTM (global transaction manager, utility
called initgtm)
- Relation-size functions
- Regression stabilization

The project can be followed on Source Forge:
http://postgres-xc.sourceforge.net/.
And we use a couple of GIT repositories for development:
- SourceForge:
http://postgres-xc.git.sourceforge.net/git/gitweb.cgi?p=postgres-xc/postgres-xc;a=summary
- Github: https://github.com/postgres-xc/postgres-xc

Postgres-XC tutorial at PGCon in Ottawa this May will use a 1.0 version, so
be sure to touch this beta version to have an idea of what is Postgres-XC
before coming!
Since the last release, a special effort has been made on stabilization and
performance improvement, but be sure to give your feedback in order to
provide a stable 1.0 release for PostgreSQL community.

Thanks,

-- 
Michael Paquier
http://michael.otacoo.com

On Thu, Apr 12, 2012 at 9:12 AM, Tejeswar Mupparti <tej...@ho...>wrote:

>  Hi,
>
> In my recent experiments with GTM, I made some changes to the file
>
> "gtm/recovery/register_gtm.cpp"
>
Neither PostgreSQL nor Postgres-XC have c++ files. I am sure you are
talking about register_gtm.c.



>
> and called
>
> gtm> make
>
Is it a make done from the folder src/gtm, or the folder src/gtm/main.
In src/gtm/main, recovery library is referenced already so it should be
enough with the current structure, or I am missing smth.


>  to build the "gtm binary" with my latest changes, but later realized it's
> not picking up my chnages to any of the files made under "gtm/recovery"
> directory.  It looks like the depenedency
> link in Makefile is missing
>
> > gtm:$(OBJS)                <---- Original
> ---
> < gtm:$(OBJS) $(OTHERS)     <-- works
>
>
> p.s. I'm not a Makefile expert :), but just hacked it this way, please let
> me know if I'm not setting up my codeline incorrect.
>
I am not sure what you are trying to achieve, if you can provide a patch
though...
-- 
Michael Paquier
http://michael.otacoo.com

It seems that you added CPP source file, not C source.   Because
Postgres-XC does not assume to have CPP source, it does not include
rules to produce .o from .cpp.   You may have to write these rules in
Makefile manually.   Also, you need to add $(OTHERS) files to be
included in cc command argument to be included.   If you're going to
replace register_gtm.c with register_gtm.cpp, you may need some more
tweaks of Makefile.

Regards;
----------
Koichi Suzuki



2012年4月12日9:12 Tejeswar Mupparti <tej...@ho...>:
> Hi,
>
> In my recent experiments with GTM, I made some changes to the file
>
> "gtm/recovery/register_gtm.cpp"
>
> and called
>
> gtm> make
>
> to build the "gtm binary" with my latest changes, but later realized it's
> not picking up my chnages to any of the files made under "gtm/recovery"
> directory.  It looks like the depenedency
> link in Makefile is missing
>
>> gtm:$(OBJS)                <---- Original
> ---
> < gtm:$(OBJS) $(OTHERS)     <-- works
>
>
> p.s. I'm not a Makefile expert :), but just hacked it this way, please let
> me know if I'm not setting up my codeline incorrect.
>
> Thanks
> Teja
>
> ------------------------------------------------------------------------------
> For Developers, A Lot Can Happen In A Second.
> Boundary is the first to Know...and Tell You.
> Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
> http://p.sf.net/sfu/Boundary-d2dvs2
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>

Hi, In my recent experiments with GTM, I made some changes to the file  "gtm/recovery/register_gtm.cpp"  and called gtm> make to build the "gtm binary" with my latest changes, but later realized it's not picking up my chnages to any of the files made under "gtm/recovery" directory.  It looks like the depenedencylink in Makefile is missing
> gtm:$(OBJS)                <---- Original---< gtm:$(OBJS) $(OTHERS)     <-- works

p.s. I'm not a Makefile expert :), but just hacked it this way, please let me know if I'm not setting up my codeline incorrect. ThanksTeja 		 	   		  

On Thu, Apr 5, 2012 at 2:09 AM, Tejeswar Mupparti <tej...@ho...>wrote:

>  Hi,
>
> I have a quick question on a GTM behavior
>
> We are a starting a new initiative and exploring PG-XC for our
> Cloud-database. I installed a 2 node PG-XC cluster and in one of the
> scenario I did a kill -9 on one of the data node and later restarted it,
> but it failed to join the cluster(repeatedly) with this message
>
> FATAL:  Can not register Datanode on GTM
>
> When I looked at the GTM code, the routine pgxcnode_add_info() is trying
> to add the datanode into the global table and finds a duplicate and rejects
> the registration, this is obvious since the datanode wasn't a clean
> shutdown. Is this considered a bug or an expected behavior?
>
As cited, this is expected behavior, we cannot take the risk to have a not
registering once again in cluster once it has been shutdown abruptly.

>
> pgxcnode_add_info
> Recovery_PGXCNodeRegister
> ProcessPGXCNodeRegister
> ProcessPGXCNodeCommand (argp=0x10c8ca8) at main.cpp:1421
>
> In general for a cluster system we will have membership layer, heartbeat
> check is performed and datanode death might have been detected and a
> cleanup is done for global hash table.
>
> BTW I have worked around the problem by calling Unregister just before
> calling pgxcnode_add_info(). Can you please share your ideas and thoughts.
>
There are currently no direct way to reallow a node to register with the
same node name in cluster for the time being. You need to erase the
registration information written on disk by GTM and then restart your
cluster.
But here are some thoughts about that.
1) We could add an option in postgres binary to block node registration
when node is up. Something like:
postgres --block-register
2) We could add a GUC parameter in postgresql.conf, simply an on/off
switch, that controls if the node needs to register in case it recovers
from a crash.
Those are just some ideas of course, but option 2 looks more flexible.

Regards,
-- 
Michael Paquier
http://michael.otacoo.com

On Wed, 4 Apr 2012 10:09:17 -0700
Tejeswar Mupparti <tej...@ho...> wrote:

> Hi, I have a quick question on a GTM behavior We are a starting a new initiative and exploring PG-XC for our Cloud-database. I installed a 2 node PG-XC cluster and in one of the scenario I did a kill -9 on one of the data node and later restarted it, but it failed to join the cluster(repeatedly) with this message FATAL:  Can not register Datanode on GTM When I looked at the GTM code, the routine pgxcnode_add_info() is trying to add the datanode into the global table and finds a duplicate and rejects the registration, this is obvious since the datanode wasn't a clean shutdown. Is this considered a bug or an expected behavior? pgxcnode_add_infoRecovery_PGXCNodeRegister
> ProcessPGXCNodeRegisterProcessPGXCNodeCommand (argp=0x10c8ca8) at main.cpp:1421 In general for a cluster system we will have membership layer, heartbeat check is performed and datanode death might have been detected and a cleanup is done for global hash table. BTW I have worked around the problem by calling Unregister just before  calling pgxcnode_add_info(). Can you please share your ideas and thoughts.  ThanksTeja  		 	   		  

This is expected behavior.   This is caused by terminating the node abnormally.   In this case, if datanode stops abnormally or crashes, I advise to restart the whole cluster.   You should remove register.node file before you restart GTM.

Regards;
---
Koichi Suzuki

Hi, I have a quick question on a GTM behavior We are a starting a new initiative and exploring PG-XC for our Cloud-database. I installed a 2 node PG-XC cluster and in one of the scenario I did a kill -9 on one of the data node and later restarted it, but it failed to join the cluster(repeatedly) with this message FATAL:  Can not register Datanode on GTM When I looked at the GTM code, the routine pgxcnode_add_info() is trying to add the datanode into the global table and finds a duplicate and rejects the registration, this is obvious since the datanode wasn't a clean shutdown. Is this considered a bug or an expected behavior? pgxcnode_add_infoRecovery_PGXCNodeRegister
ProcessPGXCNodeRegisterProcessPGXCNodeCommand (argp=0x10c8ca8) at main.cpp:1421 In general for a cluster system we will have membership layer, heartbeat check is performed and datanode death might have been detected and a cleanup is done for global hash table. BTW I have worked around the problem by calling Unregister just before  calling pgxcnode_add_info(). Can you please share your ideas and thoughts.  ThanksTeja