postgres-xc-general Mailing List for Postgres-XC

Paul,

I am working on moving a 25 node cluster into production with PGXC trunk (against advice); if deployment works out we will likely put some development time in.

Feel free to send me an email, Best,

Arni Sumarlidason | Software Engineer, Information Technology
MDA | 820 West Diamond Ave | Gaithersburg, MD | USA
O: 240-833-8200 D: 240-833-8318 M: 256-393-2803
arn...@md...<mailto:arn...@md...>| http://www.mdaus.com<http://www.mdaus.com/>

Hi Koichi,

See answers below

On 04/01/2013 07:32 AM, Koichi Suzuki wrote:
> Please check if --prefix (or /usr/local/pgsql, the default) is the 
> same as your pgxc_ctl configuration's xc installation directory.  This 
> is needed to make "deploy all" work correctly.

They are the same:
./configure --prefix=/var/lib/postgres-xc/pgxc
pgxcInstallDir=/var/lib/postgres-xc/pgxc


deploy all works without errors


>
> pgxc_monitor uses psql ... -c "select 1" to check if the target is 
> running.  That is, it checks if the target can handle a statement 
> correctly.   Sometimes, for example, if the slave is connected 
> synchronously (pgxc_ctl uses synchronous replication) and the slave 
> does not work, the master cannot handle incoming statements until the 
> slave replies to the master correctly.   This is one possibility that 
> the master is determined "not working".   (In 1.1, I'm improving this).

postgres-xc@node-pgxcdb01:~$ psql pgxc_ctl init gives:

http://pastebin.com/sccGQm1u

Apparently the problem is here:
bash: /var/lib/postgres-xc/pgxc/nodes/coord/pg_hba.conf: No such file or 
directory
bash: /var/lib/postgres-xc/pgxc/nodes/coord/pg_hba.conf: No such file or 
directory
bash: /var/lib/postgres-xc/pgxc/nodes/coord/pg_hba.conf: No such file or 
directory
bash: /var/lib/postgres-xc/pgxc/nodes/coord/pg_hba.conf: No such file or 
directory

Any idea why it fails to create them?


Is this the correct test command? :

postgres-xc@node-pgxcdb01:~$ psql -p 20008 postgres -c select 1
psql: FATAL:  role "1" does not exist


>
> Could you please check one of the log for datanode master/slave and 
> see if there's any log indicating errors? This is very helpful to 
> determine what is wrong.
>
Logs on dn_master:

postgres-xc@node-pgxcdb01:~/pgxc/nodes/dn_master/pg_log$ cat 
postgresql-2013-04-03_155857.log
LOG:  database system was shut down at 2013-04-03 15:57:49 EEST
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started
FATAL:  no pg_hba.conf entry for replication connection from host 
"192.168.20.103", user "postgres-xc"
LOG:  received smart shutdown request
LOG:  autovacuum launcher shutting down
LOG:  shutting down
LOG:  database system is shut down

postgres-xc@node-pgxcdb01:~/pgxc/nodes/dn_master/pg_log$ cat 
postgresql-2013-04-03_160100.log
LOG:  database system was shut down at 2013-04-03 15:59:04 EEST
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started
FATAL:  no pg_hba.conf entry for host "192.168.20.101", user 
"postgres-xc", database "postgres"
FATAL:  no pg_hba.conf entry for host "192.168.20.101", user 
"postgres-xc", database "postgres"
FATAL:  role "1" does not exist
FATAL:  database "1" does not exist
FATAL:  role "1" does not exist

dn_slave failed to replicate so, no logs.

> I think these partly answer your question a) and c).
>
> I don't think b) works here, because deploy all just copies all the 
> binaries to the target servers.   From your report, if ./configure and 
> pgxc_ctl configuration matches, I found no problem in it.
>
> Hope it helps.
>
> Please feel free to write for further help.
>
> Regards;
> ---
> Koichi Suzuki
>
> ----------
> Koichi Suzuki
>
>
> 2013/3/30 Theodotos Andreou <th...@ub... <mailto:th...@ub...>>
>
>     Guys I am using pgxc_ctl to built a HA postgres-xc cluster with 4
>     coordinator/datanodes and 2 gtm nodes:
>
>     I use the postgres-xc user and I have passwordless ssh between all
>     nodes
>     and the separate pgxc_ctl machine.
>     The dirs ~/bin and ~/pgxc/bin are in the $PATH of postgres-xc user.
>
>     These are the steps I followed:
>
>     1) Complile pgxc and tools:
>
>     postgres-xc@node-pgxc-ctl:~/pgxc-v1.0.2$ ./configure
>     --prefix=/var/lib/postgres-xc/pgxc
>     postgres-xc@node-pgxc-ctl:~/pgxc-v1.0.2$ make
>     postgres-xc@node-pgxc-ctl:~/pgxc-v1.0.2$ make install
>     postgres-xc@node-pgxc-ctl:~/pgxc-v1.0.2$ cd contrib/
>     postgres-xc@node-pgxc-ctl:~/pgxc-v1.0.2/contrib$ make install
>     postgres-xc@node-pgxc-ctl:~/pgxc-v1.0.2/contrib$ cd pgxc_monitor/
>     # This
>     is not included in latest pgxc. I had to download it separately.
>     postgres-xc@node-pgxc-ctl:~/pgxc-v1.0.2/contrib/pgxc_monitor$ make
>     install
>
>     2) Download script:
>
>     I downloaded the pgxc_ctl in ~/bin and patched line 4566 (put a
>     space in
>     front of ']')
>
>     3) Cinfiguration of pgxc_ctl
>
>     Created the config file:
>
>     postgres-xc@node-pgxc-ctl:~$ pgxc_ctl prepare config
>
>     I edited ~/pgxc/pgxcConf such as:
>
>     http://pastebin.com/DHwpwdr1
>
>     4) Deployed:
>
>     postgres-xc@node-pgxc-ctl:~$ pgxc_ctl deploy all
>     wk.tgz 100% 7365KB   7.2MB/s   00:00
>     wk.tgz 100% 7365KB   7.2MB/s   00:00
>     wk.tgz 100% 7365KB   7.2MB/s   00:00
>     wk.tgz 100% 7365KB   7.2MB/s   00:00
>     wk.tgz 100% 7365KB   7.2MB/s   00:00
>     wk.tgz 100% 7365KB   7.2MB/s   00:01
>
>     deploy log:
>
>     http://pastebin.com/eETjNmmV
>
>     5) Initialize all:
>
>     postgres-xc@node-pgxc-ctl:~$ pgxc_ctl init
>
>     This gives a lot of errors. Init log:
>
>     http://pastebin.com/ML5HLJ2i
>
>     6) Monitor all
>
>     postgres-xc@node-pgxc-ctl:~$ pgxc_ctl monitor all
>     GTM master (gtm): running.    host: node-pgxcgtm01, port: 20001, dir:
>     /var/lib/postgres-xc/pgxc/nodes/gtm
>     GTM slave (gtm): running.    host: node-pgxcgtm02, port: 20001, dir:
>     /var/lib/postgres-xc/pgxc/nodes/gtm
>     GTM proxy (gtm_pxy1): running.    host: node-pgxcdb01, port:
>     20001, dir:
>     /var/lib/postgres-xc/pgxc/nodes/gtm_pxy
>     GTM proxy (gtm_pxy2): running.    host: node-pgxcdb02, port:
>     20001, dir:
>     /var/lib/postgres-xc/pgxc/nodes/gtm_pxy
>     GTM proxy (gtm_pxy3): running.    host: node-pgxcdb03, port:
>     20001, dir:
>     /var/lib/postgres-xc/pgxc/nodes/gtm_pxy
>     GTM proxy (gtm_pxy4): running.    host: node-pgxcdb04, port:
>     20001, dir:
>     /var/lib/postgres-xc/pgxc/nodes/gtm_pxy
>     Coordinator master (coord1): running.    host: node-pgxcdb01, port:
>     20004, dir: /var/lib/postgres-xc/pgxc/nodes/coord
>     Coordinator master (coord2): running.    host: node-pgxcdb02, port:
>     20005, dir: /var/lib/postgres-xc/pgxc/nodes/coord
>     Coordinator master (coord3): running.    host: node-pgxcdb03, port:
>     20004, dir: /var/lib/postgres-xc/pgxc/nodes/coord
>     Coordinator master (coord4): running.    host: node-pgxcdb04, port:
>     20005, dir: /var/lib/postgres-xc/pgxc/nodes/coord
>     Coordinator slave (coord1): running.    host: node-pgxcdb02, port:
>     20004, dir: /var/lib/postgres-xc/pgxc/nodes/coord_slave
>     Coordinator slave (coord2): running.    host: node-pgxcdb03, port:
>     20005, dir: /var/lib/postgres-xc/pgxc/nodes/coord_slave
>     Coordinator slave (coord3): running.    host: node-pgxcdb04, port:
>     20004, dir: /var/lib/postgres-xc/pgxc/nodes/coord_slave
>     Coordinator slave (coord4): running.    host: node-pgxcdb01, port:
>     20005, dir: /var/lib/postgres-xc/pgxc/nodes/coord_slave
>     Datanode master (datanode1): not running.    host: node-pgxcdb01,
>     port:
>     20008, dir: /var/lib/postgres-xc/pgxc/nodes/dn_master
>     Datanode master (datanode2): not running.    host: node-pgxcdb02,
>     port:
>     20009, dir: /var/lib/postgres-xc/pgxc/nodes/dn_master
>     Datanode master (datanode3): not running.    host: node-pgxcdb03,
>     port:
>     20008, dir: /var/lib/postgres-xc/pgxc/nodes/dn_master
>     Datanode master (datanode4): not running.    host: node-pgxcdb04,
>     port:
>     20009, dir: /var/lib/postgres-xc/pgxc/nodes/dn_master
>     Datanode slave (datanode1): not running.    host: node-pgxcdb02, port:
>     20008, dir: /var/lib/postgres-xc/pgxc/nodes/dn_slave
>     Datanode slave (datanode2): not running.    host: node-pgxcdb03, port:
>     20009, dir: /var/lib/postgres-xc/pgxc/nodes/dn_slave
>     Datanode slave (datanode3): not running.    host: node-pgxcdb04, port:
>     20008, dir: /var/lib/postgres-xc/pgxc/nodes/dn_slave
>     Datanode slave (datanode4): not running.    host: node-pgxcdb01, port:
>     20009, dir: /var/lib/postgres-xc/pgxc/nodes/dn_slave
>
>     But not all of the above information is true. Coordinators are all up
>     but only the datanode masters are working:
>
>     # dsh -a 'echo "" ; echo $HOSTNAME ; echo "------------" ; ps auxw |
>     grep postgres | grep dn'
>
>     node-pgxcdb01
>     ------------
>     999      13843  0.0  0.3  53236  7308 ?        S    13:15 0:00
>     /var/lib/postgres-xc/pgxc/bin/postgres -X -D
>     /var/lib/postgres-xc/pgxc/nodes/dn_master -i
>     root     18229  0.0  0.0  11008  1432 ?        Ss   14:21 0:00 bash -c
>     echo "" ; echo $HOSTNAME ; echo "------------" ; ps auxw | grep
>     postgres
>     | grep dn
>
>     node-pgxcdb02
>     ------------
>     999      12911  0.0  0.3  53232  7304 ?        S    13:15 0:00
>     /var/lib/postgres-xc/pgxc/bin/postgres -X -D
>     /var/lib/postgres-xc/pgxc/nodes/dn_master -i
>     root     17295  0.0  0.0  11008  1436 ?        Ss   14:21 0:00 bash -c
>     echo "" ; echo $HOSTNAME ; echo "------------" ; ps auxw | grep
>     postgres
>     | grep dn
>
>     node-pgxcdb03
>     ------------
>     999      12918  0.0  0.3  53232  7312 ?        S    13:15 0:00
>     /var/lib/postgres-xc/pgxc/bin/postgres -X -D
>     /var/lib/postgres-xc/pgxc/nodes/dn_master -i
>     root     17383  0.0  0.0  11008  1436 ?        Ss   14:21 0:00 bash -c
>     echo "" ; echo $HOSTNAME ; echo "------------" ; ps auxw | grep
>     postgres
>     | grep dn
>
>     node-pgxcdb04
>     ------------
>     999      12920  0.0  0.3  53240  7316 ?        S    13:15 0:00
>     /var/lib/postgres-xc/pgxc/bin/postgres -X -D
>     /var/lib/postgres-xc/pgxc/nodes/dn_master -i
>     root     17270  0.0  0.0  11008  1432 ?        Ss   14:21 0:00 bash -c
>     echo "" ; echo $HOSTNAME ; echo "------------" ; ps auxw | grep
>     postgres
>     | grep dn
>
>     node-pgxcgtm01
>     ------------
>     root      7087  0.0  0.0  11008  1432 ?        Ss   14:21 0:00 bash -c
>     echo "" ; echo $HOSTNAME ; echo "------------" ; ps auxw | grep
>     postgres
>     | grep dn
>
>     node-pgxcgtm02
>     ------------
>     root      5913  0.0  0.0  11008  1432 ?        Ss   14:21 0:00 bash -c
>     echo "" ; echo $HOSTNAME ; echo "------------" ; ps auxw | grep
>     postgres
>     | grep dn
>
>
>     Trying to start the slaves manually I get:
>     postgres-xc@node-pgxcdb01:~$ /var/lib/postgres-xc/pgxc/bin/postgres -X
>     -D /var/lib/postgres-xc/pgxc/nodes/dn_slave/ -i
>     FATAL:  "/var/lib/postgres-xc/pgxc/nodes/dn_slave" is not a valid data
>     directory
>     DETAIL:  File "/var/lib/postgres-xc/pgxc/nodes/dn_slave/PG_VERSION" is
>     missing.
>
>     The datanode slaves have not been initialized because pg_basebackup
>     fails with:
>
>     Questions:
>
>     a) Any idea what I am doing wrong? Init logs are not very clear
>     what is
>     the problem.
>
>     b) Can I run "pgxc_ctl deploy all" again? Will this mess up the
>     current
>     setup if I do?
>
>     c) Why does pgxc_monitor reports that datanode masters are down
>     when are
>     in fact working?
>
>     ------------------------------------------------------------------------------
>     Own the Future-Intel(R) Level Up Game Demo Contest 2013
>     Rise to greatness in Intel's independent game demo contest. Compete
>     for recognition, cash, and the chance to get your game on Steam.
>     $5K grand prize plus 10 genre and skill prizes. Submit your demo
>     by 6/6/13. http://altfarm.mediaplex.com/ad/ck/12124-176961-30367-2
>     _______________________________________________
>     Postgres-xc-general mailing list
>     Pos...@li...
>     <mailto:Pos...@li...>
>     https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
>

Another possibility is using partitioning and replication to take advantage
of PostgreSQL constraint exclusion + replication in XC.

You can partition the table using inheritance and every child and parent
table is replicated. That way, partitions can be pruned using constraint
exclusion and replication will give advantages of JOIN/SQL operation
push-down.


On Tue, Apr 2, 2013 at 7:43 PM, Michael Paquier
<mic...@gm...>wrote:

>
>
>
> On Tue, Apr 2, 2013 at 10:27 PM, German M. Bravo <ger...@gm...>wrote:
>
>> I have this question, as I want to do distribution by some other criteria
>> other than replication, modulo/hash or round-robin. I need to distribute
>> tables as to have certain rows replicated in certain nodes and other rows
>> replicated in yet other nodes.
>>
>> My idea is that of implementing something similar to what is proposed by
>> the very interesting paper "The Little Engine(s) That Could: Scaling Online
>> Social Networks" (http://ccr.sigcomm.org/online/files/p375.pdf), where
>> one can partition and replicate a OSN based on the relationships the users
>> have among them. The proposal guarantees all data for each user, as well as
>> that of all of its neighborhoods' (or direct friends') are located in the
>> same node, for performance reasons and to reduce network traffic. For that
>> intend, some users end up being replicated in several nodes.
>>
>> To make this possible in Postgres XC, I'd need to be able to tell pgxc
>> where (and how) to distribute each row. My thinking was: I wish I could
>> distribute by a column that is a "vector" of values (if only a value it'd
>> be simple distribution, as that of modulo or hash, if multiple values, it'd
>> be replication as well as distribution), where, at the application level, I
>> could tell the database: this row (user X) should be located in node X *and
>> also* replicated in all the nodes of all of its friends (User A - User
>> J)... so the distribution vector would be (X, A, B, C, ..., J). Replication
>> of row for user X, thus, would exist in all of those nodes.
>>
>> Any thoughts about how difficult/tricky this would be? My first guess is
>> this shouldn't be too hard to implement, thinking pgxc already supports
>> both replication and distribution and this would be something like
>> something in-between.
>>
>> Michael Paquier suggested there might be other solutions to this problem,
>> using what is already there in place within existent PgXC. Can anyone think
>> and explain about that?
>>
> For the time being you should use partitions on tables. You create a
> parent table, and child subtables having different distribution types, that
> are partitioned a given way. more information about this idea here:
> http://sourceforge.net/mailarchive/message.php?msg_id=30421838
> Regards,
> --
> Michael
>
>
> ------------------------------------------------------------------------------
> Own the Future-Intel(R) Level Up Game Demo Contest 2013
> Rise to greatness in Intel's independent game demo contest. Compete
> for recognition, cash, and the chance to get your game on Steam.
> $5K grand prize plus 10 genre and skill prizes. Submit your demo
> by 6/6/13. http://altfarm.mediaplex.com/ad/ck/12124-176961-30367-2
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
>


-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

I'm interested in using "partition" on XC tables.   Could you share your
experience?
----------
Koichi Suzuki


2013/4/2 Michael Paquier <mic...@gm...>

>
>
>
> On Tue, Apr 2, 2013 at 10:27 PM, German M. Bravo <ger...@gm...>wrote:
>
>> I have this question, as I want to do distribution by some other criteria
>> other than replication, modulo/hash or round-robin. I need to distribute
>> tables as to have certain rows replicated in certain nodes and other rows
>> replicated in yet other nodes.
>>
>> My idea is that of implementing something similar to what is proposed by
>> the very interesting paper "The Little Engine(s) That Could: Scaling Online
>> Social Networks" (http://ccr.sigcomm.org/online/files/p375.pdf), where
>> one can partition and replicate a OSN based on the relationships the users
>> have among them. The proposal guarantees all data for each user, as well as
>> that of all of its neighborhoods' (or direct friends') are located in the
>> same node, for performance reasons and to reduce network traffic. For that
>> intend, some users end up being replicated in several nodes.
>>
>> To make this possible in Postgres XC, I'd need to be able to tell pgxc
>> where (and how) to distribute each row. My thinking was: I wish I could
>> distribute by a column that is a "vector" of values (if only a value it'd
>> be simple distribution, as that of modulo or hash, if multiple values, it'd
>> be replication as well as distribution), where, at the application level, I
>> could tell the database: this row (user X) should be located in node X *and
>> also* replicated in all the nodes of all of its friends (User A - User
>> J)... so the distribution vector would be (X, A, B, C, ..., J). Replication
>> of row for user X, thus, would exist in all of those nodes.
>>
>> Any thoughts about how difficult/tricky this would be? My first guess is
>> this shouldn't be too hard to implement, thinking pgxc already supports
>> both replication and distribution and this would be something like
>> something in-between.
>>
>> Michael Paquier suggested there might be other solutions to this problem,
>> using what is already there in place within existent PgXC. Can anyone think
>> and explain about that?
>>
> For the time being you should use partitions on tables. You create a
> parent table, and child subtables having different distribution types, that
> are partitioned a given way. more information about this idea here:
> http://sourceforge.net/mailarchive/message.php?msg_id=30421838
> Regards,
> --
> Michael
>
>
> ------------------------------------------------------------------------------
> Own the Future-Intel(R) Level Up Game Demo Contest 2013
> Rise to greatness in Intel's independent game demo contest. Compete
> for recognition, cash, and the chance to get your game on Steam.
> $5K grand prize plus 10 genre and skill prizes. Submit your demo
> by 6/6/13. http://altfarm.mediaplex.com/ad/ck/12124-176961-30367-2
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
>