postgres-xc-general Mailing List for Postgres-XC

Here few more logs of the component which doesn't start

root@db02:~# tail -f
/home/postgres/pgxc/nodes/dn_master/pg_log/postgresql-2014-04-14_114619.log
LOG:  database system was interrupted; last known up at 2014-04-14 11:00:55
WAT
LOG:  database system was not properly shut down; automatic recovery in
progress
LOG:  record with zero length at 4/9A01A588
LOG:  redo is not required
FATAL:  the database system is starting up
LOG:  autovacuum launcher started
LOG:  database system is ready to accept connections
WARNING:  worker took too long to start; canceled
FATAL:  Can not register Datanode on GTM


root@db02:~# tail -f
/home/postgres/pgxc/nodes/coord/pg_log/postgresql-2014-04-14_114612.log
DEBUG:  00000: name: unnamed; blockState:       DEFAULT; state: INPROGR,
xid/subid/cid: 0/1/0, nestlvl: 1, children:
LOCATION:  ShowTransactionStateRec, xact.c:5238
DEBUG:  00000: Autovacuum launcher: connection to GTM closed
LOCATION:  CloseGTM, gtm.c:116
DEBUG:  00000: Autovacuum launcher: connection established to GTM with
string host=db02 port=20002 node_name=coord1
LOCATION:  InitGTM, gtm.c:84
WARNING:  01000: Xid is invalid.
LOCATION:  GetNewTransactionId, varsup.c:160
DEBUG:  00000: Getting snapshot. Current XID = 0
LOCATION:  GetSnapshotDataCoordinator, procarray.c:3054
DEBUG:  00000: Autovacuum launcher: connection to GTM closed
LOCATION:  CloseGTM, gtm.c:116
DEBUG:  00000: Autovacuum launcher: connection established to GTM with
string host=db02 port=20002 node_name=coord1
LOCATION:  InitGTM, gtm.c:84





On Mon, Apr 14, 2014 at 3:56 PM, Juned Khan <jkh...@gm...> wrote:

> Hi all,
>
> Yesterday due some problem datanode slave stopped working on one of my DB
> server. After figuring out the issue i freed out the space on that server.
> now almost 50% disk is free on server.
>
> the problem is after that incident i am not able start all my pgxc
> components, now its uncertain some of components starts and sometimes it
> does not each time when i execute "stop all" and "start all" scenario is
> different.I am not able to figure out the problem which causing this issue.
>
> I have enabled the debug log though and tried to identify the issue and
> after reviewing the logs it seems memory related issue, i am not sure about
> this.
>
> PGXC start all
> Start GTM master
> gtm_ctl: PID file "/home/postgres/pgxc/nodes/gtm/gtm.pid" does not exist
> Is server running?
> server starting
> Start GTM slavegtm_ctl: PID file "/home/postgres/pgxc/nodes/gtm/gtm.pid"
> does not exist
> Is server running?
> server starting
> Done.
> Starting all the gtm proxies.
> Starting gtm proxy gtm_pxy1.
> Starting gtm proxy gtm_pxy2.
> gtm_proxy: no process found
> server starting
> gtm_proxy: no process found
> server starting
> Done.
> Starting coordinator master.
> Starting coordinator master coord1
> Starting coordinator master coord2
> DEBUG:  00000: postgres: PostmasterMain: initial environment dump:
> LOCATION:  PostmasterMain, postmaster.c:962
> DEBUG:  00000: -----------------------------------------
> LOCATION:  PostmasterMain, postmaster.c:964
> DEBUG:  00000:  MAIL=/var/mail/postgres
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  SSH_CLIENT=41.218.72.115 48992 59696
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  USER=postgres
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  LANGUAGE=en_ZA:en
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  SHLVL=1
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  HOME=/home/postgres
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  PG_GRANDPARENT_PID=19891
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  LOGNAME=postgres
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  _=/usr/local/bin/pg_ctl
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  PGSYSCONFDIR=/usr/local/pgsql/etc
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  LANG=en_ZA.UTF-8
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  SHELL=/bin/bash
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  PWD=/home/postgres
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  SSH_CONNECTION=41.218.72.115 48992 41.218.72.115 59696
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  PGDATA=/home/postgres/pgxc/nodes/coord
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  LC_COLLATE=en_ZA.UTF-8
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  LC_CTYPE=en_ZA.UTF-8
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  LC_MESSAGES=en_ZA.UTF-8
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  LC_MONETARY=C
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  LC_NUMERIC=C
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  LC_TIME=C
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000: -----------------------------------------
> LOCATION:  PostmasterMain, postmaster.c:969
> DEBUG:  00000: invoking IpcMemoryCreate(size=148193280)
> LOCATION:  CreateSharedMemoryAndSemaphores, ipci.c:149
> DEBUG:  00000: SlruScanDirectory invoking callback on pg_notify/0000
> LOCATION:  SlruScanDirectory, slru.c:1312
> DEBUG:  00000: removing file "pg_notify/0000"
> LOCATION:  SlruScanDirCbDeleteAll, slru.c:1277
> DEBUG:  00000: max_safe_fds = 984, usable_fds = 1000, already_open = 6
> LOCATION:  set_max_safe_fds, fd.c:548
> LOG:  00000: redirecting log output to logging collector process
> HINT:  Future log output will appear in directory "pg_log".
> LOCATION:  SysLogger_Start, syslogger.c:649
> DEBUG:  00000: postgres: PostmasterMain: initial environment dump:
> LOCATION:  PostmasterMain, postmaster.c:962
> DEBUG:  00000: -----------------------------------------
> LOCATION:  PostmasterMain, postmaster.c:964
> DEBUG:  00000:  MAIL=/var/mail/postgres
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  SSH_CLIENT=41.218.72.115 33169 59696
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  USER=postgres
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  LANGUAGE=en_ZA:en
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  SHLVL=1
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  HOME=/home/postgres
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  PG_GRANDPARENT_PID=19507
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  LOGNAME=postgres
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  _=/usr/local/bin/pg_ctl
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  PGSYSCONFDIR=/usr/local/pgsql/etc
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  LANG=en_ZA.UTF-8
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  SHELL=/bin/bash
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  PWD=/home/postgres
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  SSH_CONNECTION=41.218.72.115 33169 41.218.72.114 59696
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  PGDATA=/home/postgres/pgxc/nodes/coord
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  LC_COLLATE=en_ZA.UTF-8
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  LC_CTYPE=en_ZA.UTF-8
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  LC_MESSAGES=en_ZA.UTF-8
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  LC_MONETARY=C
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  LC_NUMERIC=C
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000:  LC_TIME=C
> LOCATION:  PostmasterMain, postmaster.c:967
> DEBUG:  00000: -----------------------------------------
> LOCATION:  PostmasterMain, postmaster.c:969
> DEBUG:  00000: invoking IpcMemoryCreate(size=148193280)
> LOCATION:  CreateSharedMemoryAndSemaphores, ipci.c:149
> DEBUG:  00000: SlruScanDirectory invoking callback on pg_notify/0000
> LOCATION:  SlruScanDirectory, slru.c:1312
> DEBUG:  00000: removing file "pg_notify/0000"
> LOCATION:  SlruScanDirCbDeleteAll, slru.c:1277
> DEBUG:  00000: max_safe_fds = 984, usable_fds = 1000, already_open = 6
> LOCATION:  set_max_safe_fds, fd.c:548
> LOG:  00000: redirecting log output to logging collector process
> HINT:  Future log output will appear in directory "pg_log".
> LOCATION:  SysLogger_Start, syslogger.c:649
> DEBUG:  00000: logger shutting down
> LOCATION:  SysLoggerMain, syslogger.c:517
> DEBUG:  00000: shmem_exit(0): 0 callbacks to make
> LOCATION:  shmem_exit, ipc.c:212
> DEBUG:  00000: proc_exit(0): 0 callbacks to make
> LOCATION:  proc_exit_prepare, ipc.c:184
> DEBUG:  00000: exit(0)
> LOCATION:  proc_exit, ipc.c:135
> DEBUG:  00000: shmem_exit(-1): 0 callbacks to make
> LOCATION:  shmem_exit, ipc.c:212
> DEBUG:  00000: proc_exit(-1): 0 callbacks to make
> LOCATION:  proc_exit_prepare, ipc.c:184
> pg_ctl: could not start server
> Examine the log output.
> Done.
> Starting all the datanode masters.
> Starting datanode master datanode1.
> LOG:  redirecting log output to logging collector process
> HINT:  Future log output will appear in directory "pg_log".
> Done.
> Starting all the datanode slaves.
> Starting datanode slave datanode1.
> LOG:  redirecting log output to logging collector process
> HINT:  Future log output will appear in directory "pg_log".
> Done.
> PGXC monitor all
> Running: gtm master
> Running: gtm slave
> Running: gtm proxy gtm_pxy1
> Running: gtm proxy gtm_pxy2
> Running: coordinator master coord1
> Not running: coordinator master coord2
> Not running: datanode master datanode1
> Running: datanode slave datanode1
>
>
> My DB server has 32 GB RAM and 260 GB of Hard disk and i am using
> pgxc-1.2.1. what would be the optimal memory related postgresql.conf
> configuration for this server.
>
> Anyone has an idea about this issue ?
>
> --
> Thanks,
> Juned Khan
> iNextrix Technologies Pvt Ltd.
> www.inextrix.com
>



-- 
Thanks,
Juned Khan
iNextrix Technologies Pvt Ltd.
www.inextrix.com

Hi all,

Yesterday due some problem datanode slave stopped working on one of my DB
server. After figuring out the issue i freed out the space on that server.
now almost 50% disk is free on server.

the problem is after that incident i am not able start all my pgxc
components, now its uncertain some of components starts and sometimes it
does not each time when i execute "stop all" and "start all" scenario is
different.I am not able to figure out the problem which causing this issue.

I have enabled the debug log though and tried to identify the issue and
after reviewing the logs it seems memory related issue, i am not sure about
this.

PGXC start all
Start GTM master
gtm_ctl: PID file "/home/postgres/pgxc/nodes/gtm/gtm.pid" does not exist
Is server running?
server starting
Start GTM slavegtm_ctl: PID file "/home/postgres/pgxc/nodes/gtm/gtm.pid"
does not exist
Is server running?
server starting
Done.
Starting all the gtm proxies.
Starting gtm proxy gtm_pxy1.
Starting gtm proxy gtm_pxy2.
gtm_proxy: no process found
server starting
gtm_proxy: no process found
server starting
Done.
Starting coordinator master.
Starting coordinator master coord1
Starting coordinator master coord2
DEBUG:  00000: postgres: PostmasterMain: initial environment dump:
LOCATION:  PostmasterMain, postmaster.c:962
DEBUG:  00000: -----------------------------------------
LOCATION:  PostmasterMain, postmaster.c:964
DEBUG:  00000:  MAIL=/var/mail/postgres
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  SSH_CLIENT=41.218.72.115 48992 59696
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  USER=postgres
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  LANGUAGE=en_ZA:en
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  SHLVL=1
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  HOME=/home/postgres
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  PG_GRANDPARENT_PID=19891
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  LOGNAME=postgres
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  _=/usr/local/bin/pg_ctl
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  PGSYSCONFDIR=/usr/local/pgsql/etc
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  LANG=en_ZA.UTF-8
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  SHELL=/bin/bash
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  PWD=/home/postgres
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  SSH_CONNECTION=41.218.72.115 48992 41.218.72.115 59696
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  PGDATA=/home/postgres/pgxc/nodes/coord
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  LC_COLLATE=en_ZA.UTF-8
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  LC_CTYPE=en_ZA.UTF-8
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  LC_MESSAGES=en_ZA.UTF-8
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  LC_MONETARY=C
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  LC_NUMERIC=C
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  LC_TIME=C
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000: -----------------------------------------
LOCATION:  PostmasterMain, postmaster.c:969
DEBUG:  00000: invoking IpcMemoryCreate(size=148193280)
LOCATION:  CreateSharedMemoryAndSemaphores, ipci.c:149
DEBUG:  00000: SlruScanDirectory invoking callback on pg_notify/0000
LOCATION:  SlruScanDirectory, slru.c:1312
DEBUG:  00000: removing file "pg_notify/0000"
LOCATION:  SlruScanDirCbDeleteAll, slru.c:1277
DEBUG:  00000: max_safe_fds = 984, usable_fds = 1000, already_open = 6
LOCATION:  set_max_safe_fds, fd.c:548
LOG:  00000: redirecting log output to logging collector process
HINT:  Future log output will appear in directory "pg_log".
LOCATION:  SysLogger_Start, syslogger.c:649
DEBUG:  00000: postgres: PostmasterMain: initial environment dump:
LOCATION:  PostmasterMain, postmaster.c:962
DEBUG:  00000: -----------------------------------------
LOCATION:  PostmasterMain, postmaster.c:964
DEBUG:  00000:  MAIL=/var/mail/postgres
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  SSH_CLIENT=41.218.72.115 33169 59696
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  USER=postgres
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  LANGUAGE=en_ZA:en
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  SHLVL=1
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  HOME=/home/postgres
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  PG_GRANDPARENT_PID=19507
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  LOGNAME=postgres
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  _=/usr/local/bin/pg_ctl
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  PGSYSCONFDIR=/usr/local/pgsql/etc
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  LANG=en_ZA.UTF-8
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  SHELL=/bin/bash
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  PWD=/home/postgres
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  SSH_CONNECTION=41.218.72.115 33169 41.218.72.114 59696
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  PGDATA=/home/postgres/pgxc/nodes/coord
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  LC_COLLATE=en_ZA.UTF-8
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  LC_CTYPE=en_ZA.UTF-8
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  LC_MESSAGES=en_ZA.UTF-8
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  LC_MONETARY=C
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  LC_NUMERIC=C
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000:  LC_TIME=C
LOCATION:  PostmasterMain, postmaster.c:967
DEBUG:  00000: -----------------------------------------
LOCATION:  PostmasterMain, postmaster.c:969
DEBUG:  00000: invoking IpcMemoryCreate(size=148193280)
LOCATION:  CreateSharedMemoryAndSemaphores, ipci.c:149
DEBUG:  00000: SlruScanDirectory invoking callback on pg_notify/0000
LOCATION:  SlruScanDirectory, slru.c:1312
DEBUG:  00000: removing file "pg_notify/0000"
LOCATION:  SlruScanDirCbDeleteAll, slru.c:1277
DEBUG:  00000: max_safe_fds = 984, usable_fds = 1000, already_open = 6
LOCATION:  set_max_safe_fds, fd.c:548
LOG:  00000: redirecting log output to logging collector process
HINT:  Future log output will appear in directory "pg_log".
LOCATION:  SysLogger_Start, syslogger.c:649
DEBUG:  00000: logger shutting down
LOCATION:  SysLoggerMain, syslogger.c:517
DEBUG:  00000: shmem_exit(0): 0 callbacks to make
LOCATION:  shmem_exit, ipc.c:212
DEBUG:  00000: proc_exit(0): 0 callbacks to make
LOCATION:  proc_exit_prepare, ipc.c:184
DEBUG:  00000: exit(0)
LOCATION:  proc_exit, ipc.c:135
DEBUG:  00000: shmem_exit(-1): 0 callbacks to make
LOCATION:  shmem_exit, ipc.c:212
DEBUG:  00000: proc_exit(-1): 0 callbacks to make
LOCATION:  proc_exit_prepare, ipc.c:184
pg_ctl: could not start server
Examine the log output.
Done.
Starting all the datanode masters.
Starting datanode master datanode1.
LOG:  redirecting log output to logging collector process
HINT:  Future log output will appear in directory "pg_log".
Done.
Starting all the datanode slaves.
Starting datanode slave datanode1.
LOG:  redirecting log output to logging collector process
HINT:  Future log output will appear in directory "pg_log".
Done.
PGXC monitor all
Running: gtm master
Running: gtm slave
Running: gtm proxy gtm_pxy1
Running: gtm proxy gtm_pxy2
Running: coordinator master coord1
Not running: coordinator master coord2
Not running: datanode master datanode1
Running: datanode slave datanode1


My DB server has 32 GB RAM and 260 GB of Hard disk and i am using
pgxc-1.2.1. what would be the optimal memory related postgresql.conf
configuration for this server.

Anyone has an idea about this issue ?

-- 
Thanks,
Juned Khan
iNextrix Technologies Pvt Ltd.
www.inextrix.com

Thank you Aaron for the detailed analysis.   As long as the issue is just for XC, we need a fix for it to work correctly regardless the compiler optimization.

Did to locate where such wrong estimation takes place?   And what compilation option did you use?

They are very helpful.

Best;
---
Koichi Suzuki

2014/04/12 11:40、Aaron Jackson <aja...@re...<mailto:aja...@re...>> のメール：

It appears that problem is a compiler optimization issue.  I narrowed the issue down to the loop at the end of the ProcArrayRemove method.  I'm not entirely sure why, but the compiler generated code that evaluates the test block of the loop improperly.  Since changing the compiler options, the problem has been resolved.

Aaron

________________________________
From: Aaron Jackson [aja...@re...<mailto:aja...@re...>]
Sent: Friday, April 11, 2014 1:07 AM
To: pos...@li...<mailto:pos...@li...>
Subject: Re: [Postgres-xc-general] failed to find proc - increasing numProcs

I forgot to mention that if I injected a context switch (sleep(0) did the trick as did an elog statement) during the test in the ProcArrayRemove, that it no longer failed.  Hopefully that will help in understanding the reasons why that may have triggered the ProcArrayRemove to succeed.
------------------------------------------------------------------------------
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration
Continuously Automate Build, Test & Deployment
Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees_______________________________________________
Postgres-xc-general mailing list
Pos...@li...
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general