postgres-xc-general Mailing List for Postgres-XC

Hi,
I am setting up a High Availability (HA) architecture and I use PG-XC. 
Thank you very much for your work, it works fine and you are the only 
one active project for synchronous multi master replication.

I don't understand why GTM proxies and GTM standby don't 
reconnect/promote themselves in case of GTM crash.
At the moment, I use shell script to do this but I am not enthusiast 
with this because of maintenance trouble in future.
Is there any reason of this lack of automation ?

Still about HA feature, I realized that when all cluster start, if GTM 
doesn't work neither does GTM standby. Obviously database doesn't work 
in that case.
I also made a script to handle this situation and change GTM standby 
config in order to transform it in GTM before starting. (Not using 
promote command, only changing configuration file). Maybe I missed 
something to make it work ? If not, could it be a new feature for future 
release ?

You can find my script on my puppet module for postgres-xc : 
https://forge.puppetlabs.com/echoes/postgres_xc or 
https://github.com/echoes-tech/puppet-postgres_xc.

Best regards,
Thibault Marquand

I’m afraid you need to share how to reproduce the problem to fix the issue.    Could you share the following?

1) Cluster configuration, if possible, pgxc_ctl configuration file.
2) DDL and DML you used to reproduce the issue.

The addition you made is outside the loop of the issue.   Internally, we had similar problem between O2 and non-O2 optimization, which was caused by missing memory initialization.   O2 or higher optimization requires more strict memory initialization.

Because there are so many candidate of the cause, I believe it is important to share how to reproduce the issue.

Thank you .
---
Koichi Suzuki

2014/04/26 15:06、Aaron Jackson <aja...@re...<mailto:aja...@re...>> のメール：

I honestly have no idea what the optimizer is doing, however, I have isolated the behavior down to a simple change that eliminates the problem for the -O2 optimization.

        for (index = 0; index < arrayP->numProcs; index++)
        {
                if (arrayP->pgprocnos[index] == proc->pgprocno)
                {
                        /* Keep the PGPROC array sorted. See notes above */
                        memmove(&arrayP->pgprocnos[index], &arrayP->pgprocnos[i$
                                        (arrayP->numProcs - index - 1) * sizeof$
                        arrayP->pgprocnos[arrayP->numProcs - 1] = -1;          $
                        arrayP->numProcs--;
                        LWLockRelease(ProcArrayLock);
                        return;
                }
        }

        /* Ooops */
        LWLockRelease(ProcArrayLock);

        elog(LOG, "ProcArrayRemove(post-test): %p", &index);
        elog(LOG, "failed to find proc %p in ProcArray", proc);
}

The *only* change I made is to log the pointer to the index after the loop.  I tried many things, but it was a necessity to do an operation that forced the evaluation of index's address.

Hope this helps,

Aaron

________________________________
From: Aaron Jackson [aja...@re...<mailto:aja...@re...>]
Sent: Friday, April 25, 2014 4:26 PM
To: pos...@li...<mailto:pos...@li...>
Subject: Re: [Postgres-xc-general] failed to find proc - increasing numProcs

It's quite possible I'm missing something obvious, but here is how I've modified procarray.c - the idea was to capture the values that were failing to understand why it was failing.

void
ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
{
        ProcArrayStruct *arrayP = procArray;
        int                     index;
        int    _xNumProcs;
        int    _xIndex;
....
        for (index = 0; (_xIndex = index) < (_xNumProcs = arrayP->numProcs); index++)
        {
                if (arrayP->pgprocnos[index] == proc->pgprocno)
                {
                        /* Keep the PGPROC array sorted. See notes above */
                        memmove(&arrayP->pgprocnos[index], &arrayP->pgprocnos[index + 1],
                                        (arrayP->numProcs - index - 1) * sizeof(int));
                        arrayP->pgprocnos[arrayP->numProcs - 1] = -1;           /* for debugging */
                        arrayP->numProcs--;
                        LWLockRelease(ProcArrayLock);
                        return;
                }
        }

        /* Ooops */
        LWLockRelease(ProcArrayLock);
        elog(LOG, "ProcArrayRemove(post-test): %d | %d | %d | %d", _xIndex, _xNumProcs, arrayP->numProcs, _xIndex < _xNumProcs);

        elog(LOG, "failed to find proc %p in ProcArray", proc);
}

With CFLAGS="" this works as expected.  Once I set CFLAGS="-O2" (or anything else similar) it falls apart.  For example, the fall through case triggered and it showed the following ...

ProcArrayRemove(post-test): 1 | 9 | 9 | 1

Which means the loop test should have succeeded.  I could take this one step further and cache the result of the for loop, however, I can tell you from prior experience, _xIndex < _xNumProcs evaluated as FALSE.  Really not sure what the compiler is doing to draw that conclusion from 1 < 9.

Aaron

________________________________
From: Aaron Jackson [aja...@re...<mailto:aja...@re...>]
Sent: Friday, April 25, 2014 3:05 PM
To: 鈴木 幸市
Cc: pos...@li...<mailto:pos...@li...>
Subject: Re: [Postgres-xc-general] failed to find proc - increasing numProcs

CFLAGS="-O2"
gcc version 4.8.1 (Ubuntu/Linaro 4.8.1-10ubuntu9)

The failed evaluation occurs on line 421 of backend/storage/ipc/procarray.c

The test portion of the clause fails.  I'm not entirely sure why gcc specifically fails, but if I were taking an educated guess, it would be that arrayP->numProcs was volatile and the resultant value of the test was optimized and cached.  I've used several techniques (none of which I like) to fool gcc into believing the value is volatile and discarding the value of arrayP->numProcs.  It concerns me more because the ProcArrayLock should be locked during this sequence.

Aaron

________________________________
From: 鈴木 幸市 [ko...@in...<mailto:ko...@in...>]
Sent: Sunday, April 13, 2014 7:55 PM
To: Aaron Jackson
Cc: pos...@li...<mailto:pos...@li...>
Subject: Re: [Postgres-xc-general] failed to find proc - increasing numProcs

Thank you Aaron for the detailed analysis.   As long as the issue is just for XC, we need a fix for it to work correctly regardless the compiler optimization.

Did to locate where such wrong estimation takes place?   And what compilation option did you use?

They are very helpful.

Best;
---
Koichi Suzuki

2014/04/12 11:40、Aaron Jackson <aja...@re...<mailto:aja...@re...>> のメール：

It appears that problem is a compiler optimization issue.  I narrowed the issue down to the loop at the end of the ProcArrayRemove method.  I'm not entirely sure why, but the compiler generated code that evaluates the test block of the loop improperly.  Since changing the compiler options, the problem has been resolved.

Aaron

________________________________
From: Aaron Jackson [aja...@re...<mailto:aja...@re...>]
Sent: Friday, April 11, 2014 1:07 AM
To: pos...@li...<mailto:pos...@li...>
Subject: Re: [Postgres-xc-general] failed to find proc - increasing numProcs

I forgot to mention that if I injected a context switch (sleep(0) did the trick as did an elog statement) during the test in the ProcArrayRemove, that it no longer failed.  Hopefully that will help in understanding the reasons why that may have triggered the ProcArrayRemove to succeed.
------------------------------------------------------------------------------
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration
Continuously Automate Build, Test & Deployment
Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees_______________________________________________
Postgres-xc-general mailing list
Pos...@li...<mailto:Pos...@li...>
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

------------------------------------------------------------------------------
Start Your Social Network Today - Download eXo Platform
Build your Enterprise Intranet with eXo Platform Software
Java Based Open Source Intranet - Social, Extensible, Cloud Ready
Get Started Now And Turn Your Intranet Into A Collaboration Platform
http://p.sf.net/sfu/ExoPlatform_______________________________________________
Postgres-xc-general mailing list
Pos...@li...
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

When I load data into my table "detail" with COPY, the table loads at a rate of about 56k rows per second.  The data is distributed on a key to achieve this rate of insert (width is 678).  However, when I do the following:

INSERT INTO DETAIL SELECT 123 as Id, ... FROM DETAIL WHERE Id = 500;

I see the write performance drop to only 2.5K rows per second.  The total data set loaded from Id = 500 is 200k rows and takes about 7s to load into the data coordinator.  So, I can attribute almost all of the time (about 80 seconds) directly to the insert.

Insert on detail  (cost=0.00..10.00 rows=1000 width=678) (actual time=79438.038..79438.038 rows=0 loops=1)
   Node/s: node_pgs01_1, node_pgs01_2, node_pgs02_1, node_pgs02_2
   Node expr: productid
   ->  Data Node Scan on detail "_REMOTE_TABLE_QUERY_"  (cost=0.00..10.00 rows=1000 width=678) (actual time=3.917..2147.231 rows=200000 loops=1)
         Node/s: node_pgs01_1, node_pgs01_2, node_pgs02_1, node_pgs02_2

IMO, it seems like an insert like this should approach the performance of a COPY.  Am I missing something or can you recommend a different approach?

Your configuration is wrong.    Gtm_host and gtm_port in
gtm_proxy.conf should be the same as host and port in pgtm.conf.

Because there's too may pitfalls like this in configuring PGXC
manually, I strongly advise to begin with pgxc_ctl, which does all
these for your behalf unless you'd like to lean more in deep of PGXC
architecture and configuration.

Regards;
---
Koichi Suzuki


2014-04-25 15:55 GMT+09:00 张紫宇 <zha...@gm...>:
> I installed pgxc 1.2.1 and I wanted to start gtm and gtm_proxy on the same
> sever, but gtm_proxy didn't work.
>
> I did it as such:
> sudo su
> mkdir /usr/local/pgsql/data_gtm
> mkdir /usr/local/pgsql/data_gtm_proxy
> chown l /usr/local/pgsql/data_gtm
> chown l /usr/local/pgsql/data_gtm_proxy
> su l
> initgtm -Z gtm -D /usr/local/pgsql/data_gtm
> initgtm -Z gtm_proxy -D /usr/local/pgsql/data_gtm_proxy
> gtm -D /usr/local/pgsql/data_gtm &
> gtm_proxy -D /usr/local/pgsql/data_gtm_proxy
>
> On the last step,it returns CST -FATAL:  can not connect to GTM
> LOCATION:  ConnectGTM, proxy_main.c:3344.
> I followed it in the gtm_proxy,I found errorno 111 which means Connection
> refused in
> function connectFailureMessage
> in GTMPQconnectPoll
> in connectGTMStart
> in PQconnectGTMStart
> in PQconnectGTM
> in ConnectGTM
> in RegisterProxy
> in BaseInit
> in main.
>
> My os is ubuntu 12.04 amd64 and I also tested it on centos 6.I also
> installed pgxc 1.2.1 on both of them. But they all get the same error.I
> found a mail
> "http://sourceforge.net/p/postgres-xc/mailman/message/30755193/", we are
> exactly the same.
>
> I followed and tried each every pages on net, but still can't solve it. Can
> you please tell me what I can do? Any help here would be really appreciated.
>
> gtm.conf and gtm_proxy.conf come as follow:
>
>
>
>
> gtm.conf:
> # ----------------------
> # GTM configuration file
> # ----------------------
> #
> # This file must be placed on gtm working directory
> # specified by -D command line option of gtm or gtm_ctl.  The
> # configuration file name must be "gtm.conf"
> #
> #
> # This file consists of lines of the form
> #
> #  name = value
> #
> # (The "=" is optional.) Whitespace may be used.   Comments are
> # introduced with "#" anywhere on a line.  The complete list of
> # parameter names and allowed values can be found in the
> # Postgres-XC documentation.
> #
> # The commented-out settings shown in this file represent the default
> # values.
> #
> # Re-commenting a setting is NOT sufficient to revert it to the default
> # value.
> #
> # You need to restart the server.
>
> #------------------------------------------------------------------------------
> # GENERAL PARAMETERS
> #------------------------------------------------------------------------------
> nodename = 'one' # Specifies the node name.
> # (changes requires restart)
> #listen_addresses = '*' # Listen addresses of this GTM.
> # (changes requires restart)
> port = 6666 # Port number of this GTM.
> # (changes requires restart)
>
> #startup = ACT # Start mode. ACT/STANDBY.
>
> #------------------------------------------------------------------------------
> # GTM STANDBY PARAMETERS
> #------------------------------------------------------------------------------
> #Those parameters are effective when GTM is activated as a standby server
> #active_host = '' # Listen address of active GTM.
> # (changes requires restart)
> #active_port = # Port number of active GTM.
> # (changes requires restart)
>
> #---------------------------------------
> # OTHER OPTIONS
> #---------------------------------------
> #keepalives_idle = 0 # Keepalives_idle parameter.
> #keepalives_interval = 0 # Keepalives_interval parameter.
> #keepalives_count = 0 # Keepalives_count internal parameter.
> #log_file = 'gtm.log' # Log file name
> #log_min_messages = WARNING # log_min_messages.  Default WARNING.
>   # Valid value: DEBUG, DEBUG5, DEBUG4, DEBUG3,
> # DEBUG2, DEBUG1, INFO, NOTICE, WARNING,
> # ERROR, LOG, FATAL, PANIC
> #synchronous_backup = off # If backup to standby is synchronous
>
>
>
>
>
>
>
> gtm_proxy.conf:
>
> #-----------------------------
> # GTM Proxy configuration file
> #-----------------------------
> #
> # This file must be placed on gtm working directory
> # specified by -D command line option of gtm_proxy or gtm_ctl.
> # The configuration file name must be "gtm_proxy.conf"
> #
> #
> # This file consists of lines of the form
> #
> #  name = value
> #
> # (The "=" is optional.) Whitespace may be used.   Comments are
> # introduced with "#" anywhere on a line.  The complete list of
> # parameter names and allowed values can be found in the
> # Postgres-XC documentation.
> #
> # The commented-out settings shown in this file represent the default
> # values.
> #
> # Re-commenting a setting is NOT sufficient to revert it to the default
> # value.
> #
> # You need to restart the server.
>
> #------------------------------------------------------------------------------
> # GENERAL PARAMETERS
> #------------------------------------------------------------------------------
> nodename = 'one' # Specifies the node name.
> # (changes requires restart)
> #listen_addresses = '*' # Listen addresses of this GTM.
> # (changes requires restart)
> port = 6666 # Port number of this GTM.
> # (changes requires restart)
>
> #------------------------------------------------------------------------------
> # GTM PROXY PARAMETERS
> #------------------------------------------------------------------------------
> #worker_threads = 1 # Number of the worker thread of this
> # GTM proxy
> # (changes requires restart)
>
> #------------------------------------------------------------------------------
> # GTM CONNECTION PARAMETERS
> #------------------------------------------------------------------------------
> # Those parameters are used to connect to a GTM server
> gtm_host = 'localhost' # Listen address of the active GTM.
> # (changes requires restart)
> gtm_port = 6668 # Port number of the active GTM.
> # (changes requires restart)
>
> #------------------------------------------------------------------------------
> # Behavior at GTM communication error
> #------------------------------------------------------------------------------
> #gtm_connect_retry_interval = 0 # How long (in secs) to wait until the next
> # retry to connect to GTM.
> #
> #
> #------------------------------------------------------------------------------
> # Other options
> #------------------------------------------------------------------------------
> #keepalives_idle = 0 # Keepalives_idle parameter.
> #keepalives_interval = 0 # Keepalives_interval parameter.
> #keepalives_count = 0 # Keepalives_count internal parameter.
> #log_file = 'gtm_proxy.log' # Log file name
> #log_min_messages = WARNING # log_min_messages.  Default WARNING.
>   # Valid value: DEBUG, DEBUG5, DEBUG4, DEBUG3,
> # DEBUG2, DEBUG1, INFO, NOTICE, WARNING,
> # ERROR, LOG, FATAL, PANIC.
>
>
>
> --Ronian
>
> ------------------------------------------------------------------------------
> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> Instantly run your Selenium tests across 300+ browser/OS combos.  Get
> unparalleled scalability from the best Selenium testing platform available.
> Simple to use. Nothing to install. Get started now for free."
> http://p.sf.net/sfu/SauceLabs
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>

That was my thinking as well.  I indirectly executed the queries independently.  Basically, each query takes about 5-10 seconds each for 4 data nodes - specifically, I used psql -h <datanode> -p <port> - to time the individual data node performance individually.  So you figure, worst case = 10 seconds x 4 nodes = 40 seconds of aggregate time on a serial request.  But I'm seeing 65 seconds which means there's some other overhead that I'm missing.

The 65 second aggregate is also the reason why I asked if the requests were parallel or serial because it *feels* serial though it could be other factors.

I'll retest and update.

________________________________
From: Ashutosh Bapat [ash...@en...]
Sent: Tuesday, April 29, 2014 1:05 AM
To: Aaron Jackson
Cc: amul sul; pos...@li...
Subject: Re: [Postgres-xc-general] Data Node Scan Performance

Hi Aaron,
Can you please take the timing of executing "EXECUTE DIRECT <query to the datanode>" to some datanode. I suspect that the delay you are seeing is added by the sheer communication between coord and datanode. Some of that would be libpq overhead and some of it will be network overhead.


On Tue, Apr 29, 2014 at 10:58 AM, Aaron Jackson <aja...@re...<mailto:aja...@re...>> wrote:
Interesting,

So, I wonder why I am seeing query times that are more than the sum of the total times required to perform the process without the coordinator.  For example, let's say the query was 'SELECT 500 as Id, Foo, Bar from MyTable WHERE Id = 186' - I could perform this query at all 4 nodes and they would take no more than 10 seconds to run individually.  However, when performed against the coordinator, this same query takes 65 seconds.  That's more than the total aggregate of all data nodes.

Any thoughts - is it completely attributed to the coordinator?


________________________________________
From: amul sul [sul...@ya...<mailto:sul...@ya...>]
Sent: Tuesday, April 29, 2014 12:23 AM
To: Aaron Jackson; pos...@li...<mailto:pos...@li...>
Subject: Re: [Postgres-xc-general] Data Node Scan Performance

>On Tuesday, 29 April 2014 10:38 AM, Aaron Jackson <aja...@re...<mailto:aja...@re...>> wrote:
> my question is, does the coordinator execute the data node scan serially
>or in parallel - and if it's serially,
>is there any thought around how to make it parallel?

IMO, scan on data happens independently i.e parallel, the scan result is collected at co-ordinator and returned to client.

Referring distributed table using other than distribution key(in your case it Q instead of k), has little penalty.

Regards,
Amul Sul

------------------------------------------------------------------------------
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.  Get
unparalleled scalability from the best Selenium testing platform available.
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs
_______________________________________________
Postgres-xc-general mailing list
Pos...@li...<mailto:Pos...@li...>
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general



--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Hi Aaron,
Can you please take the timing of executing "EXECUTE DIRECT <query to the
datanode>" to some datanode. I suspect that the delay you are seeing is
added by the sheer communication between coord and datanode. Some of that
would be libpq overhead and some of it will be network overhead.


On Tue, Apr 29, 2014 at 10:58 AM, Aaron Jackson <aja...@re...>wrote:

> Interesting,
>
> So, I wonder why I am seeing query times that are more than the sum of the
> total times required to perform the process without the coordinator.  For
> example, let's say the query was 'SELECT 500 as Id, Foo, Bar from MyTable
> WHERE Id = 186' - I could perform this query at all 4 nodes and they would
> take no more than 10 seconds to run individually.  However, when performed
> against the coordinator, this same query takes 65 seconds.  That's more
> than the total aggregate of all data nodes.
>
> Any thoughts - is it completely attributed to the coordinator?
>
>
> ________________________________________
> From: amul sul [sul...@ya...]
> Sent: Tuesday, April 29, 2014 12:23 AM
> To: Aaron Jackson; pos...@li...
> Subject: Re: [Postgres-xc-general] Data Node Scan Performance
>
> >On Tuesday, 29 April 2014 10:38 AM, Aaron Jackson <aja...@re...>
> wrote:
> > my question is, does the coordinator execute the data node scan serially
> >or in parallel - and if it's serially,
> >is there any thought around how to make it parallel?
>
> IMO, scan on data happens independently i.e parallel, the scan result is
> collected at co-ordinator and returned to client.
>
> Referring distributed table using other than distribution key(in your case
> it Q instead of k), has little penalty.
>
> Regards,
> Amul Sul
>
>
> ------------------------------------------------------------------------------
> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> Instantly run your Selenium tests across 300+ browser/OS combos.  Get
> unparalleled scalability from the best Selenium testing platform available.
> Simple to use. Nothing to install. Get started now for free."
> http://p.sf.net/sfu/SauceLabs
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>



-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

On Tuesday, 29 April 2014 10:58 AM, Aaron Jackson <aja...@re...> wrote: 
Interesting, 

>Any thoughts - is it completely attributed to the coordinator? 
I am not sure, but in your example, MyTable is distributed on foo and search on ID, if somehow you able to add distribution key search in WHERE condition (eg. ....WHERE Id = 186 AND Foo=xyz ), planner would get help to locate candidate tuple easily. 
Current situation it scanning all the datanodes & combine result at coordinator. 
Dose ID is unique in MyTable ? if so can't you can distribute on ID? 
Regards, 
Amul Sul

Interesting,

So, I wonder why I am seeing query times that are more than the sum of the total times required to perform the process without the coordinator.  For example, let's say the query was 'SELECT 500 as Id, Foo, Bar from MyTable WHERE Id = 186' - I could perform this query at all 4 nodes and they would take no more than 10 seconds to run individually.  However, when performed against the coordinator, this same query takes 65 seconds.  That's more than the total aggregate of all data nodes.

Any thoughts - is it completely attributed to the coordinator?

 
________________________________________
From: amul sul [sul...@ya...]
Sent: Tuesday, April 29, 2014 12:23 AM
To: Aaron Jackson; pos...@li...
Subject: Re: [Postgres-xc-general] Data Node Scan Performance

>On Tuesday, 29 April 2014 10:38 AM, Aaron Jackson <aja...@re...> wrote:
> my question is, does the coordinator execute the data node scan serially
>or in parallel - and if it's serially,
>is there any thought around how to make it parallel?

IMO, scan on data happens independently i.e parallel, the scan result is collected at co-ordinator and returned to client.

Referring distributed table using other than distribution key(in your case it Q instead of k), has little penalty.

Regards,
Amul Sul

>On Tuesday, 29 April 2014 10:38 AM, Aaron Jackson <aja...@re...> wrote:
> my question is, does the coordinator execute the data node scan serially 
>or in parallel - and if it's serially, 
>is there any thought around how to make it parallel?

IMO, scan on data happens independently i.e parallel, the scan result is collected at co-ordinator and returned to client.

Referring distributed table using other than distribution key(in your case it Q instead of k), has little penalty. 

Regards,
Amul Sul

Also, if it helps, this is actually an operational query, not a data warehouse type query.  In this case, we have an object that owns a considerable amount of data below it (in this case, 1:N:M relationships where rows in table A have children in table B and grandchildren in table C).  The operation I want to perform is a scalable clone of the data.  The actually query looks more like the following:

INSERT INTO MyTable SELECT 500 as Id, Foo, Bar from MyTable WHERE Id = $1

For testing purposes (I'm simply tinkering), the data is distributed on Foo and indexed on Id.  So in this case I'm copying the "values" from the old entity to the new entity.  The plan appears to require the data to be retrieved into the coordinator and then dispersed back down into the data nodes.

An optimization for this specific problem would be to push the 'INSERT INTO ... SELECT' directly down into the data nodes since there isn't any inherent benefit to the coordinator consuming the data.  The distribution key is identical - so the data will be sent right back to the data node it came from.

Any other thoughts on how to make this performant - if not, I'll go back to my tinkering table.

________________________________
From: Aaron Jackson [aja...@re...]
Sent: Tuesday, April 29, 2014 12:06 AM
To: pos...@li...
Subject: [Postgres-xc-general] Data Node Scan Performance

I have a table that I've distributed by some key K.  When I want to query by some other dimension Q, the coordinator explain plan indicates that it does a Data Node Scan on *table* "_REMOTE_TABLE_QUERY"

Now what I've noticed is that if I have 4 nodes, the coordinator based scan may take 65 seconds, however, the individual date nodes usually finish within 5-10 seconds.  The individual explain plains from each data node reveal nothing.  So my question is, does the coordinator execute the data node scan serially or in parallel - and if it's serially, is there any thought around how to make it parallel?

In the event it is already parallel, is the time differential I'm seeing simply attributed to the coordinator gathering results in preparation to return to the requesting client?

Thanks

I have a table that I've distributed by some key K.  When I want to query by some other dimension Q, the coordinator explain plan indicates that it does a Data Node Scan on *table* "_REMOTE_TABLE_QUERY"

Now what I've noticed is that if I have 4 nodes, the coordinator based scan may take 65 seconds, however, the individual date nodes usually finish within 5-10 seconds.  The individual explain plains from each data node reveal nothing.  So my question is, does the coordinator execute the data node scan serially or in parallel - and if it's serially, is there any thought around how to make it parallel?

In the event it is already parallel, is the time differential I'm seeing simply attributed to the coordinator gathering results in preparation to return to the requesting client?

Thanks

-------- Original Message --------
Subject: 	GTM Proxy can't start
Date: 	Fri, 25 Apr 2014 14:55:01 +0800
From: 	张紫宇 <zha...@gm...>
To: 	pos...@li...



I installed pgxc 1.2.1 and I wanted to start gtm and gtm_proxy on the 
same sever, but gtm_proxy didn't work.

I did it as such:
sudo su
mkdir /usr/local/pgsql/data_gtm
mkdir /usr/local/pgsql/data_gtm_proxy
chown l /usr/local/pgsql/data_gtm
chown l /usr/local/pgsql/data_gtm_proxy
su l
initgtm -Z gtm -D /usr/local/pgsql/data_gtm
initgtm -Z gtm_proxy -D /usr/local/pgsql/data_gtm_proxy
gtm -D /usr/local/pgsql/data_gtm &
gtm_proxy -D /usr/local/pgsql/data_gtm_proxy

On the last step,it returns CST -FATAL:  can not connect to GTM
LOCATION:  ConnectGTM, proxy_main.c:3344.
I followed it in the gtm_proxy,I found errorno 111 which means 
Connection refused in
function connectFailureMessage
in GTMPQconnectPoll
in connectGTMStart
in PQconnectGTMStart
in PQconnectGTM
in ConnectGTM
in RegisterProxy
in BaseInit
in main.

My os is ubuntu 12.04 amd64 and I also tested it on centos 6.I also 
installed pgxc 1.2.1 on both of them. But they all get the same error.I 
found a mail 
"http://sourceforge.net/p/postgres-xc/mailman/message/30755193/", we are 
exactly the same.

I followed and tried each every pages on net, but still can't solve it. 
Can you please tell me what I can do? Any help here would be really 
appreciated.

gtm.conf and gtm_proxy.conf come as follow:




gtm.conf:
# ----------------------
# GTM configuration file
# ----------------------
#
# This file must be placed on gtm working directory
# specified by -D command line option of gtm or gtm_ctl.  The
# configuration file name must be "gtm.conf"
#
#
# This file consists of lines of the form
#
#  name = value
#
# (The "=" is optional.) Whitespace may be used.   Comments are
# introduced with "#" anywhere on a line.  The complete list of
# parameter names and allowed values can be found in the
# Postgres-XC documentation.
#
# The commented-out settings shown in this file represent the default
# values.
#
# Re-commenting a setting is NOT sufficient to revert it to the default
# value.
#
# You need to restart the server.

#------------------------------------------------------------------------------
# GENERAL PARAMETERS
#------------------------------------------------------------------------------
nodename = 'one'# Specifies the node name.
# (changes requires restart)
#listen_addresses = '*'# Listen addresses of this GTM.
# (changes requires restart)
port = 6666# Port number of this GTM.
# (changes requires restart)

#startup = ACT# Start mode. ACT/STANDBY.

#------------------------------------------------------------------------------
# GTM STANDBY PARAMETERS
#------------------------------------------------------------------------------
#Those parameters are effective when GTM is activated as a standby server
#active_host = ''# Listen address of active GTM.
# (changes requires restart)
#active_port =# Port number of active GTM.
# (changes requires restart)

#---------------------------------------
# OTHER OPTIONS
#---------------------------------------
#keepalives_idle = 0# Keepalives_idle parameter.
#keepalives_interval = 0# Keepalives_interval parameter.
#keepalives_count = 0# Keepalives_count internal parameter.
#log_file = 'gtm.log'# Log file name
#log_min_messages = WARNING# log_min_messages.  Default WARNING.
# Valid value: DEBUG, DEBUG5, DEBUG4, DEBUG3,
# DEBUG2, DEBUG1, INFO, NOTICE, WARNING,
# ERROR, LOG, FATAL, PANIC
#synchronous_backup = off# If backup to standby is synchronous







gtm_proxy.conf:

#-----------------------------
# GTM Proxy configuration file
#-----------------------------
#
# This file must be placed on gtm working directory
# specified by -D command line option of gtm_proxy or gtm_ctl.
# The configuration file name must be "gtm_proxy.conf"
#
#
# This file consists of lines of the form
#
#  name = value
#
# (The "=" is optional.) Whitespace may be used.   Comments are
# introduced with "#" anywhere on a line.  The complete list of
# parameter names and allowed values can be found in the
# Postgres-XC documentation.
#
# The commented-out settings shown in this file represent the default
# values.
#
# Re-commenting a setting is NOT sufficient to revert it to the default
# value.
#
# You need to restart the server.

#------------------------------------------------------------------------------
# GENERAL PARAMETERS
#------------------------------------------------------------------------------
nodename = 'one'# Specifies the node name.
# (changes requires restart)
#listen_addresses = '*'# Listen addresses of this GTM.
# (changes requires restart)
port = 6666# Port number of this GTM.
# (changes requires restart)

#------------------------------------------------------------------------------
# GTM PROXY PARAMETERS
#------------------------------------------------------------------------------
#worker_threads = 1# Number of the worker thread of this
# GTM proxy
# (changes requires restart)

#------------------------------------------------------------------------------
# GTM CONNECTION PARAMETERS
#------------------------------------------------------------------------------
# Those parameters are used to connect to a GTM server
gtm_host = 'localhost' # Listen address of the active GTM.
# (changes requires restart)
gtm_port = 6668 # Port number of the active GTM.
# (changes requires restart)

#------------------------------------------------------------------------------
# Behavior at GTM communication error
#------------------------------------------------------------------------------
#gtm_connect_retry_interval = 0# How long (in secs) to wait until the next
# retry to connect to GTM.
#
#
#------------------------------------------------------------------------------
# Other options
#------------------------------------------------------------------------------
#keepalives_idle = 0# Keepalives_idle parameter.
#keepalives_interval = 0# Keepalives_interval parameter.
#keepalives_count = 0# Keepalives_count internal parameter.
#log_file = 'gtm_proxy.log'# Log file name
#log_min_messages = WARNING# log_min_messages.  Default WARNING.
# Valid value: DEBUG, DEBUG5, DEBUG4, DEBUG3,
# DEBUG2, DEBUG1, INFO, NOTICE, WARNING,
# ERROR, LOG, FATAL, PANIC.



--Ronian

in logs i found this
DETAIL:  The failed archive command was: rsync
pg_xlog/000000010000001C0000006A postgres@db01
:/home/postgres/pgxc/nodes/datanode_archlog/000000010000001C0000006A
rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]:
Broken pipe (32)
rsync: write failed on
"/home/postgres/pgxc/nodes/datanode_archlog/000000010000001C0000006A": No
space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(322) [receiver=3.0.9]
rsync: connection unexpectedly closed (28 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605)
[sender=3.0.9]
LOG:  archive command failed with exit code 12



On Mon, Apr 28, 2014 at 4:51 PM, Juned Khan <jkh...@gm...> wrote:

> Hi all,
>
> I setup pgxc, but today when i tried fire one manual query i got below
> message. i am not able to show table descriptions as well.
>
> database=# \d
> ERROR:  could not access status of transaction 0
> DETAIL:  Could not write to file "pg_clog/000B" at offset 40960: No space
> left on device.
>
> sometimes its showing table description in below format.
>
> database=# \dt
> <table border="1">
>   <caption>List of relations</caption>
>   <tr>
>     <th align="center">Schema</th>
>     <th align="center">Name</th>
>     <th align="center">Type</th>
>     <th align="center">Owner</th>
>   </tr>
>
>
> --
> Thanks,
> Juned Khan
>
> <http://www.inextrix.com/>
>



-- 
Thanks,
Juned Khan
iNextrix Technologies Pvt Ltd.
www.inextrix.com

Hi all,

I setup pgxc, but today when i tried fire one manual query i got below
message. i am not able to show table descriptions as well.

database=# \d
ERROR:  could not access status of transaction 0
DETAIL:  Could not write to file "pg_clog/000B" at offset 40960: No space
left on device.

sometimes its showing table description in below format.

database=# \dt
<table border="1">
  <caption>List of relations</caption>
  <tr>
    <th align="center">Schema</th>
    <th align="center">Name</th>
    <th align="center">Type</th>
    <th align="center">Owner</th>
  </tr>


-- 
Thanks,
Juned Khan

<http://www.inextrix.com/>

I honestly have no idea what the optimizer is doing, however, I have isolated the behavior down to a simple change that eliminates the problem for the -O2 optimization.

        for (index = 0; index < arrayP->numProcs; index++)
        {
                if (arrayP->pgprocnos[index] == proc->pgprocno)
                {
                        /* Keep the PGPROC array sorted. See notes above */
                        memmove(&arrayP->pgprocnos[index], &arrayP->pgprocnos[i$
                                        (arrayP->numProcs - index - 1) * sizeof$
                        arrayP->pgprocnos[arrayP->numProcs - 1] = -1;          $
                        arrayP->numProcs--;
                        LWLockRelease(ProcArrayLock);
                        return;
                }
        }

        /* Ooops */
        LWLockRelease(ProcArrayLock);

        elog(LOG, "ProcArrayRemove(post-test): %p", &index);
        elog(LOG, "failed to find proc %p in ProcArray", proc);
}

The *only* change I made is to log the pointer to the index after the loop.  I tried many things, but it was a necessity to do an operation that forced the evaluation of index's address.

Hope this helps,

Aaron

________________________________
From: Aaron Jackson [aja...@re...]
Sent: Friday, April 25, 2014 4:26 PM
To: pos...@li...
Subject: Re: [Postgres-xc-general] failed to find proc - increasing numProcs

It's quite possible I'm missing something obvious, but here is how I've modified procarray.c - the idea was to capture the values that were failing to understand why it was failing.

void
ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
{
        ProcArrayStruct *arrayP = procArray;
        int                     index;
        int    _xNumProcs;
        int    _xIndex;
...
        for (index = 0; (_xIndex = index) < (_xNumProcs = arrayP->numProcs); index++)
        {
                if (arrayP->pgprocnos[index] == proc->pgprocno)
                {
                        /* Keep the PGPROC array sorted. See notes above */
                        memmove(&arrayP->pgprocnos[index], &arrayP->pgprocnos[index + 1],
                                        (arrayP->numProcs - index - 1) * sizeof(int));
                        arrayP->pgprocnos[arrayP->numProcs - 1] = -1;           /* for debugging */
                        arrayP->numProcs--;
                        LWLockRelease(ProcArrayLock);
                        return;
                }
        }

        /* Ooops */
        LWLockRelease(ProcArrayLock);
        elog(LOG, "ProcArrayRemove(post-test): %d | %d | %d | %d", _xIndex, _xNumProcs, arrayP->numProcs, _xIndex < _xNumProcs);

        elog(LOG, "failed to find proc %p in ProcArray", proc);
}

With CFLAGS="" this works as expected.  Once I set CFLAGS="-O2" (or anything else similar) it falls apart.  For example, the fall through case triggered and it showed the following ...

ProcArrayRemove(post-test): 1 | 9 | 9 | 1

Which means the loop test should have succeeded.  I could take this one step further and cache the result of the for loop, however, I can tell you from prior experience, _xIndex < _xNumProcs evaluated as FALSE.  Really not sure what the compiler is doing to draw that conclusion from 1 < 9.

Aaron

________________________________
From: Aaron Jackson [aja...@re...]
Sent: Friday, April 25, 2014 3:05 PM
To: 鈴木 幸市
Cc: pos...@li...
Subject: Re: [Postgres-xc-general] failed to find proc - increasing numProcs

CFLAGS="-O2"
gcc version 4.8.1 (Ubuntu/Linaro 4.8.1-10ubuntu9)

The failed evaluation occurs on line 421 of backend/storage/ipc/procarray.c

The test portion of the clause fails.  I'm not entirely sure why gcc specifically fails, but if I were taking an educated guess, it would be that arrayP->numProcs was volatile and the resultant value of the test was optimized and cached.  I've used several techniques (none of which I like) to fool gcc into believing the value is volatile and discarding the value of arrayP->numProcs.  It concerns me more because the ProcArrayLock should be locked during this sequence.

Aaron

________________________________
From: 鈴木 幸市 [ko...@in...]
Sent: Sunday, April 13, 2014 7:55 PM
To: Aaron Jackson
Cc: pos...@li...
Subject: Re: [Postgres-xc-general] failed to find proc - increasing numProcs

Thank you Aaron for the detailed analysis.   As long as the issue is just for XC, we need a fix for it to work correctly regardless the compiler optimization.

Did to locate where such wrong estimation takes place?   And what compilation option did you use?

They are very helpful.

Best;
---
Koichi Suzuki

2014/04/12 11:40、Aaron Jackson <aja...@re...<mailto:aja...@re...>> のメール：

It appears that problem is a compiler optimization issue.  I narrowed the issue down to the loop at the end of the ProcArrayRemove method.  I'm not entirely sure why, but the compiler generated code that evaluates the test block of the loop improperly.  Since changing the compiler options, the problem has been resolved.

Aaron

________________________________
From: Aaron Jackson [aja...@re...<mailto:aja...@re...>]
Sent: Friday, April 11, 2014 1:07 AM
To: pos...@li...<mailto:pos...@li...>
Subject: Re: [Postgres-xc-general] failed to find proc - increasing numProcs

I forgot to mention that if I injected a context switch (sleep(0) did the trick as did an elog statement) during the test in the ProcArrayRemove, that it no longer failed.  Hopefully that will help in understanding the reasons why that may have triggered the ProcArrayRemove to succeed.
------------------------------------------------------------------------------
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration
Continuously Automate Build, Test & Deployment
Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees_______________________________________________
Postgres-xc-general mailing list
Pos...@li...
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

It's quite possible I'm missing something obvious, but here is how I've modified procarray.c - the idea was to capture the values that were failing to understand why it was failing.

void
ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
{
        ProcArrayStruct *arrayP = procArray;
        int                     index;
        int    _xNumProcs;
        int    _xIndex;
...
        for (index = 0; (_xIndex = index) < (_xNumProcs = arrayP->numProcs); index++)
        {
                if (arrayP->pgprocnos[index] == proc->pgprocno)
                {
                        /* Keep the PGPROC array sorted. See notes above */
                        memmove(&arrayP->pgprocnos[index], &arrayP->pgprocnos[index + 1],
                                        (arrayP->numProcs - index - 1) * sizeof(int));
                        arrayP->pgprocnos[arrayP->numProcs - 1] = -1;           /* for debugging */
                        arrayP->numProcs--;
                        LWLockRelease(ProcArrayLock);
                        return;
                }
        }

        /* Ooops */
        LWLockRelease(ProcArrayLock);
        elog(LOG, "ProcArrayRemove(post-test): %d | %d | %d | %d", _xIndex, _xNumProcs, arrayP->numProcs, _xIndex < _xNumProcs);

        elog(LOG, "failed to find proc %p in ProcArray", proc);
}

With CFLAGS="" this works as expected.  Once I set CFLAGS="-O2" (or anything else similar) it falls apart.  For example, the fall through case triggered and it showed the following ...

ProcArrayRemove(post-test): 1 | 9 | 9 | 1

Which means the loop test should have succeeded.  I could take this one step further and cache the result of the for loop, however, I can tell you from prior experience, _xIndex < _xNumProcs evaluated as FALSE.  Really not sure what the compiler is doing to draw that conclusion from 1 < 9.

Aaron

________________________________
From: Aaron Jackson [aja...@re...]
Sent: Friday, April 25, 2014 3:05 PM
To: 鈴木 幸市
Cc: pos...@li...
Subject: Re: [Postgres-xc-general] failed to find proc - increasing numProcs

CFLAGS="-O2"
gcc version 4.8.1 (Ubuntu/Linaro 4.8.1-10ubuntu9)

The failed evaluation occurs on line 421 of backend/storage/ipc/procarray.c

The test portion of the clause fails.  I'm not entirely sure why gcc specifically fails, but if I were taking an educated guess, it would be that arrayP->numProcs was volatile and the resultant value of the test was optimized and cached.  I've used several techniques (none of which I like) to fool gcc into believing the value is volatile and discarding the value of arrayP->numProcs.  It concerns me more because the ProcArrayLock should be locked during this sequence.

Aaron

________________________________
From: 鈴木 幸市 [ko...@in...]
Sent: Sunday, April 13, 2014 7:55 PM
To: Aaron Jackson
Cc: pos...@li...
Subject: Re: [Postgres-xc-general] failed to find proc - increasing numProcs

Thank you Aaron for the detailed analysis.   As long as the issue is just for XC, we need a fix for it to work correctly regardless the compiler optimization.

Did to locate where such wrong estimation takes place?   And what compilation option did you use?

They are very helpful.

Best;
---
Koichi Suzuki

2014/04/12 11:40、Aaron Jackson <aja...@re...<mailto:aja...@re...>> のメール：

It appears that problem is a compiler optimization issue.  I narrowed the issue down to the loop at the end of the ProcArrayRemove method.  I'm not entirely sure why, but the compiler generated code that evaluates the test block of the loop improperly.  Since changing the compiler options, the problem has been resolved.

Aaron

________________________________
From: Aaron Jackson [aja...@re...<mailto:aja...@re...>]
Sent: Friday, April 11, 2014 1:07 AM
To: pos...@li...<mailto:pos...@li...>
Subject: Re: [Postgres-xc-general] failed to find proc - increasing numProcs

I forgot to mention that if I injected a context switch (sleep(0) did the trick as did an elog statement) during the test in the ProcArrayRemove, that it no longer failed.  Hopefully that will help in understanding the reasons why that may have triggered the ProcArrayRemove to succeed.
------------------------------------------------------------------------------
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration
Continuously Automate Build, Test & Deployment
Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees_______________________________________________
Postgres-xc-general mailing list
Pos...@li...
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

CFLAGS="-O2"
gcc version 4.8.1 (Ubuntu/Linaro 4.8.1-10ubuntu9)

The failed evaluation occurs on line 421 of backend/storage/ipc/procarray.c

The test portion of the clause fails.  I'm not entirely sure why gcc specifically fails, but if I were taking an educated guess, it would be that arrayP->numProcs was volatile and the resultant value of the test was optimized and cached.  I've used several techniques (none of which I like) to fool gcc into believing the value is volatile and discarding the value of arrayP->numProcs.  It concerns me more because the ProcArrayLock should be locked during this sequence.

Aaron

________________________________
From: 鈴木 幸市 [ko...@in...]
Sent: Sunday, April 13, 2014 7:55 PM
To: Aaron Jackson
Cc: pos...@li...
Subject: Re: [Postgres-xc-general] failed to find proc - increasing numProcs

Thank you Aaron for the detailed analysis.   As long as the issue is just for XC, we need a fix for it to work correctly regardless the compiler optimization.

Did to locate where such wrong estimation takes place?   And what compilation option did you use?

They are very helpful.

Best;
---
Koichi Suzuki

2014/04/12 11:40、Aaron Jackson <aja...@re...<mailto:aja...@re...>> のメール：

It appears that problem is a compiler optimization issue.  I narrowed the issue down to the loop at the end of the ProcArrayRemove method.  I'm not entirely sure why, but the compiler generated code that evaluates the test block of the loop improperly.  Since changing the compiler options, the problem has been resolved.

Aaron

________________________________
From: Aaron Jackson [aja...@re...<mailto:aja...@re...>]
Sent: Friday, April 11, 2014 1:07 AM
To: pos...@li...<mailto:pos...@li...>
Subject: Re: [Postgres-xc-general] failed to find proc - increasing numProcs

I forgot to mention that if I injected a context switch (sleep(0) did the trick as did an elog statement) during the test in the ProcArrayRemove, that it no longer failed.  Hopefully that will help in understanding the reasons why that may have triggered the ProcArrayRemove to succeed.
------------------------------------------------------------------------------
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration
Continuously Automate Build, Test & Deployment
Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees_______________________________________________
Postgres-xc-general mailing list
Pos...@li...
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

I installed pgxc 1.2.1 and I wanted to start gtm and gtm_proxy on the same sever, but gtm_proxy didn't work.
I did it as such:sudo sumkdir /usr/local/pgsql/data_gtmmkdir /usr/local/pgsql/data_gtm_proxychown l /usr/local/pgsql/data_gtmchown l /usr/local/pgsql/data_gtm_proxysu linitgtm -Z gtm -D /usr/local/pgsql/data_gtminitgtm -Z gtm_proxy -D /usr/local/pgsql/data_gtm_proxygtm -D /usr/local/pgsql/data_gtm &gtm_proxy -D /usr/local/pgsql/data_gtm_proxy
On the last step,it returns CST -FATAL:  can not connect to GTMLOCATION:  ConnectGTM, proxy_main.c:3344.I followed it in the gtm_proxy,I found errorno 111 which means Connection refused infunction connectFailureMessagein GTMPQconnectPollin connectGTMStartin PQconnectGTMStartin PQconnectGTMin ConnectGTMin RegisterProxyin BaseInit in main.
My os is ubuntu 12.04 amd64 and I also tested it on centos 6.I also installed pgxc 1.2.1 on both of them. But they all get the same error.I found a mail "http://sourceforge.net/p/postgres-xc/mailman/message/30755193/", we are exactly the same.
I followed and tried each every pages on net, but still can't solve it. Can you please tell me what I can do? Any help here would be really appreciated.
gtm.conf and gtm_proxy.conf come as follow:



gtm.conf:# ----------------------# GTM configuration file# ----------------------## This file must be placed on gtm working directory # specified by -D command line option of gtm or gtm_ctl.  The# configuration file name must be "gtm.conf"### This file consists of lines of the form##  name = value## (The "=" is optional.) Whitespace may be used.   Comments are# introduced with "#" anywhere on a line.  The complete list of# parameter names and allowed values can be found in the# Postgres-XC documentation.## The commented-out settings shown in this file represent the default# values.## Re-commenting a setting is NOT sufficient to revert it to the default# value.## You need to restart the server.
#------------------------------------------------------------------------------# GENERAL PARAMETERS#------------------------------------------------------------------------------nodename = 'one'				# Specifies the node name.					# (changes requires restart)#listen_addresses = '*'			# Listen addresses of this GTM.					# (changes requires restart)port = 6666				# Port number of this GTM.					# (changes requires restart)
#startup = ACT				# Start mode. ACT/STANDBY.
#------------------------------------------------------------------------------# GTM STANDBY PARAMETERS#------------------------------------------------------------------------------#Those parameters are effective when GTM is activated as a standby server#active_host = ''			# Listen address of active GTM.					# (changes requires restart)#active_port =				# Port number of active GTM.					# (changes requires restart)
#---------------------------------------# OTHER OPTIONS#---------------------------------------#keepalives_idle = 0			# Keepalives_idle parameter.#keepalives_interval = 0		# Keepalives_interval parameter.#keepalives_count = 0			# Keepalives_count internal parameter.#log_file = 'gtm.log'			# Log file name#log_min_messages = WARNING		# log_min_messages.  Default WARNING.				  	# Valid value: DEBUG, DEBUG5, DEBUG4, DEBUG3,					# DEBUG2, DEBUG1, INFO, NOTICE, WARNING,					# ERROR, LOG, FATAL, PANIC#synchronous_backup = off	# If backup to standby is synchronous






gtm_proxy.conf:
#-----------------------------# GTM Proxy configuration file#-----------------------------## This file must be placed on gtm working directory # specified by -D command line option of gtm_proxy or gtm_ctl.# The configuration file name must be "gtm_proxy.conf"### This file consists of lines of the form##  name = value## (The "=" is optional.) Whitespace may be used.   Comments are# introduced with "#" anywhere on a line.  The complete list of# parameter names and allowed values can be found in the# Postgres-XC documentation.## The commented-out settings shown in this file represent the default# values.## Re-commenting a setting is NOT sufficient to revert it to the default# value.## You need to restart the server.
#------------------------------------------------------------------------------# GENERAL PARAMETERS#------------------------------------------------------------------------------nodename = 'one'					# Specifies the node name.								# (changes requires restart)#listen_addresses = '*'			# Listen addresses of this GTM.								# (changes requires restart)port = 6666					# Port number of this GTM.								# (changes requires restart)
#------------------------------------------------------------------------------# GTM PROXY PARAMETERS#------------------------------------------------------------------------------#worker_threads = 1				# Number of the worker thread of this								# GTM proxy								# (changes requires restart)
#------------------------------------------------------------------------------# GTM CONNECTION PARAMETERS#------------------------------------------------------------------------------# Those parameters are used to connect to a GTM servergtm_host = 'localhost' 					# Listen address of the active GTM.								# (changes requires restart)gtm_port = 6668 					# Port number of the active GTM.								# (changes requires restart)
#------------------------------------------------------------------------------# Behavior at GTM communication error#------------------------------------------------------------------------------#gtm_connect_retry_interval = 0	# How long (in secs) to wait until the next								# retry to connect to GTM.###------------------------------------------------------------------------------# Other options#------------------------------------------------------------------------------#keepalives_idle = 0			# Keepalives_idle parameter.#keepalives_interval = 0		# Keepalives_interval parameter.#keepalives_count = 0			# Keepalives_count internal parameter.#log_file = 'gtm_proxy.log'		# Log file name#log_min_messages = WARNING		# log_min_messages.  Default WARNING.							  	# Valid value: DEBUG, DEBUG5, DEBUG4, DEBUG3,								# DEBUG2, DEBUG1, INFO, NOTICE, WARNING,								# ERROR, LOG, FATAL, PANIC.


--Ronian

I installed pgxc 1.2.1 and I wanted to start gtm and gtm_proxy on the same
sever, but gtm_proxy didn't work.

I did it as such:
sudo su
mkdir /usr/local/pgsql/data_gtm
mkdir /usr/local/pgsql/data_gtm_proxy
chown l /usr/local/pgsql/data_gtm
chown l /usr/local/pgsql/data_gtm_proxy
su l
initgtm -Z gtm -D /usr/local/pgsql/data_gtm
initgtm -Z gtm_proxy -D /usr/local/pgsql/data_gtm_proxy
gtm -D /usr/local/pgsql/data_gtm &
gtm_proxy -D /usr/local/pgsql/data_gtm_proxy

On the last step,it returns CST -FATAL:  can not connect to GTM
LOCATION:  ConnectGTM, proxy_main.c:3344.
I followed it in the gtm_proxy,I found errorno 111 which means Connection
refused in
function connectFailureMessage
in GTMPQconnectPoll
in connectGTMStart
in PQconnectGTMStart
in PQconnectGTM
in ConnectGTM
in RegisterProxy
in BaseInit
in main.

My os is ubuntu 12.04 amd64 and I also tested it on centos 6.I also
installed pgxc 1.2.1 on both of them. But they all get the same error.I
found a mail "http://sourceforge.net/p/postgres-xc/mailman/message/30755193/",
we are exactly the same.

I followed and tried each every pages on net, but still can't solve it. Can
you please tell me what I can do? Any help here would be really appreciated.

gtm.conf and gtm_proxy.conf come as follow:




gtm.conf:
# ----------------------
# GTM configuration file
# ----------------------
#
# This file must be placed on gtm working directory
# specified by -D command line option of gtm or gtm_ctl.  The
# configuration file name must be "gtm.conf"
#
#
# This file consists of lines of the form
#
#  name = value
#
# (The "=" is optional.) Whitespace may be used.   Comments are
# introduced with "#" anywhere on a line.  The complete list of
# parameter names and allowed values can be found in the
# Postgres-XC documentation.
#
# The commented-out settings shown in this file represent the default
# values.
#
# Re-commenting a setting is NOT sufficient to revert it to the default
# value.
#
# You need to restart the server.

#------------------------------------------------------------------------------
# GENERAL PARAMETERS
#------------------------------------------------------------------------------
nodename = 'one' # Specifies the node name.
# (changes requires restart)
#listen_addresses = '*' # Listen addresses of this GTM.
# (changes requires restart)
port = 6666 # Port number of this GTM.
# (changes requires restart)

#startup = ACT # Start mode. ACT/STANDBY.

#------------------------------------------------------------------------------
# GTM STANDBY PARAMETERS
#------------------------------------------------------------------------------
#Those parameters are effective when GTM is activated as a standby server
#active_host = '' # Listen address of active GTM.
# (changes requires restart)
#active_port = # Port number of active GTM.
# (changes requires restart)

#---------------------------------------
# OTHER OPTIONS
#---------------------------------------
#keepalives_idle = 0 # Keepalives_idle parameter.
#keepalives_interval = 0 # Keepalives_interval parameter.
#keepalives_count = 0 # Keepalives_count internal parameter.
#log_file = 'gtm.log' # Log file name
#log_min_messages = WARNING # log_min_messages.  Default WARNING.
   # Valid value: DEBUG, DEBUG5, DEBUG4, DEBUG3,
# DEBUG2, DEBUG1, INFO, NOTICE, WARNING,
# ERROR, LOG, FATAL, PANIC
#synchronous_backup = off # If backup to standby is synchronous







gtm_proxy.conf:

#-----------------------------
# GTM Proxy configuration file
#-----------------------------
#
# This file must be placed on gtm working directory
# specified by -D command line option of gtm_proxy or gtm_ctl.
# The configuration file name must be "gtm_proxy.conf"
#
#
# This file consists of lines of the form
#
#  name = value
#
# (The "=" is optional.) Whitespace may be used.   Comments are
# introduced with "#" anywhere on a line.  The complete list of
# parameter names and allowed values can be found in the
# Postgres-XC documentation.
#
# The commented-out settings shown in this file represent the default
# values.
#
# Re-commenting a setting is NOT sufficient to revert it to the default
# value.
#
# You need to restart the server.

#------------------------------------------------------------------------------
# GENERAL PARAMETERS
#------------------------------------------------------------------------------
nodename = 'one' # Specifies the node name.
# (changes requires restart)
#listen_addresses = '*' # Listen addresses of this GTM.
# (changes requires restart)
port = 6666 # Port number of this GTM.
# (changes requires restart)

#------------------------------------------------------------------------------
# GTM PROXY PARAMETERS
#------------------------------------------------------------------------------
#worker_threads = 1 # Number of the worker thread of this
# GTM proxy
# (changes requires restart)

#------------------------------------------------------------------------------
# GTM CONNECTION PARAMETERS
#------------------------------------------------------------------------------
# Those parameters are used to connect to a GTM server
gtm_host = 'localhost' # Listen address of the active GTM.
# (changes requires restart)
gtm_port = 6668 # Port number of the active GTM.
# (changes requires restart)

#------------------------------------------------------------------------------
# Behavior at GTM communication error
#------------------------------------------------------------------------------
#gtm_connect_retry_interval = 0 # How long (in secs) to wait until the next
# retry to connect to GTM.
#
#
#------------------------------------------------------------------------------
# Other options
#------------------------------------------------------------------------------
#keepalives_idle = 0 # Keepalives_idle parameter.
#keepalives_interval = 0 # Keepalives_interval parameter.
#keepalives_count = 0 # Keepalives_count internal parameter.
#log_file = 'gtm_proxy.log' # Log file name
#log_min_messages = WARNING # log_min_messages.  Default WARNING.
   # Valid value: DEBUG, DEBUG5, DEBUG4, DEBUG3,
# DEBUG2, DEBUG1, INFO, NOTICE, WARNING,
# ERROR, LOG, FATAL, PANIC.



--Ronian

Did you backup the file you cleared?

if not, it was not a good idea to clean things because GTM writes its restart point at its working directory.   If you did, then you can restore the file gtm.control back.   It is in fact a text file so if you would like, you can edit it for repair.  First like is the next GXID value.    Following lines define each sequence.   This could be a bit harmful but should work.   If you ran gtm slave, it will also have gtm.control which can be used as well.

Good luck.
---
Koichi Suzuki

2014/04/25 2:20、Aaron Jackson <aja...@re...<mailto:aja...@re...>> のメール：

So I noticed a problem creating a sequence on my cluster - the logs indicated that there was a gtm failure.  The GTM indicated that the sequence existed.  So I stopped the GTM but left the two coordinators (and proxies) running.  I then wiped out everything exception the configuration file and restarted.  Soon after that, I noticed that the schemas I had created were no longer available through the coordinator - through attempts to create them said they existed.  The best I can guess is that my reset caused the transaction #s to get munged and the schemas were not visible at the mvcc level I was currently querying.  Long story short, I did this one to myself - however, in a real scenario, how would you go about resetting the environment if the GTM suffered a failure and needed to be rebuilt?
------------------------------------------------------------------------------
Start Your Social Network Today - Download eXo Platform
Build your Enterprise Intranet with eXo Platform Software
Java Based Open Source Intranet - Social, Extensible, Cloud Ready
Get Started Now And Turn Your Intranet Into A Collaboration Platform
http://p.sf.net/sfu/ExoPlatform_______________________________________________
Postgres-xc-general mailing list
Pos...@li...
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

So I noticed a problem creating a sequence on my cluster - the logs indicated that there was a gtm failure.  The GTM indicated that the sequence existed.  So I stopped the GTM but left the two coordinators (and proxies) running.  I then wiped out everything exception the configuration file and restarted.  Soon after that, I noticed that the schemas I had created were no longer available through the coordinator - through attempts to create them said they existed.  The best I can guess is that my reset caused the transaction #s to get munged and the schemas were not visible at the mvcc level I was currently querying.  Long story short, I did this one to myself - however, in a real scenario, how would you go about resetting the environment if the GTM suffered a failure and needed to be rebuilt?

At present pgxc_ctl does this.
---
Koichi Suzuki

2014/04/24 13:31、Pavan Deolasee <pav...@gm...<mailto:pav...@gm...>> のメール：



I just want to make sure I'm not missing something obvious.  If this is the way it's designed to work, then that's fine.

Yes, this is per design. Having said that, I wonder if we should make it easier for users. One reason why we don't do it automatically today is because the nodes being added may not be online when they are being added. We discussed a scheme to handle this in the past. One option is to add an ONLINE option to create node command and if specified, coordinator can try connecting to the new node and make appropriate catalog changes there too. Command can fail if the node is Unreachable. We can also find a way to fire pgxc_pool_reload automatically when cluster definition changes.

Thanks,
Pavan
------------------------------------------------------------------------------
Start Your Social Network Today - Download eXo Platform
Build your Enterprise Intranet with eXo Platform Software
Java Based Open Source Intranet - Social, Extensible, Cloud Ready
Get Started Now And Turn Your Intranet Into A Collaboration Platform
http://p.sf.net/sfu/ExoPlatform_______________________________________________
Postgres-xc-general mailing list
Pos...@li...
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

> 
> I just want to make sure I'm not missing something obvious.  If this is the way it's designed to work, then that's fine.
> 
Yes, this is per design. Having said that, I wonder if we should make it easier for users. One reason why we don't do it automatically today is because the nodes being added may not be online when they are being added. We discussed a scheme to handle this in the past. One option is to add an ONLINE option to create node command and if specified, coordinator can try connecting to the new node and make appropriate catalog changes there too. Command can fail if the node is Unreachable. We can also find a way to fire pgxc_pool_reload automatically when cluster definition changes.

Thanks,
Pavan

On Thu, Apr 24, 2014 at 12:49 AM, Aaron Jackson <aja...@re...>wrote:

>   I've read through several pieces of documentation but couldn't find an
> answer, short of experimentation.  I'm really unclear about how
> coordinators are "managed" when they are in a cluster.  From what I can
> tell, it looks like each node in a cluster must be individually setup ...
> for example.
>
>  I have three nodes, A, B & C.
>
>  - A is my GTM instance.
> - B & C newly created data nodes and coordinators - nothing in them
>
>  Now let's say I want to make B & C aware of each other at the
> coordinator level.  So on server B, I connect to the coordinator and issue
> the following:
>
>  CREATE NODE coord_2 WITH (TYPE = 'coordinator', PORT = 5432, HOST =
> 'B');
> select pgxc_pool_reload();
> select * from pg_catalog.pgxc_node;
>
>  And as expected, I now have two entries in pgxc_node.  I hop over to
> machine C ...
>
>  select * from pg_catalog.pgxc_node;
>
>  This returns only a row for itself.  Okay, so I reload the pool and
> still one row.  This implies that adding a coordinator node only affects
> the coordinator where it was run on.  So, I must go to all machines in my
> cluster and tell them that there is a new coordinator.
>
>  I just want to make sure I'm not missing something obvious.  If this is
> the way it's designed to work, then that's fine.
>
>
You are right. We have to "introduce" each node (not just coordinator but
datanode as well) to all (other, if applicable), coordinators.


>  Aaron
>
>
> ------------------------------------------------------------------------------
> Start Your Social Network Today - Download eXo Platform
> Build your Enterprise Intranet with eXo Platform Software
> Java Based Open Source Intranet - Social, Extensible, Cloud Ready
> Get Started Now And Turn Your Intranet Into A Collaboration Platform
> http://p.sf.net/sfu/ExoPlatform
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
>


-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

I've read through several pieces of documentation but couldn't find an answer, short of experimentation.  I'm really unclear about how coordinators are "managed" when they are in a cluster.  From what I can tell, it looks like each node in a cluster must be individually setup ... for example.

I have three nodes, A, B & C.

- A is my GTM instance.
- B & C newly created data nodes and coordinators - nothing in them

Now let's say I want to make B & C aware of each other at the coordinator level.  So on server B, I connect to the coordinator and issue the following:

CREATE NODE coord_2 WITH (TYPE = 'coordinator', PORT = 5432, HOST = 'B');
select pgxc_pool_reload();
select * from pg_catalog.pgxc_node;

And as expected, I now have two entries in pgxc_node.  I hop over to machine C ...

select * from pg_catalog.pgxc_node;

This returns only a row for itself.  Okay, so I reload the pool and still one row.  This implies that adding a coordinator node only affects the coordinator where it was run on.  So, I must go to all machines in my cluster and tell them that there is a new coordinator.

I just want to make sure I'm not missing something obvious.  If this is the way it's designed to work, then that's fine.

Aaron