postgres-xc-general Mailing List for Postgres-XC (Page 6)

Hi,

How i can pooling the connections on the coordinators using java? PGpool?


Thanks!

-- 
Att,
Bruno Cezar de Oliveira.
bru...@ho...
-------------------------------------------------------
Uberlândia - MG.

Hi, Aaron

I think you've done right analysis.

The order of evaluation of sub-expressions and the order in which side
effects take place are frequently defined as unspecified behavior by
the C Standard. As you saw, it's one of the such case that the order
in which the arguments to a function are evaluated.

The patch looks good, and I think we must back-patch it to all the
versions after 1.0.

Regards.

On 3 June 2014 16:20, Aaron Jackson <aja...@re...> wrote:
> I've been able to work my way backwards through the problem and have
> discovered the underlying problem.  When a data coordinator is paired with a
> GTM proxy, it forwards its message to the GTM proxy who adds some data to
> the payload and forwards it to the GTM.  Here is what I saw when looking at
> the wire.
>
> The message captured between the data coordinator and the GTM proxy was as
> follows:
>
> 430000003d000000270000001064656d6f2e7075626c69632e666f6f0001000000000000000100000000000000ffffffffffffff7f010000000000000000
>
> The message captured between the GTM proxy and the GTM was as follows:
>
> 430000000a00000000002746000000080000003b
>
>
> Definitely a horrible truncation of the payload.  The problem is in
> GTMProxy_ProxyCommand, specifically, the two calls to pq_getmsgunreadlen().
> The assumption is that these are called before anything else.
> Unfortunately, the intel compiler calls pq_getmsgbytes() and subsequently
> calls the second instance of pq_getmsgunreadlen().  The second time it is
> called, the value returns zero and we end up with all kinds of byte
> truncation.  I've attached a patch to fix the issue.
>
> --- postgres-xc-1.2.1-orig/src/gtm/proxy/proxy_main.c   2014-04-03
> 05:18:38.000000000 +0000
> +++ postgres-xc-1.2.1/src/gtm/proxy/proxy_main.c        2014-06-03
> 07:14:58.451411000 +0000
> @@ -2390,6 +2390,7 @@
>         GTMProxy_CommandInfo *cmdinfo;
>         GTMProxy_ThreadInfo *thrinfo = GetMyThreadInfo;
>         GTM_ProxyMsgHeader proxyhdr;
> +       size_t msgunreadlen = pq_getmsgunreadlen(message);
>
>         proxyhdr.ph_conid = conninfo->con_id;
>
> @@ -2397,8 +2398,8 @@
>         if (gtmpqPutMsgStart('C', true, gtm_conn) ||
>                 gtmpqPutnchar((char *)&proxyhdr, sizeof
> (GTM_ProxyMsgHeader), gtm_conn) ||
>                 gtmpqPutInt(mtype, sizeof (GTM_MessageType), gtm_conn) ||
> -               gtmpqPutnchar(pq_getmsgbytes(message,
> pq_getmsgunreadlen(message)),
> -                                         pq_getmsgunreadlen(message),
> gtm_conn))
> +               gtmpqPutnchar(pq_getmsgbytes(message, msgunreadlen),
> +                                         msgunreadlen, gtm_conn))
>                 elog(ERROR, "Error proxing data");
>
>         /*
>
>
>
> Aaron
> ________________________________
> From: Aaron Jackson [aja...@re...]
> Sent: Monday, June 02, 2014 4:11 PM
> To: pos...@li...
> Subject: [Postgres-xc-general] Unable to create sequences
>
> I tried to create a database as follows ...
>
> CREATE TABLE Schema.TableFoo(
>     SomeId serial NOT NULL,
>     ForeignId int NOT NULL,
>     ...
> ) WITH (OIDS = FALSE);
>
>
> The server returned the following...
>
> ERROR:  GTM error, could not create sequence
>
>
> Looked at the server logs for the gtm_proxy, nothing so I went to the gtm.
>
> LOCATION:  pq_copymsgbytes, pqformat.c:554
> 1:140488486782720:2014-06-02 21:10:58.870 UTC -WARNING:  No transaction
> handle for gxid: 0
> LOCATION:  GTM_GXIDToHandle, gtm_txn.c:163
> 1:140488486782720:2014-06-02 21:10:58.870 UTC -WARNING:  Invalid transaction
> handle: -1
> LOCATION:  GTM_HandleToTransactionInfo, gtm_txn.c:213
> 1:140488486782720:2014-06-02 21:10:58.870 UTC -ERROR:  Failed to get a
> snapshot
> LOCATION:  ProcessGetSnapshotCommandMulti, gtm_snap.c:420
> 1:140488478390016:2014-06-02 21:10:58.871 UTC -ERROR:  insufficient data
> left in message
> LOCATION:  pq_copymsgbytes, pqformat.c:554
> 1:140488486782720:2014-06-02 21:10:58.871 UTC -ERROR:  insufficient data
> left in message
> LOCATION:  pq_copymsgbytes, pqformat.c:554
>
>
> I'm definitely confused here.  This cluster has been running fine for
> several days now.  And now the GTM is failing. I performed a restart of the
> gtm and proxies (each using gtm_ctl to stop and restart the instance).
> Nothing has changed, the GTM continues to fail and will not create the
> sequence.
>
> Any ideas?
>
> Aaron
>
> ------------------------------------------------------------------------------
> Learn Graph Databases - Download FREE O'Reilly Book
> "Graph Databases" is the definitive new guide to graph databases and their
> applications. Written by three acclaimed leaders in the field,
> this first edition is now available. Download your free book today!
> http://p.sf.net/sfu/NeoTech
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>

I've been able to work my way backwards through the problem and have discovered the underlying problem.  When a data coordinator is paired with a GTM proxy, it forwards its message to the GTM proxy who adds some data to the payload and forwards it to the GTM.  Here is what I saw when looking at the wire.

The message captured between the data coordinator and the GTM proxy was as follows:
430000003d000000270000001064656d6f2e7075626c69632e666f6f0001000000000000000100000000000000ffffffffffffff7f010000000000000000
The message captured between the GTM proxy and the GTM was as follows:
430000000a00000000002746000000080000003b

Definitely a horrible truncation of the payload.  The problem is in GTMProxy_ProxyCommand, specifically, the two calls to pq_getmsgunreadlen().  The assumption is that these are called before anything else.  Unfortunately, the intel compiler calls pq_getmsgbytes() and subsequently calls the second instance of pq_getmsgunreadlen().  The second time it is called, the value returns zero and we end up with all kinds of byte truncation.  I've attached a patch to fix the issue.

--- postgres-xc-1.2.1-orig/src/gtm/proxy/proxy_main.c   2014-04-03 05:18:38.000000000 +0000
+++ postgres-xc-1.2.1/src/gtm/proxy/proxy_main.c        2014-06-03 07:14:58.451411000 +0000
@@ -2390,6 +2390,7 @@
        GTMProxy_CommandInfo *cmdinfo;
        GTMProxy_ThreadInfo *thrinfo = GetMyThreadInfo;
        GTM_ProxyMsgHeader proxyhdr;
+       size_t msgunreadlen = pq_getmsgunreadlen(message);

        proxyhdr.ph_conid = conninfo->con_id;

@@ -2397,8 +2398,8 @@
        if (gtmpqPutMsgStart('C', true, gtm_conn) ||
                gtmpqPutnchar((char *)&proxyhdr, sizeof (GTM_ProxyMsgHeader), gtm_conn) ||
                gtmpqPutInt(mtype, sizeof (GTM_MessageType), gtm_conn) ||
-               gtmpqPutnchar(pq_getmsgbytes(message, pq_getmsgunreadlen(message)),
-                                         pq_getmsgunreadlen(message), gtm_conn))
+               gtmpqPutnchar(pq_getmsgbytes(message, msgunreadlen),
+                                         msgunreadlen, gtm_conn))
                elog(ERROR, "Error proxing data");

        /*


Aaron
________________________________
From: Aaron Jackson [aja...@re...]
Sent: Monday, June 02, 2014 4:11 PM
To: pos...@li...
Subject: [Postgres-xc-general] Unable to create sequences

I tried to create a database as follows ...

CREATE TABLE Schema.TableFoo(
    SomeId serial NOT NULL,
    ForeignId int NOT NULL,
    ...
) WITH (OIDS = FALSE);

The server returned the following...

ERROR:  GTM error, could not create sequence

Looked at the server logs for the gtm_proxy, nothing so I went to the gtm.

LOCATION:  pq_copymsgbytes, pqformat.c:554
1:140488486782720:2014-06-02 21:10:58.870 UTC -WARNING:  No transaction handle for gxid: 0
LOCATION:  GTM_GXIDToHandle, gtm_txn.c:163
1:140488486782720:2014-06-02 21:10:58.870 UTC -WARNING:  Invalid transaction handle: -1
LOCATION:  GTM_HandleToTransactionInfo, gtm_txn.c:213
1:140488486782720:2014-06-02 21:10:58.870 UTC -ERROR:  Failed to get a snapshot
LOCATION:  ProcessGetSnapshotCommandMulti, gtm_snap.c:420
1:140488478390016:2014-06-02 21:10:58.871 UTC -ERROR:  insufficient data left in message
LOCATION:  pq_copymsgbytes, pqformat.c:554
1:140488486782720:2014-06-02 21:10:58.871 UTC -ERROR:  insufficient data left in message
LOCATION:  pq_copymsgbytes, pqformat.c:554

I'm definitely confused here.  This cluster has been running fine for several days now.  And now the GTM is failing. I performed a restart of the gtm and proxies (each using gtm_ctl to stop and restart the instance).  Nothing has changed, the GTM continues to fail and will not create the sequence.

Any ideas?

Aaron

I tried to create a database as follows ...

CREATE TABLE Schema.TableFoo(
    SomeId serial NOT NULL,
    ForeignId int NOT NULL,
    ...
) WITH (OIDS = FALSE);

The server returned the following...

ERROR:  GTM error, could not create sequence

Looked at the server logs for the gtm_proxy, nothing so I went to the gtm.

LOCATION:  pq_copymsgbytes, pqformat.c:554
1:140488486782720:2014-06-02 21:10:58.870 UTC -WARNING:  No transaction handle for gxid: 0
LOCATION:  GTM_GXIDToHandle, gtm_txn.c:163
1:140488486782720:2014-06-02 21:10:58.870 UTC -WARNING:  Invalid transaction handle: -1
LOCATION:  GTM_HandleToTransactionInfo, gtm_txn.c:213
1:140488486782720:2014-06-02 21:10:58.870 UTC -ERROR:  Failed to get a snapshot
LOCATION:  ProcessGetSnapshotCommandMulti, gtm_snap.c:420
1:140488478390016:2014-06-02 21:10:58.871 UTC -ERROR:  insufficient data left in message
LOCATION:  pq_copymsgbytes, pqformat.c:554
1:140488486782720:2014-06-02 21:10:58.871 UTC -ERROR:  insufficient data left in message
LOCATION:  pq_copymsgbytes, pqformat.c:554

I'm definitely confused here.  This cluster has been running fine for several days now.  And now the GTM is failing. I performed a restart of the gtm and proxies (each using gtm_ctl to stop and restart the instance).  Nothing has changed, the GTM continues to fail and will not create the sequence.

Any ideas?

Aaron

On Sun, May 25, 2014 at 2:04 PM, Koichi Suzuki <koi...@gm...>
wrote:

> I see.  Your have good usecase for read-only transactions.
>
> Because of the nature of log shipping and sharing/clustering, it is
> not simple to provide read-only transaction in XC.


There may be cases where it is being used as a data warehouse (thinking a
bit more about Postgres-XL here), where there is a nightly load process and
then reads are done during the day.  If the standbys are dedicated, being
able to load balance them could be useful.



>  Two essential
> reasons:
>
> 1. Delay in WAL playback in each slave may be different.   It makes
> providing consistent database view extremely difficult.
>


> 2. At present, slave calculates snapshot of the transaction from the
> WAL.   Current code does not allow missing XIDs.   There will be
> memory leak and crash by OOM if there's many missing XIDs in the WAL
> stream.  In XC, it is disabled and the database view may be
> inconsistent.   Please note that this does not affect recovery and
> promotion.
>
>
>
Agreed, there may be some level of effort for this.

-- 
Mason Sharp

TransLattice - http://www.translattice.com
Distributed and Clustered Database Solutions

On Thu, May 29, 2014 at 7:27 AM, Alan Bryant <ala...@hd...> wrote:

>
>
>  - Can we shard on schema? (i.e. put some schemas in one location and
> other schemas in different locations, sharded the same way rows are?) Or if
> not, can we shard one table, and have functions execute on the same node as
> some controlling row?
>
>
>  No.
>
>    Interesting. This leaves us with the "one big table" approach, just
> spread across the data nodes. Aren't there performance issues with having
> massive individual tables? What techniques exist to help with this?
>

The individual tables can be replicated or sharded, such as a hash based on
one of the columns in the table.

A parent-child type of relation can be sharded on a primary key of the
parent, and the foreign key referencing that table in the child. The
planner recognizes that this data will co-exist on the same node and
"push-down" joins.

Other tables can be replicated, with an exact copy on each node if desired.
Here joins can always be pushed down. If a query only involves replicated
tables, it will just be sent to one single node. If a coordinator and data
node are on the same server, one can configure the system to prefer to get
this data from a local data node.


>
> Partitioning is one option I guess, using a different (but compatible)
> scheme than the sharding key, perhaps a different bitmask on the same key.
>

At the moment users cannot specify an arbitrary expression like a bitmask.
A modulo option exists, however. I think this is an item that should be a
relatively high-ish priority on the to do list.


>
> Another question is upgrade. We use schemas primarily to make logical
> upgrades possible while online, but what about Postgres-XC itself... what
> is the upgrade model for Postgres-XC? (say, between major PostgreSQL
> versions)
>
>
This is another area that can be improved, leveraging pg_upgrade. For now,
dump and restore...



> Thanks!
> -Alan
>

Regards,

-- 
Mason Sharp

TransLattice - http://www.translattice.com
Distributed and Clustered Database Solutions

Ok. Thank you!

--- Mensagem Original ---

De: "Koichi Suzuki" <koi...@gm...>
Enviado: 29 de maio de 2014 22:48
Para: "Bruno Cezar Oliveira" <bru...@ho...>
Cc: "Postgres-XC mailing list" <pos...@li...>
Assunto: Re: [Postgres-xc-general] New Datanode in real-time

You can download this information from sourceforge page
http://sourceforge.net/projects/postgres-xc/files/Pgxc_ctl_primer/

Please download Pgxc_ctl_primer_20140507.pdf and take a look at it.

This material describes configuration and operation of Postgres-XC
cluster using pgxc_ctl.   Also, this describes how each
configuration/operation works including datanode addition/removal.

Hope this helps.
---
Koichi Suzuki


2014-05-30 4:04 GMT+09:00 Bruno Cezar Oliveira <bru...@ho...>:
> Hi!
>
> i am studying the postgres-xc to use this database in the enterprise  i
> work.
> I can add a datanode in real-time? How? If not, how i can do this?
> Looking in the docs i could not find.
>
>
> Thanks!
>
>
> ------------------------------------------------------------------------------
> Time is money. Stop wasting it! Get your web API in 5 minutes.
> www.restlet.com/download
> http://p.sf.net/sfu/restlet
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>

You can download this information from sourceforge page
http://sourceforge.net/projects/postgres-xc/files/Pgxc_ctl_primer/

Please download Pgxc_ctl_primer_20140507.pdf and take a look at it.

This material describes configuration and operation of Postgres-XC
cluster using pgxc_ctl.   Also, this describes how each
configuration/operation works including datanode addition/removal.

Hope this helps.
---
Koichi Suzuki


2014-05-30 4:04 GMT+09:00 Bruno Cezar Oliveira <bru...@ho...>:
> Hi!
>
> i am studying the postgres-xc to use this database in the enterprise  i
> work.
> I can add a datanode in real-time? How? If not, how i can do this?
> Looking in the docs i could not find.
>
>
> Thanks!
>
>
> ------------------------------------------------------------------------------
> Time is money. Stop wasting it! Get your web API in 5 minutes.
> www.restlet.com/download
> http://p.sf.net/sfu/restlet
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>

XC replicates schema at all the nodes.   It has all the information how each table is sharded or replicated.   So, schema itself is not sharded but tables can be sharded.

Hope this makes sense.

Regards;
---
Koichi Suzuki

2014/05/29 20:27、Alan Bryant <ala...@hd...<mailto:ala...@hd...>> のメール：



- Can we shard on schema? (i.e. put some schemas in one location and other schemas in different locations, sharded the same way rows are?) Or if not, can we shard one table, and have functions execute on the same node as some controlling row?

No.

Interesting. This leaves us with the "one big table" approach, just spread across the data nodes. Aren't there performance issues with having massive individual tables? What techniques exist to help with this?

Partitioning is one option I guess, using a different (but compatible) scheme than the sharding key, perhaps a different bitmask on the same key.

Another question is upgrade. We use schemas primarily to make logical upgrades possible while online, but what about Postgres-XC itself... what is the upgrade model for Postgres-XC? (say, between major PostgreSQL versions)

Thanks!
-Alan

------------------------------------------------------------------------------
Time is money. Stop wasting it! Get your web API in 5 minutes.
www.restlet.com/download<http://www.restlet.com/download>
http://p.sf.net/sfu/restlet_______________________________________________
Postgres-xc-general mailing list
Pos...@li...
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

Hi!

i am studying the postgres-xc to use this database in the enterprise  i
work.
I can add a datanode in real-time? How? If not, how i can do this?
Looking in the docs i could not find.


Thanks!

- Can we shard on schema? (i.e. put some schemas in one location and other schemas in different locations, sharded the same way rows are?) Or if not, can we shard one table, and have functions execute on the same node as some controlling row?

No.

Interesting. This leaves us with the "one big table" approach, just spread across the data nodes. Aren't there performance issues with having massive individual tables? What techniques exist to help with this?

Partitioning is one option I guess, using a different (but compatible) scheme than the sharding key, perhaps a different bitmask on the same key.

Another question is upgrade. We use schemas primarily to make logical upgrades possible while online, but what about Postgres-XC itself... what is the upgrade model for Postgres-XC? (say, between major PostgreSQL versions)

Thanks!
-Alan

2014/05/29 12:29、Ashutosh Bapat <ash...@en...<mailto:ash...@en...>> のメール：




On Thu, May 29, 2014 at 6:41 AM, 鈴木 幸市 <ko...@in...<mailto:ko...@in...>> wrote:
2014/05/29 1:27、Josh Berkus <jo...@ag...<mailto:jo...@ag...>> のメール：

> On 05/27/2014 07:33 PM, Koichi Suzuki wrote:
>> We can reduce the additional latency by performing prepare and commit
>> in parallel, I mean, sending command to all the target remote nodes
>> first and then receive their response afterwords.
>>
>> As I supposed, alternative is to use BDR.   This has very small
>> overhead.   We can detect conflicting writes among transactions and if
>> each transaction does not conflict, we can apply these writes in
>> parallel, not in a single thread as we're doing in streaming
>> replication.
>
> Mind you, like BDR itself, we'll still need to figure out how to handle DDL.

Thank you for the info.   I believe this issue is shared with other use cases such as SLONY.   Are there any other discussion how to handle this?

>
>>
>> This needs some more work and I think this worth to spend some time.
>
> Yes.  Otherwise we have two unpalatable choices:
>
> - Massive data loss (roll back to barrier) every time we lose a node, or
> - Doubling write latency (at least)

In the case of statement-based redundancy, we need to determine what node to write first (at least) and this should be the same in all the nodes.   It is needed to handle conflicting wrights consistently.   This means the first write has to be done synchronously and the rest can be done asynchronously.   Because most of the i/o work is done at prepare/commit, I hope this does not impact the whole throughput or latency badly.


I am doubtful as to whether the eventual consistency scheme will work in XC. If we commit a change to two nodes, and do not wait for the third node and the third node is not able to apply the change, we will have an inconsistency on the third node. This inconsistency may not be repairable depending upon the failure to write to that node. If the two nodes fail, the data that we will retrieve from the third node would be inconsistent. So, probably there is no point in keeping the third copy. That means that the cluster can tolerate N failures, where N is the number of synchronous copies made.

The above is synchronous because prepares and commits are done in synchronous way, by receiving all the response at last.   No prepare/commit response is  left outstanding.   All prepare responses will be handled before subsequent commot/abort.   All commit/abort will be handled before beginning next transaction. The idea is to do things in parallel only when it makes sense.

When an error occurs in prepare, we can abort it and no inconsistency will be left.   If one of following commit fails, this is not expected but may happen when system level failure occurs, such node should be detached from the cluster and need repair.

If abort fails during aborting failed prepare, this is also a failure to the node and such node should be detached from the cluster.

One thing we should think here is: we don’t have good means to simply repair such failed node.   It seems to me that we have to copy everything from a model node.   This takes time.    Maybe we need something like pg_rewind in statement-based replication or other tool to make a replica quickly.

Regards;
---
Koichi Suzuki


Regards;
—
Koichi Suzuki
>
> --
> Josh Berkus
> PostgreSQL Experts Inc.
> http://pgexperts.com<http://pgexperts.com/>
>
> ------------------------------------------------------------------------------
> Time is money. Stop wasting it! Get your web API in 5 minutes.
> www.restlet.com/download<http://www.restlet.com/download>
> http://p.sf.net/sfu/restlet
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...<mailto:Pos...@li...>
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>

------------------------------------------------------------------------------
Time is money. Stop wasting it! Get your web API in 5 minutes.
www.restlet.com/download<http://www.restlet.com/download>
http://p.sf.net/sfu/restlet
_______________________________________________
Postgres-xc-general mailing list
Pos...@li...<mailto:Pos...@li...>
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general



--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

On Thu, May 29, 2014 at 6:41 AM, 鈴木 幸市 <ko...@in...> wrote:

> 2014/05/29 1:27、Josh Berkus <jo...@ag...> のメール：
>
> > On 05/27/2014 07:33 PM, Koichi Suzuki wrote:
> >> We can reduce the additional latency by performing prepare and commit
> >> in parallel, I mean, sending command to all the target remote nodes
> >> first and then receive their response afterwords.
> >>
> >> As I supposed, alternative is to use BDR.   This has very small
> >> overhead.   We can detect conflicting writes among transactions and if
> >> each transaction does not conflict, we can apply these writes in
> >> parallel, not in a single thread as we're doing in streaming
> >> replication.
> >
> > Mind you, like BDR itself, we'll still need to figure out how to handle
> DDL.
>
> Thank you for the info.   I believe this issue is shared with other use
> cases such as SLONY.   Are there any other discussion how to handle this?
>
> >
> >>
> >> This needs some more work and I think this worth to spend some time.
> >
> > Yes.  Otherwise we have two unpalatable choices:
> >
> > - Massive data loss (roll back to barrier) every time we lose a node, or
> > - Doubling write latency (at least)
>
> In the case of statement-based redundancy, we need to determine what node
> to write first (at least) and this should be the same in all the nodes.
> It is needed to handle conflicting wrights consistently.   This means the
> first write has to be done synchronously and the rest can be done
> asynchronously.   Because most of the i/o work is done at prepare/commit, I
> hope this does not impact the whole throughput or latency badly.
>
>
I am doubtful as to whether the eventual consistency scheme will work in
XC. If we commit a change to two nodes, and do not wait for the third node
and the third node is not able to apply the change, we will have an
inconsistency on the third node. This inconsistency may not be repairable
depending upon the failure to write to that node. If the two nodes fail,
the data that we will retrieve from the third node would be inconsistent.
So, probably there is no point in keeping the third copy. That means that
the cluster can tolerate N failures, where N is the number of synchronous
copies made.


> Regards;
> —
> Koichi Suzuki
> >
> > --
> > Josh Berkus
> > PostgreSQL Experts Inc.
> > http://pgexperts.com
> >
> >
> ------------------------------------------------------------------------------
> > Time is money. Stop wasting it! Get your web API in 5 minutes.
> > www.restlet.com/download
> > http://p.sf.net/sfu/restlet
> > _______________________________________________
> > Postgres-xc-general mailing list
> > Pos...@li...
> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> >
>
>
> ------------------------------------------------------------------------------
> Time is money. Stop wasting it! Get your web API in 5 minutes.
> www.restlet.com/download
> http://p.sf.net/sfu/restlet
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>



-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

2014/05/29 0:08、Alan Bryant <ala...@hd...<mailto:ala...@hd...>> のメール：


I have a few questions about Postgres-XC that I couldn't find answers to in the documentation or with google searches:

- How does postgres-xc work with extensions, specifically PGQ? Do they work? Where do they "run?" on the data nodes or on the Coordinator?

I’m afraid nobody tested it.   At least, you should build the binary with XC environment.   Some of internal structure is not binary-compatible with PostgreSQL.

They run either on coordinator/datanode, depending upon specific functions.   Basically, only immutable functions run at datanodes.   Others run at coordinators.


- Similarly, how/where do user-defined functions get installed/run?

As mentioned above, immutable functions run at coordinators.   Others run at datanodes.


- And can user-defined functions perform DML and DDL?

Yes.


- Can we shard on schema? (i.e. put some schemas in one location and other schemas in different locations, sharded the same way rows are?) Or if not, can we shard one table, and have functions execute on the same node as some controlling row?

No.

Thank you;
—
Koichi Suzuki


Thanks,
-Alan

------------------------------------------------------------------------------
Time is money. Stop wasting it! Get your web API in 5 minutes.
www.restlet.com/download<http://www.restlet.com/download>
http://p.sf.net/sfu/restlet_______________________________________________
Postgres-xc-general mailing list
Pos...@li...
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

As Josh Berkus announced, there were bunch of XC-related discussions
in PGCon 2014. Whole meeting was very supportive to XC.

Materials will be found at following pages:
https://wiki.postgresql.org/wiki/PgCon2014ClusterSummit
https://wiki.postgresql.org/wiki/Pgcon2014PostgresXCmeeting

Also, there are a couple of unconference discussions related to XC.
Unconference page is
https://wiki.postgresql.org/wiki/Pgcon2014unconference

Hope these meeting notes are available soon.

Postgres-XC meeting was attended by several major PostgreSQL community
members as well as people interested in XC in their applications.

Through these discussions, I found the following are very important:

* XC has sufficient feature.   Stability is the crucial issue.
Priority of merge with PG9.4 is not high.  Missing features such as
WHERE CURRENT OF are not used.
* XC community should be more open.   For example, roadmap discussion
has not been done in public.
* Concentrate on simple benchmark, such as DBT-1, DBT-2 and pgbench.
* It is important to share evaluation and user experience.

I appreciate all the people in PGCon sessions/meetings for such
productive and fruitful discussions.

Give these suggestions, I'd like to begin open discussion of
Postgres-XC roadmap for release 1.3 and beyond, hopefully this
weekend.    Current core members are revising current roadmap to
reflect these discussions.   This will be published as a material for
discussions/proposals/comments/etc.

Any more input is very welcome.

Thank you very much.
---
Koichi Suzuki

Thanks for the report.   In this case, you can do the following:

1. Stop all the nodes just in case.   Use gtm_ctl and pg_ctl.  Do not kill them.  (You will waste some sequence and XID value if you kill them).  If your cluster was configured with pgxc_ctl, use it to stop the cluster.
2. Remove register.node file at GTM’s working directory.
3. Restart everything.

I hope this fixes the issue.

1.2.1 included several fix for cases as you saw but may have some more to improve.

Sorry for your inconvenience.

Best;
---
Koichi Suzuki

2014/05/29 1:17、Aaron Jackson <aja...@re...<mailto:aja...@re...>> のメール：

We started noticing outages with one of the nodes in our cluster this morning.  When I looked at the box, gtm proxy was running hot.  So, I looked at the logs of the GTM proxy - which was spooling rapidly.

LOCATION:  pgxcnode_add_info, register_common.c:249
1:139746454378240:2014-05-28 16:06:44.641 UTC -LOG:  Node with the given ID number already exists
LOCATION:  pgxcnode_add_info, register_common.c:249
1:139746445985536:2014-05-28 16:06:44.641 UTC -LOG:  Node with the given ID number already exists
LOCATION:  pgxcnode_add_info, register_common.c:249
1:139746454378240:2014-05-28 16:06:44.641 UTC -LOG:  Node with the given ID number already exists
LOCATION:  pgxcnode_add_info, register_common.c:249
1:139746445985536:2014-05-28 16:06:44.641 UTC -LOG:  Node with the given ID number already exists
LOCATION:  pgxcnode_add_info, register_common.c:249

After several restarts, it was painfully obvious that this proxy was not going to start.  In an attempt to recover the GTM proxy, I saved the GTM proxy directory and rebuilt it from scratch expecting to rebuild its state from the master.  No dice, here's what the data coordinator reported.

26885 | 2014-05-28 16:01:49 UTC | LOG:  autovacuum launcher started
26885 | 2014-05-28 16:02:50 UTC | ERROR:  GTM error, could not obtain snapshot XID = 1193752
26885 | 2014-05-28 16:03:51 UTC | WARNING:  Xid is invalid.
26885 | 2014-05-28 16:04:31 UTC | WARNING:  Xid is invalid.
26863 | 2014-05-28 16:04:49 UTC | FATAL:  Can not register Coordinator on GTM
26885 | 2014-05-28 16:05:31 UTC | WARNING:  Xid is invalid.
26885 | 2014-05-28 16:05:31 UTC | ERROR:  GTM error, could not obtain snapshot XID = 0
26885 | 2014-05-28 16:06:12 UTC | WARNING:  Xid is invalid.

I believe the first portion of the log occurred when the autovacuum tried to connect to the new proxy.

Here's what I'd like to know.  Clearly the GTM proxy failure wasn't expected, but the problem here is one of recoverability.  How does one go about getting the coordinator and data node to a sane place given that the old GTM proxy was in an unrecoverable state and a new one can't seemingly be put in place to replace it?

Aaron
------------------------------------------------------------------------------
Time is money. Stop wasting it! Get your web API in 5 minutes.
www.restlet.com/download<http://www.restlet.com/download>
http://p.sf.net/sfu/restlet_______________________________________________
Postgres-xc-general mailing list
Pos...@li...
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

2014/05/29 1:27、Josh Berkus <jo...@ag...> のメール：

> On 05/27/2014 07:33 PM, Koichi Suzuki wrote:
>> We can reduce the additional latency by performing prepare and commit
>> in parallel, I mean, sending command to all the target remote nodes
>> first and then receive their response afterwords.
>> 
>> As I supposed, alternative is to use BDR.   This has very small
>> overhead.   We can detect conflicting writes among transactions and if
>> each transaction does not conflict, we can apply these writes in
>> parallel, not in a single thread as we're doing in streaming
>> replication.
> 
> Mind you, like BDR itself, we'll still need to figure out how to handle DDL.

Thank you for the info.   I believe this issue is shared with other use cases such as SLONY.   Are there any other discussion how to handle this?

> 
>> 
>> This needs some more work and I think this worth to spend some time.
> 
> Yes.  Otherwise we have two unpalatable choices:
> 
> - Massive data loss (roll back to barrier) every time we lose a node, or
> - Doubling write latency (at least)

In the case of statement-based redundancy, we need to determine what node to write first (at least) and this should be the same in all the nodes.   It is needed to handle conflicting wrights consistently.   This means the first write has to be done synchronously and the rest can be done asynchronously.   Because most of the i/o work is done at prepare/commit, I hope this does not impact the whole throughput or latency badly.

Regards;
—
Koichi Suzuki
> 
> -- 
> Josh Berkus
> PostgreSQL Experts Inc.
> http://pgexperts.com
> 
> ------------------------------------------------------------------------------
> Time is money. Stop wasting it! Get your web API in 5 minutes.
> www.restlet.com/download
> http://p.sf.net/sfu/restlet
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> 

If you have similar problem, please raise log_min_messages to DEBUG1 or higher.   This may give some more info of the problem.

Thank you;
---
Koichi Suzuki

2014/05/29 2:29、Aaron Jackson <aja...@re...> のメール：

> I'm using 1.2.1 with gcc 4.7 on Amazon i3 instances.  I'm actually becoming somewhat suspect of the i3 platform running this specific version of ubuntu.  So, I've recompiled the entire database using the intel compilers and I'm going to see if I see the same behavioral problems I've seen in the last month and a half.
> 
> I will let you know.
> 
> ________________________________________
> From: Koichi Suzuki [koi...@gm...]
> Sent: Saturday, May 24, 2014 11:09 AM
> To: Aaron Jackson
> Cc: pos...@li...
> Subject: Re: [Postgres-xc-general] Interday Degradation
> 
> Sorry for the late response.
> 
> What version are you using?   1.2.1 includes several fix for GTM connectivity.
> ---
> Koichi Suzuki
> 
> 
> 2014-05-22 12:28 GMT-04:00 Aaron Jackson <aja...@re...>:
>> Given my past experience with compiler issues, I'm a little hesitant to even
>> report this.  That said, I have a three node cluster, each with a
>> coordinator, data node and gtm proxy.  I have a standalone gtm instance
>> without a slave.  Often, when I come in after the servers have been up for a
>> while, I'm greeted with a variety of issues.
>> 
>> There are several warnings in the coordinator and data node logs, that read
>> "Do not have a GTM snapshot available" - I've discarded these as mostly
>> benign for the moment.
>> 
>> The coordinator is much worse..
>> 
>> 30770 | 2014-05-22 15:53:06 UTC | ERROR:  current transaction is aborted,
>> commands ignored until end of transaction block
>> 30770 | 2014-05-22 15:53:06 UTC | STATEMENT:  DISCARD ALL
>> 4560 | 2014-05-22 15:54:30 UTC | LOG:  failed to connect to Datanode
>> 4560 | 2014-05-22 15:54:30 UTC | LOG:  failed to connect to Datanode
>> 4560 | 2014-05-22 15:54:30 UTC | WARNING:  can not connect to node 16390
>> 30808 | 2014-05-22 15:54:30 UTC | LOG:  failed to acquire connections
>> 
>> 
>> Usually, I reset the coordinator and datanode and the world is happy again.
>> However, it makes me somewhat concerned that I'm seeing these kinds of
>> failures on a daily basis.  I wouldn't rule out the compiler again as it's
>> been the reason for previous failures, but has anyone else seen anything
>> like this??
>> 
>> Aaron
>> 
>> ------------------------------------------------------------------------------
>> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
>> Instantly run your Selenium tests across 300+ browser/OS combos.
>> Get unparalleled scalability from the best Selenium testing platform
>> available
>> Simple to use. Nothing to install. Get started now for free."
>> http://p.sf.net/sfu/SauceLabs
>> _______________________________________________
>> Postgres-xc-general mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>> 
> 
> ------------------------------------------------------------------------------
> Time is money. Stop wasting it! Get your web API in 5 minutes.
> www.restlet.com/download
> http://p.sf.net/sfu/restlet
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> 

I'm using 1.2.1 with gcc 4.7 on Amazon i3 instances.  I'm actually becoming somewhat suspect of the i3 platform running this specific version of ubuntu.  So, I've recompiled the entire database using the intel compilers and I'm going to see if I see the same behavioral problems I've seen in the last month and a half.

I will let you know.

________________________________________
From: Koichi Suzuki [koi...@gm...]
Sent: Saturday, May 24, 2014 11:09 AM
To: Aaron Jackson
Cc: pos...@li...
Subject: Re: [Postgres-xc-general] Interday Degradation

Sorry for the late response.

What version are you using?   1.2.1 includes several fix for GTM connectivity.
---
Koichi Suzuki


2014-05-22 12:28 GMT-04:00 Aaron Jackson <aja...@re...>:
> Given my past experience with compiler issues, I'm a little hesitant to even
> report this.  That said, I have a three node cluster, each with a
> coordinator, data node and gtm proxy.  I have a standalone gtm instance
> without a slave.  Often, when I come in after the servers have been up for a
> while, I'm greeted with a variety of issues.
>
> There are several warnings in the coordinator and data node logs, that read
> "Do not have a GTM snapshot available" - I've discarded these as mostly
> benign for the moment.
>
> The coordinator is much worse..
>
> 30770 | 2014-05-22 15:53:06 UTC | ERROR:  current transaction is aborted,
> commands ignored until end of transaction block
> 30770 | 2014-05-22 15:53:06 UTC | STATEMENT:  DISCARD ALL
> 4560 | 2014-05-22 15:54:30 UTC | LOG:  failed to connect to Datanode
> 4560 | 2014-05-22 15:54:30 UTC | LOG:  failed to connect to Datanode
> 4560 | 2014-05-22 15:54:30 UTC | WARNING:  can not connect to node 16390
> 30808 | 2014-05-22 15:54:30 UTC | LOG:  failed to acquire connections
>
>
> Usually, I reset the coordinator and datanode and the world is happy again.
> However, it makes me somewhat concerned that I'm seeing these kinds of
> failures on a daily basis.  I wouldn't rule out the compiler again as it's
> been the reason for previous failures, but has anyone else seen anything
> like this??
>
> Aaron
>
> ------------------------------------------------------------------------------
> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> Instantly run your Selenium tests across 300+ browser/OS combos.
> Get unparalleled scalability from the best Selenium testing platform
> available
> Simple to use. Nothing to install. Get started now for free."
> http://p.sf.net/sfu/SauceLabs
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>

On 05/27/2014 07:33 PM, Koichi Suzuki wrote:
> We can reduce the additional latency by performing prepare and commit
> in parallel, I mean, sending command to all the target remote nodes
> first and then receive their response afterwords.
> 
> As I supposed, alternative is to use BDR.   This has very small
> overhead.   We can detect conflicting writes among transactions and if
> each transaction does not conflict, we can apply these writes in
> parallel, not in a single thread as we're doing in streaming
> replication.

Mind you, like BDR itself, we'll still need to figure out how to handle DDL.

> 
> This needs some more work and I think this worth to spend some time.

Yes.  Otherwise we have two unpalatable choices:

- Massive data loss (roll back to barrier) every time we lose a node, or
- Doubling write latency (at least)

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

We started noticing outages with one of the nodes in our cluster this morning.  When I looked at the box, gtm proxy was running hot.  So, I looked at the logs of the GTM proxy - which was spooling rapidly.

LOCATION:  pgxcnode_add_info, register_common.c:249
1:139746454378240:2014-05-28 16:06:44.641 UTC -LOG:  Node with the given ID number already exists
LOCATION:  pgxcnode_add_info, register_common.c:249
1:139746445985536:2014-05-28 16:06:44.641 UTC -LOG:  Node with the given ID number already exists
LOCATION:  pgxcnode_add_info, register_common.c:249
1:139746454378240:2014-05-28 16:06:44.641 UTC -LOG:  Node with the given ID number already exists
LOCATION:  pgxcnode_add_info, register_common.c:249
1:139746445985536:2014-05-28 16:06:44.641 UTC -LOG:  Node with the given ID number already exists
LOCATION:  pgxcnode_add_info, register_common.c:249

After several restarts, it was painfully obvious that this proxy was not going to start.  In an attempt to recover the GTM proxy, I saved the GTM proxy directory and rebuilt it from scratch expecting to rebuild its state from the master.  No dice, here's what the data coordinator reported.

26885 | 2014-05-28 16:01:49 UTC | LOG:  autovacuum launcher started
26885 | 2014-05-28 16:02:50 UTC | ERROR:  GTM error, could not obtain snapshot XID = 1193752
26885 | 2014-05-28 16:03:51 UTC | WARNING:  Xid is invalid.
26885 | 2014-05-28 16:04:31 UTC | WARNING:  Xid is invalid.
26863 | 2014-05-28 16:04:49 UTC | FATAL:  Can not register Coordinator on GTM
26885 | 2014-05-28 16:05:31 UTC | WARNING:  Xid is invalid.
26885 | 2014-05-28 16:05:31 UTC | ERROR:  GTM error, could not obtain snapshot XID = 0
26885 | 2014-05-28 16:06:12 UTC | WARNING:  Xid is invalid.

I believe the first portion of the log occurred when the autovacuum tried to connect to the new proxy.

Here's what I'd like to know.  Clearly the GTM proxy failure wasn't expected, but the problem here is one of recoverability.  How does one go about getting the coordinator and data node to a sane place given that the old GTM proxy was in an unrecoverable state and a new one can't seemingly be put in place to replace it?

Aaron

I have a few questions about Postgres-XC that I couldn't find answers to in the documentation or with google searches:

- How does postgres-xc work with extensions, specifically PGQ? Do they work? Where do they "run?" on the data nodes or on the Coordinator?

- Similarly, how/where do user-defined functions get installed/run?

- And can user-defined functions perform DML and DDL?

- Can we shard on schema? (i.e. put some schemas in one location and other schemas in different locations, sharded the same way rows are?) Or if not, can we shard one table, and have functions execute on the same node as some controlling row?

Thanks,
-Alan

We can reduce the additional latency by performing prepare and commit
in parallel, I mean, sending command to all the target remote nodes
first and then receive their response afterwords.

As I supposed, alternative is to use BDR.   This has very small
overhead.   We can detect conflicting writes among transactions and if
each transaction does not conflict, we can apply these writes in
parallel, not in a single thread as we're doing in streaming
replication.

This needs some more work and I think this worth to spend some time.

Regards;
---
Koichi Suzuki


2014-05-28 9:50 GMT+09:00 Michael Paquier <mic...@gm...>:
> On Wed, May 28, 2014 at 8:42 AM, Josh Berkus <jo...@ag...> wrote:
>> On 05/26/2014 05:10 PM, Tim Uckun wrote:
>>> Would it be possible to keep the shards in multiple data nodes so that if
>>> one data node failed you'd just replace it when you can get around to it.
>>
>> Right, that's what I was describing.  It's the standard way to deal with
>> redundancy in any parallel sharded database.
>>
>> The drawback is that it increases write times, because you can't count a
>> write as saved unless all shards return.  This is why some databases
>> (e.g. Cassandra) go one step further and have, say, each shard in 3
>> places and only 2 need to return.  That gets way complicated though.
> That's an additional overhead for 2PC as you need to wait for all the
> nodes where a write occurred to have executed PREPARE. This also drops
> any hopes to have autocommit write transactions, sinking performance
> down particularly for OLTP apps, smth that XC plays well with by
> design.
> --
> Michael

On Wed, May 28, 2014 at 8:42 AM, Josh Berkus <jo...@ag...> wrote:
> On 05/26/2014 05:10 PM, Tim Uckun wrote:
>> Would it be possible to keep the shards in multiple data nodes so that if
>> one data node failed you'd just replace it when you can get around to it.
>
> Right, that's what I was describing.  It's the standard way to deal with
> redundancy in any parallel sharded database.
>
> The drawback is that it increases write times, because you can't count a
> write as saved unless all shards return.  This is why some databases
> (e.g. Cassandra) go one step further and have, say, each shard in 3
> places and only 2 need to return.  That gets way complicated though.
That's an additional overhead for 2PC as you need to wait for all the
nodes where a write occurred to have executed PREPARE. This also drops
any hopes to have autocommit write transactions, sinking performance
down particularly for OLTP apps, smth that XC plays well with by
design.
-- 
Michael

On 05/26/2014 05:10 PM, Tim Uckun wrote:
> Would it be possible to keep the shards in multiple data nodes so that if
> one data node failed you'd just replace it when you can get around to it.

Right, that's what I was describing.  It's the standard way to deal with
redundancy in any parallel sharded database.

The drawback is that it increases write times, because you can't count a
write as saved unless all shards return.  This is why some databases
(e.g. Cassandra) go one step further and have, say, each shard in 3
places and only 2 need to return.  That gets way complicated though.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com