postgres-xc-general Mailing List for Postgres-XC

Ok. Thank you!

--- Mensagem Original ---

De: "Koichi Suzuki" <koi...@gm...>
Enviado: 29 de maio de 2014 22:48
Para: "Bruno Cezar Oliveira" <bru...@ho...>
Cc: "Postgres-XC mailing list" <pos...@li...>
Assunto: Re: [Postgres-xc-general] New Datanode in real-time

You can download this information from sourceforge page
http://sourceforge.net/projects/postgres-xc/files/Pgxc_ctl_primer/

Please download Pgxc_ctl_primer_20140507.pdf and take a look at it.

This material describes configuration and operation of Postgres-XC
cluster using pgxc_ctl.   Also, this describes how each
configuration/operation works including datanode addition/removal.

Hope this helps.
---
Koichi Suzuki


2014-05-30 4:04 GMT+09:00 Bruno Cezar Oliveira <bru...@ho...>:
> Hi!
>
> i am studying the postgres-xc to use this database in the enterprise  i
> work.
> I can add a datanode in real-time? How? If not, how i can do this?
> Looking in the docs i could not find.
>
>
> Thanks!
>
>
> ------------------------------------------------------------------------------
> Time is money. Stop wasting it! Get your web API in 5 minutes.
> www.restlet.com/download
> http://p.sf.net/sfu/restlet
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>

You can download this information from sourceforge page
http://sourceforge.net/projects/postgres-xc/files/Pgxc_ctl_primer/

Please download Pgxc_ctl_primer_20140507.pdf and take a look at it.

This material describes configuration and operation of Postgres-XC
cluster using pgxc_ctl.   Also, this describes how each
configuration/operation works including datanode addition/removal.

Hope this helps.
---
Koichi Suzuki


2014-05-30 4:04 GMT+09:00 Bruno Cezar Oliveira <bru...@ho...>:
> Hi!
>
> i am studying the postgres-xc to use this database in the enterprise  i
> work.
> I can add a datanode in real-time? How? If not, how i can do this?
> Looking in the docs i could not find.
>
>
> Thanks!
>
>
> ------------------------------------------------------------------------------
> Time is money. Stop wasting it! Get your web API in 5 minutes.
> www.restlet.com/download
> http://p.sf.net/sfu/restlet
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>

XC replicates schema at all the nodes.   It has all the information how each table is sharded or replicated.   So, schema itself is not sharded but tables can be sharded.

Hope this makes sense.

Regards;
---
Koichi Suzuki

2014/05/29 20:27、Alan Bryant <ala...@hd...<mailto:ala...@hd...>> のメール：



- Can we shard on schema? (i.e. put some schemas in one location and other schemas in different locations, sharded the same way rows are?) Or if not, can we shard one table, and have functions execute on the same node as some controlling row?

No.

Interesting. This leaves us with the "one big table" approach, just spread across the data nodes. Aren't there performance issues with having massive individual tables? What techniques exist to help with this?

Partitioning is one option I guess, using a different (but compatible) scheme than the sharding key, perhaps a different bitmask on the same key.

Another question is upgrade. We use schemas primarily to make logical upgrades possible while online, but what about Postgres-XC itself... what is the upgrade model for Postgres-XC? (say, between major PostgreSQL versions)

Thanks!
-Alan

------------------------------------------------------------------------------
Time is money. Stop wasting it! Get your web API in 5 minutes.
www.restlet.com/download<http://www.restlet.com/download>
http://p.sf.net/sfu/restlet_______________________________________________
Postgres-xc-general mailing list
Pos...@li...
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

Hi!

i am studying the postgres-xc to use this database in the enterprise  i
work.
I can add a datanode in real-time? How? If not, how i can do this?
Looking in the docs i could not find.


Thanks!

- Can we shard on schema? (i.e. put some schemas in one location and other schemas in different locations, sharded the same way rows are?) Or if not, can we shard one table, and have functions execute on the same node as some controlling row?

No.

Interesting. This leaves us with the "one big table" approach, just spread across the data nodes. Aren't there performance issues with having massive individual tables? What techniques exist to help with this?

Partitioning is one option I guess, using a different (but compatible) scheme than the sharding key, perhaps a different bitmask on the same key.

Another question is upgrade. We use schemas primarily to make logical upgrades possible while online, but what about Postgres-XC itself... what is the upgrade model for Postgres-XC? (say, between major PostgreSQL versions)

Thanks!
-Alan

2014/05/29 12:29、Ashutosh Bapat <ash...@en...<mailto:ash...@en...>> のメール：




On Thu, May 29, 2014 at 6:41 AM, 鈴木 幸市 <ko...@in...<mailto:ko...@in...>> wrote:
2014/05/29 1:27、Josh Berkus <jo...@ag...<mailto:jo...@ag...>> のメール：

> On 05/27/2014 07:33 PM, Koichi Suzuki wrote:
>> We can reduce the additional latency by performing prepare and commit
>> in parallel, I mean, sending command to all the target remote nodes
>> first and then receive their response afterwords.
>>
>> As I supposed, alternative is to use BDR.   This has very small
>> overhead.   We can detect conflicting writes among transactions and if
>> each transaction does not conflict, we can apply these writes in
>> parallel, not in a single thread as we're doing in streaming
>> replication.
>
> Mind you, like BDR itself, we'll still need to figure out how to handle DDL.

Thank you for the info.   I believe this issue is shared with other use cases such as SLONY.   Are there any other discussion how to handle this?

>
>>
>> This needs some more work and I think this worth to spend some time.
>
> Yes.  Otherwise we have two unpalatable choices:
>
> - Massive data loss (roll back to barrier) every time we lose a node, or
> - Doubling write latency (at least)

In the case of statement-based redundancy, we need to determine what node to write first (at least) and this should be the same in all the nodes.   It is needed to handle conflicting wrights consistently.   This means the first write has to be done synchronously and the rest can be done asynchronously.   Because most of the i/o work is done at prepare/commit, I hope this does not impact the whole throughput or latency badly.


I am doubtful as to whether the eventual consistency scheme will work in XC. If we commit a change to two nodes, and do not wait for the third node and the third node is not able to apply the change, we will have an inconsistency on the third node. This inconsistency may not be repairable depending upon the failure to write to that node. If the two nodes fail, the data that we will retrieve from the third node would be inconsistent. So, probably there is no point in keeping the third copy. That means that the cluster can tolerate N failures, where N is the number of synchronous copies made.

The above is synchronous because prepares and commits are done in synchronous way, by receiving all the response at last.   No prepare/commit response is  left outstanding.   All prepare responses will be handled before subsequent commot/abort.   All commit/abort will be handled before beginning next transaction. The idea is to do things in parallel only when it makes sense.

When an error occurs in prepare, we can abort it and no inconsistency will be left.   If one of following commit fails, this is not expected but may happen when system level failure occurs, such node should be detached from the cluster and need repair.

If abort fails during aborting failed prepare, this is also a failure to the node and such node should be detached from the cluster.

One thing we should think here is: we don’t have good means to simply repair such failed node.   It seems to me that we have to copy everything from a model node.   This takes time.    Maybe we need something like pg_rewind in statement-based replication or other tool to make a replica quickly.

Regards;
---
Koichi Suzuki


Regards;
—
Koichi Suzuki
>
> --
> Josh Berkus
> PostgreSQL Experts Inc.
> http://pgexperts.com<http://pgexperts.com/>
>
> ------------------------------------------------------------------------------
> Time is money. Stop wasting it! Get your web API in 5 minutes.
> www.restlet.com/download<http://www.restlet.com/download>
> http://p.sf.net/sfu/restlet
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...<mailto:Pos...@li...>
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>

------------------------------------------------------------------------------
Time is money. Stop wasting it! Get your web API in 5 minutes.
www.restlet.com/download<http://www.restlet.com/download>
http://p.sf.net/sfu/restlet
_______________________________________________
Postgres-xc-general mailing list
Pos...@li...<mailto:Pos...@li...>
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general



--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

On Thu, May 29, 2014 at 6:41 AM, 鈴木 幸市 <ko...@in...> wrote:

> 2014/05/29 1:27、Josh Berkus <jo...@ag...> のメール：
>
> > On 05/27/2014 07:33 PM, Koichi Suzuki wrote:
> >> We can reduce the additional latency by performing prepare and commit
> >> in parallel, I mean, sending command to all the target remote nodes
> >> first and then receive their response afterwords.
> >>
> >> As I supposed, alternative is to use BDR.   This has very small
> >> overhead.   We can detect conflicting writes among transactions and if
> >> each transaction does not conflict, we can apply these writes in
> >> parallel, not in a single thread as we're doing in streaming
> >> replication.
> >
> > Mind you, like BDR itself, we'll still need to figure out how to handle
> DDL.
>
> Thank you for the info.   I believe this issue is shared with other use
> cases such as SLONY.   Are there any other discussion how to handle this?
>
> >
> >>
> >> This needs some more work and I think this worth to spend some time.
> >
> > Yes.  Otherwise we have two unpalatable choices:
> >
> > - Massive data loss (roll back to barrier) every time we lose a node, or
> > - Doubling write latency (at least)
>
> In the case of statement-based redundancy, we need to determine what node
> to write first (at least) and this should be the same in all the nodes.
> It is needed to handle conflicting wrights consistently.   This means the
> first write has to be done synchronously and the rest can be done
> asynchronously.   Because most of the i/o work is done at prepare/commit, I
> hope this does not impact the whole throughput or latency badly.
>
>
I am doubtful as to whether the eventual consistency scheme will work in
XC. If we commit a change to two nodes, and do not wait for the third node
and the third node is not able to apply the change, we will have an
inconsistency on the third node. This inconsistency may not be repairable
depending upon the failure to write to that node. If the two nodes fail,
the data that we will retrieve from the third node would be inconsistent.
So, probably there is no point in keeping the third copy. That means that
the cluster can tolerate N failures, where N is the number of synchronous
copies made.


> Regards;
> —
> Koichi Suzuki
> >
> > --
> > Josh Berkus
> > PostgreSQL Experts Inc.
> > http://pgexperts.com
> >
> >
> ------------------------------------------------------------------------------
> > Time is money. Stop wasting it! Get your web API in 5 minutes.
> > www.restlet.com/download
> > http://p.sf.net/sfu/restlet
> > _______________________________________________
> > Postgres-xc-general mailing list
> > Pos...@li...
> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> >
>
>
> ------------------------------------------------------------------------------
> Time is money. Stop wasting it! Get your web API in 5 minutes.
> www.restlet.com/download
> http://p.sf.net/sfu/restlet
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>



-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

2014/05/29 0:08、Alan Bryant <ala...@hd...<mailto:ala...@hd...>> のメール：


I have a few questions about Postgres-XC that I couldn't find answers to in the documentation or with google searches:

- How does postgres-xc work with extensions, specifically PGQ? Do they work? Where do they "run?" on the data nodes or on the Coordinator?

I’m afraid nobody tested it.   At least, you should build the binary with XC environment.   Some of internal structure is not binary-compatible with PostgreSQL.

They run either on coordinator/datanode, depending upon specific functions.   Basically, only immutable functions run at datanodes.   Others run at coordinators.


- Similarly, how/where do user-defined functions get installed/run?

As mentioned above, immutable functions run at coordinators.   Others run at datanodes.


- And can user-defined functions perform DML and DDL?

Yes.


- Can we shard on schema? (i.e. put some schemas in one location and other schemas in different locations, sharded the same way rows are?) Or if not, can we shard one table, and have functions execute on the same node as some controlling row?

No.

Thank you;
—
Koichi Suzuki


Thanks,
-Alan

------------------------------------------------------------------------------
Time is money. Stop wasting it! Get your web API in 5 minutes.
www.restlet.com/download<http://www.restlet.com/download>
http://p.sf.net/sfu/restlet_______________________________________________
Postgres-xc-general mailing list
Pos...@li...
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

As Josh Berkus announced, there were bunch of XC-related discussions
in PGCon 2014. Whole meeting was very supportive to XC.

Materials will be found at following pages:
https://wiki.postgresql.org/wiki/PgCon2014ClusterSummit
https://wiki.postgresql.org/wiki/Pgcon2014PostgresXCmeeting

Also, there are a couple of unconference discussions related to XC.
Unconference page is
https://wiki.postgresql.org/wiki/Pgcon2014unconference

Hope these meeting notes are available soon.

Postgres-XC meeting was attended by several major PostgreSQL community
members as well as people interested in XC in their applications.

Through these discussions, I found the following are very important:

* XC has sufficient feature.   Stability is the crucial issue.
Priority of merge with PG9.4 is not high.  Missing features such as
WHERE CURRENT OF are not used.
* XC community should be more open.   For example, roadmap discussion
has not been done in public.
* Concentrate on simple benchmark, such as DBT-1, DBT-2 and pgbench.
* It is important to share evaluation and user experience.

I appreciate all the people in PGCon sessions/meetings for such
productive and fruitful discussions.

Give these suggestions, I'd like to begin open discussion of
Postgres-XC roadmap for release 1.3 and beyond, hopefully this
weekend.    Current core members are revising current roadmap to
reflect these discussions.   This will be published as a material for
discussions/proposals/comments/etc.

Any more input is very welcome.

Thank you very much.
---
Koichi Suzuki

Thanks for the report.   In this case, you can do the following:

1. Stop all the nodes just in case.   Use gtm_ctl and pg_ctl.  Do not kill them.  (You will waste some sequence and XID value if you kill them).  If your cluster was configured with pgxc_ctl, use it to stop the cluster.
2. Remove register.node file at GTM’s working directory.
3. Restart everything.

I hope this fixes the issue.

1.2.1 included several fix for cases as you saw but may have some more to improve.

Sorry for your inconvenience.

Best;
---
Koichi Suzuki

2014/05/29 1:17、Aaron Jackson <aja...@re...<mailto:aja...@re...>> のメール：

We started noticing outages with one of the nodes in our cluster this morning.  When I looked at the box, gtm proxy was running hot.  So, I looked at the logs of the GTM proxy - which was spooling rapidly.

LOCATION:  pgxcnode_add_info, register_common.c:249
1:139746454378240:2014-05-28 16:06:44.641 UTC -LOG:  Node with the given ID number already exists
LOCATION:  pgxcnode_add_info, register_common.c:249
1:139746445985536:2014-05-28 16:06:44.641 UTC -LOG:  Node with the given ID number already exists
LOCATION:  pgxcnode_add_info, register_common.c:249
1:139746454378240:2014-05-28 16:06:44.641 UTC -LOG:  Node with the given ID number already exists
LOCATION:  pgxcnode_add_info, register_common.c:249
1:139746445985536:2014-05-28 16:06:44.641 UTC -LOG:  Node with the given ID number already exists
LOCATION:  pgxcnode_add_info, register_common.c:249

After several restarts, it was painfully obvious that this proxy was not going to start.  In an attempt to recover the GTM proxy, I saved the GTM proxy directory and rebuilt it from scratch expecting to rebuild its state from the master.  No dice, here's what the data coordinator reported.

26885 | 2014-05-28 16:01:49 UTC | LOG:  autovacuum launcher started
26885 | 2014-05-28 16:02:50 UTC | ERROR:  GTM error, could not obtain snapshot XID = 1193752
26885 | 2014-05-28 16:03:51 UTC | WARNING:  Xid is invalid.
26885 | 2014-05-28 16:04:31 UTC | WARNING:  Xid is invalid.
26863 | 2014-05-28 16:04:49 UTC | FATAL:  Can not register Coordinator on GTM
26885 | 2014-05-28 16:05:31 UTC | WARNING:  Xid is invalid.
26885 | 2014-05-28 16:05:31 UTC | ERROR:  GTM error, could not obtain snapshot XID = 0
26885 | 2014-05-28 16:06:12 UTC | WARNING:  Xid is invalid.

I believe the first portion of the log occurred when the autovacuum tried to connect to the new proxy.

Here's what I'd like to know.  Clearly the GTM proxy failure wasn't expected, but the problem here is one of recoverability.  How does one go about getting the coordinator and data node to a sane place given that the old GTM proxy was in an unrecoverable state and a new one can't seemingly be put in place to replace it?

Aaron
------------------------------------------------------------------------------
Time is money. Stop wasting it! Get your web API in 5 minutes.
www.restlet.com/download<http://www.restlet.com/download>
http://p.sf.net/sfu/restlet_______________________________________________
Postgres-xc-general mailing list
Pos...@li...
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

2014/05/29 1:27、Josh Berkus <jo...@ag...> のメール：

> On 05/27/2014 07:33 PM, Koichi Suzuki wrote:
>> We can reduce the additional latency by performing prepare and commit
>> in parallel, I mean, sending command to all the target remote nodes
>> first and then receive their response afterwords.
>> 
>> As I supposed, alternative is to use BDR.   This has very small
>> overhead.   We can detect conflicting writes among transactions and if
>> each transaction does not conflict, we can apply these writes in
>> parallel, not in a single thread as we're doing in streaming
>> replication.
> 
> Mind you, like BDR itself, we'll still need to figure out how to handle DDL.

Thank you for the info.   I believe this issue is shared with other use cases such as SLONY.   Are there any other discussion how to handle this?

> 
>> 
>> This needs some more work and I think this worth to spend some time.
> 
> Yes.  Otherwise we have two unpalatable choices:
> 
> - Massive data loss (roll back to barrier) every time we lose a node, or
> - Doubling write latency (at least)

In the case of statement-based redundancy, we need to determine what node to write first (at least) and this should be the same in all the nodes.   It is needed to handle conflicting wrights consistently.   This means the first write has to be done synchronously and the rest can be done asynchronously.   Because most of the i/o work is done at prepare/commit, I hope this does not impact the whole throughput or latency badly.

Regards;
—
Koichi Suzuki
> 
> -- 
> Josh Berkus
> PostgreSQL Experts Inc.
> http://pgexperts.com
> 
> ------------------------------------------------------------------------------
> Time is money. Stop wasting it! Get your web API in 5 minutes.
> www.restlet.com/download
> http://p.sf.net/sfu/restlet
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> 

If you have similar problem, please raise log_min_messages to DEBUG1 or higher.   This may give some more info of the problem.

Thank you;
---
Koichi Suzuki

2014/05/29 2:29、Aaron Jackson <aja...@re...> のメール：

> I'm using 1.2.1 with gcc 4.7 on Amazon i3 instances.  I'm actually becoming somewhat suspect of the i3 platform running this specific version of ubuntu.  So, I've recompiled the entire database using the intel compilers and I'm going to see if I see the same behavioral problems I've seen in the last month and a half.
> 
> I will let you know.
> 
> ________________________________________
> From: Koichi Suzuki [koi...@gm...]
> Sent: Saturday, May 24, 2014 11:09 AM
> To: Aaron Jackson
> Cc: pos...@li...
> Subject: Re: [Postgres-xc-general] Interday Degradation
> 
> Sorry for the late response.
> 
> What version are you using?   1.2.1 includes several fix for GTM connectivity.
> ---
> Koichi Suzuki
> 
> 
> 2014-05-22 12:28 GMT-04:00 Aaron Jackson <aja...@re...>:
>> Given my past experience with compiler issues, I'm a little hesitant to even
>> report this.  That said, I have a three node cluster, each with a
>> coordinator, data node and gtm proxy.  I have a standalone gtm instance
>> without a slave.  Often, when I come in after the servers have been up for a
>> while, I'm greeted with a variety of issues.
>> 
>> There are several warnings in the coordinator and data node logs, that read
>> "Do not have a GTM snapshot available" - I've discarded these as mostly
>> benign for the moment.
>> 
>> The coordinator is much worse..
>> 
>> 30770 | 2014-05-22 15:53:06 UTC | ERROR:  current transaction is aborted,
>> commands ignored until end of transaction block
>> 30770 | 2014-05-22 15:53:06 UTC | STATEMENT:  DISCARD ALL
>> 4560 | 2014-05-22 15:54:30 UTC | LOG:  failed to connect to Datanode
>> 4560 | 2014-05-22 15:54:30 UTC | LOG:  failed to connect to Datanode
>> 4560 | 2014-05-22 15:54:30 UTC | WARNING:  can not connect to node 16390
>> 30808 | 2014-05-22 15:54:30 UTC | LOG:  failed to acquire connections
>> 
>> 
>> Usually, I reset the coordinator and datanode and the world is happy again.
>> However, it makes me somewhat concerned that I'm seeing these kinds of
>> failures on a daily basis.  I wouldn't rule out the compiler again as it's
>> been the reason for previous failures, but has anyone else seen anything
>> like this??
>> 
>> Aaron
>> 
>> ------------------------------------------------------------------------------
>> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
>> Instantly run your Selenium tests across 300+ browser/OS combos.
>> Get unparalleled scalability from the best Selenium testing platform
>> available
>> Simple to use. Nothing to install. Get started now for free."
>> http://p.sf.net/sfu/SauceLabs
>> _______________________________________________
>> Postgres-xc-general mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>> 
> 
> ------------------------------------------------------------------------------
> Time is money. Stop wasting it! Get your web API in 5 minutes.
> www.restlet.com/download
> http://p.sf.net/sfu/restlet
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> 

I'm using 1.2.1 with gcc 4.7 on Amazon i3 instances.  I'm actually becoming somewhat suspect of the i3 platform running this specific version of ubuntu.  So, I've recompiled the entire database using the intel compilers and I'm going to see if I see the same behavioral problems I've seen in the last month and a half.

I will let you know.

________________________________________
From: Koichi Suzuki [koi...@gm...]
Sent: Saturday, May 24, 2014 11:09 AM
To: Aaron Jackson
Cc: pos...@li...
Subject: Re: [Postgres-xc-general] Interday Degradation

Sorry for the late response.

What version are you using?   1.2.1 includes several fix for GTM connectivity.
---
Koichi Suzuki


2014-05-22 12:28 GMT-04:00 Aaron Jackson <aja...@re...>:
> Given my past experience with compiler issues, I'm a little hesitant to even
> report this.  That said, I have a three node cluster, each with a
> coordinator, data node and gtm proxy.  I have a standalone gtm instance
> without a slave.  Often, when I come in after the servers have been up for a
> while, I'm greeted with a variety of issues.
>
> There are several warnings in the coordinator and data node logs, that read
> "Do not have a GTM snapshot available" - I've discarded these as mostly
> benign for the moment.
>
> The coordinator is much worse..
>
> 30770 | 2014-05-22 15:53:06 UTC | ERROR:  current transaction is aborted,
> commands ignored until end of transaction block
> 30770 | 2014-05-22 15:53:06 UTC | STATEMENT:  DISCARD ALL
> 4560 | 2014-05-22 15:54:30 UTC | LOG:  failed to connect to Datanode
> 4560 | 2014-05-22 15:54:30 UTC | LOG:  failed to connect to Datanode
> 4560 | 2014-05-22 15:54:30 UTC | WARNING:  can not connect to node 16390
> 30808 | 2014-05-22 15:54:30 UTC | LOG:  failed to acquire connections
>
>
> Usually, I reset the coordinator and datanode and the world is happy again.
> However, it makes me somewhat concerned that I'm seeing these kinds of
> failures on a daily basis.  I wouldn't rule out the compiler again as it's
> been the reason for previous failures, but has anyone else seen anything
> like this??
>
> Aaron
>
> ------------------------------------------------------------------------------
> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> Instantly run your Selenium tests across 300+ browser/OS combos.
> Get unparalleled scalability from the best Selenium testing platform
> available
> Simple to use. Nothing to install. Get started now for free."
> http://p.sf.net/sfu/SauceLabs
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>

On 05/27/2014 07:33 PM, Koichi Suzuki wrote:
> We can reduce the additional latency by performing prepare and commit
> in parallel, I mean, sending command to all the target remote nodes
> first and then receive their response afterwords.
> 
> As I supposed, alternative is to use BDR.   This has very small
> overhead.   We can detect conflicting writes among transactions and if
> each transaction does not conflict, we can apply these writes in
> parallel, not in a single thread as we're doing in streaming
> replication.

Mind you, like BDR itself, we'll still need to figure out how to handle DDL.

> 
> This needs some more work and I think this worth to spend some time.

Yes.  Otherwise we have two unpalatable choices:

- Massive data loss (roll back to barrier) every time we lose a node, or
- Doubling write latency (at least)

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

We started noticing outages with one of the nodes in our cluster this morning.  When I looked at the box, gtm proxy was running hot.  So, I looked at the logs of the GTM proxy - which was spooling rapidly.

LOCATION:  pgxcnode_add_info, register_common.c:249
1:139746454378240:2014-05-28 16:06:44.641 UTC -LOG:  Node with the given ID number already exists
LOCATION:  pgxcnode_add_info, register_common.c:249
1:139746445985536:2014-05-28 16:06:44.641 UTC -LOG:  Node with the given ID number already exists
LOCATION:  pgxcnode_add_info, register_common.c:249
1:139746454378240:2014-05-28 16:06:44.641 UTC -LOG:  Node with the given ID number already exists
LOCATION:  pgxcnode_add_info, register_common.c:249
1:139746445985536:2014-05-28 16:06:44.641 UTC -LOG:  Node with the given ID number already exists
LOCATION:  pgxcnode_add_info, register_common.c:249

After several restarts, it was painfully obvious that this proxy was not going to start.  In an attempt to recover the GTM proxy, I saved the GTM proxy directory and rebuilt it from scratch expecting to rebuild its state from the master.  No dice, here's what the data coordinator reported.

26885 | 2014-05-28 16:01:49 UTC | LOG:  autovacuum launcher started
26885 | 2014-05-28 16:02:50 UTC | ERROR:  GTM error, could not obtain snapshot XID = 1193752
26885 | 2014-05-28 16:03:51 UTC | WARNING:  Xid is invalid.
26885 | 2014-05-28 16:04:31 UTC | WARNING:  Xid is invalid.
26863 | 2014-05-28 16:04:49 UTC | FATAL:  Can not register Coordinator on GTM
26885 | 2014-05-28 16:05:31 UTC | WARNING:  Xid is invalid.
26885 | 2014-05-28 16:05:31 UTC | ERROR:  GTM error, could not obtain snapshot XID = 0
26885 | 2014-05-28 16:06:12 UTC | WARNING:  Xid is invalid.

I believe the first portion of the log occurred when the autovacuum tried to connect to the new proxy.

Here's what I'd like to know.  Clearly the GTM proxy failure wasn't expected, but the problem here is one of recoverability.  How does one go about getting the coordinator and data node to a sane place given that the old GTM proxy was in an unrecoverable state and a new one can't seemingly be put in place to replace it?

Aaron

I have a few questions about Postgres-XC that I couldn't find answers to in the documentation or with google searches:

- How does postgres-xc work with extensions, specifically PGQ? Do they work? Where do they "run?" on the data nodes or on the Coordinator?

- Similarly, how/where do user-defined functions get installed/run?

- And can user-defined functions perform DML and DDL?

- Can we shard on schema? (i.e. put some schemas in one location and other schemas in different locations, sharded the same way rows are?) Or if not, can we shard one table, and have functions execute on the same node as some controlling row?

Thanks,
-Alan

We can reduce the additional latency by performing prepare and commit
in parallel, I mean, sending command to all the target remote nodes
first and then receive their response afterwords.

As I supposed, alternative is to use BDR.   This has very small
overhead.   We can detect conflicting writes among transactions and if
each transaction does not conflict, we can apply these writes in
parallel, not in a single thread as we're doing in streaming
replication.

This needs some more work and I think this worth to spend some time.

Regards;
---
Koichi Suzuki


2014-05-28 9:50 GMT+09:00 Michael Paquier <mic...@gm...>:
> On Wed, May 28, 2014 at 8:42 AM, Josh Berkus <jo...@ag...> wrote:
>> On 05/26/2014 05:10 PM, Tim Uckun wrote:
>>> Would it be possible to keep the shards in multiple data nodes so that if
>>> one data node failed you'd just replace it when you can get around to it.
>>
>> Right, that's what I was describing.  It's the standard way to deal with
>> redundancy in any parallel sharded database.
>>
>> The drawback is that it increases write times, because you can't count a
>> write as saved unless all shards return.  This is why some databases
>> (e.g. Cassandra) go one step further and have, say, each shard in 3
>> places and only 2 need to return.  That gets way complicated though.
> That's an additional overhead for 2PC as you need to wait for all the
> nodes where a write occurred to have executed PREPARE. This also drops
> any hopes to have autocommit write transactions, sinking performance
> down particularly for OLTP apps, smth that XC plays well with by
> design.
> --
> Michael

On Wed, May 28, 2014 at 8:42 AM, Josh Berkus <jo...@ag...> wrote:
> On 05/26/2014 05:10 PM, Tim Uckun wrote:
>> Would it be possible to keep the shards in multiple data nodes so that if
>> one data node failed you'd just replace it when you can get around to it.
>
> Right, that's what I was describing.  It's the standard way to deal with
> redundancy in any parallel sharded database.
>
> The drawback is that it increases write times, because you can't count a
> write as saved unless all shards return.  This is why some databases
> (e.g. Cassandra) go one step further and have, say, each shard in 3
> places and only 2 need to return.  That gets way complicated though.
That's an additional overhead for 2PC as you need to wait for all the
nodes where a write occurred to have executed PREPARE. This also drops
any hopes to have autocommit write transactions, sinking performance
down particularly for OLTP apps, smth that XC plays well with by
design.
-- 
Michael

On 05/26/2014 05:10 PM, Tim Uckun wrote:
> Would it be possible to keep the shards in multiple data nodes so that if
> one data node failed you'd just replace it when you can get around to it.

Right, that's what I was describing.  It's the standard way to deal with
redundancy in any parallel sharded database.

The drawback is that it increases write times, because you can't count a
write as saved unless all shards return.  This is why some databases
(e.g. Cassandra) go one step further and have, say, each shard in 3
places and only 2 need to return.  That gets way complicated though.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

As mentioned, counting the line of code was done in very rough way.
I'd like to make comparison a bit carefully and make a correction on
it.

30k is very small estimation.   At least, gtm has more than 10k,
pgxc_ctl more than 10k, and many others including pooler, FQS,
planner, executor, DDL, and so on.

Regards;
---
Koichi Suzuki


2014-05-27 8:37 GMT+09:00 Michael Paquier <mic...@gm...>:
> On Tue, May 27, 2014 at 3:55 AM, Josh Berkus <jo...@ag...> wrote:
>> All:
>>
>> The meeting notes from last week's meeting are up on the wiki.  If you
>> presented slides, please link them from the wiki page.
> With a link it is better:
> https://wiki.postgresql.org/wiki/Pgcon2014PostgresXCmeeting
> I cannot really believe that XC has 470k of diffs with upstream
> Postgres. With my last estimations it was closer to 30k.
> --
> Michael
>
> ------------------------------------------------------------------------------
> The best possible search technologies are now affordable for all companies.
> Download your FREE open source Enterprise Search Engine today!
> Our experts will assist you in its installation for $59/mo, no commitment.
> Test it for FREE on our Cloud platform anytime!
> http://pubads.g.doubleclick.net/gampad/clk?id=145328191&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

Year, this is what needed now.   I think we should make some more
discussion on it to get the best approach.

I understood that streaming replication is not for this usecase and we
need something better.

As Josh suggested, I'd like to have much more idea/requirement on this.

Thank you;
---
Koichi Suzuki


2014-05-27 9:10 GMT+09:00 Tim Uckun <tim...@gm...>:
> Would it be possible to keep the shards in multiple data nodes so that if
> one data node failed you'd just replace it when you can get around to it.
>
> Elasticsearch uses this strategy.
>
>
> On Sun, May 25, 2014 at 8:04 AM, Koichi Suzuki <koi...@gm...>
> wrote:
>>
>> At present, XC advises to make a replica with synchronize replication.
>>   Pgxc_ctl configures slaves in this way.
>>
>> I understand that this is not for performance and we may need some
>> other solution for this.
>>
>> To begin with, there are a couple of ideas for this.
>>
>> 1. To allow async., when a node fails, fall back whole cluster status
>> to the latest consistent state, such as pointed by a barrier.   I can
>> provide some detailed thought on this if interesting.
>>
>> 2. Allow to have a copy of shards to another node at planner/executor
>> level.
>>
>> 3. Implement another replication better for XC using BDR, just for
>> distributed tables, for example.
>>
>> At present, XC uses hash value of the node name to determine each row
>> location for distributed tables.   For ideas 2 and 3, we need to add
>> some infrastructure to make this allocation more flexible.
>>
>> Further input is welcome.
>>
>> Thank you.
>> ---
>> Koichi Suzuki
>>
>>
>> 2014-05-24 14:53 GMT-04:00 Josh Berkus <jo...@ag...>:
>> > All:
>> >
>> > So, in addition to the stability issues raised at the PostgresXC summit,
>> > I need to raise something which is a deficiency of both XC and XL and
>> > should be (in my opinion) our #2 priority after stability. And that's
>> > node/shard redundancy.
>> >
>> > Right now, if single node fails, the cluster is frozen for writes ...
>> > and fails some reads ... until the node is replaced by the user from a
>> > replica.  It's also not clear that we *can* actually replace a node from
>> > a replica because the replica will be async rep, and thus not at exactly
>> > the same GXID as the rest of the cluster.  This makes XC a
>> > low-availability solution.
>> >
>> > The answer for this is to do the same thing which every other clustering
>> > system has done: write each shard to multiple locations.  Default would
>> > be two.  If each shard is present on two different nodes, then losing a
>> > node is just a performance problem, not a downtime event.
>> >
>> > Thoughts?
>> >
>> > --
>> > Josh Berkus
>> > PostgreSQL Experts Inc.
>> > http://pgexperts.com
>> >
>> >
>> > ------------------------------------------------------------------------------
>> > "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
>> > Instantly run your Selenium tests across 300+ browser/OS combos.
>> > Get unparalleled scalability from the best Selenium testing platform
>> > available
>> > Simple to use. Nothing to install. Get started now for free."
>> > http://p.sf.net/sfu/SauceLabs
>> > _______________________________________________
>> > Postgres-xc-general mailing list
>> > Pos...@li...
>> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>
>>
>> ------------------------------------------------------------------------------
>> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
>> Instantly run your Selenium tests across 300+ browser/OS combos.
>> Get unparalleled scalability from the best Selenium testing platform
>> available
>> Simple to use. Nothing to install. Get started now for free."
>> http://p.sf.net/sfu/SauceLabs
>> _______________________________________________
>> Postgres-xc-general mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
>

Would it be possible to keep the shards in multiple data nodes so that if
one data node failed you'd just replace it when you can get around to it.

Elasticsearch uses this strategy.


On Sun, May 25, 2014 at 8:04 AM, Koichi Suzuki <koi...@gm...>wrote:

> At present, XC advises to make a replica with synchronize replication.
>   Pgxc_ctl configures slaves in this way.
>
> I understand that this is not for performance and we may need some
> other solution for this.
>
> To begin with, there are a couple of ideas for this.
>
> 1. To allow async., when a node fails, fall back whole cluster status
> to the latest consistent state, such as pointed by a barrier.   I can
> provide some detailed thought on this if interesting.
>
> 2. Allow to have a copy of shards to another node at planner/executor
> level.
>
> 3. Implement another replication better for XC using BDR, just for
> distributed tables, for example.
>
> At present, XC uses hash value of the node name to determine each row
> location for distributed tables.   For ideas 2 and 3, we need to add
> some infrastructure to make this allocation more flexible.
>
> Further input is welcome.
>
> Thank you.
> ---
> Koichi Suzuki
>
>
> 2014-05-24 14:53 GMT-04:00 Josh Berkus <jo...@ag...>:
> > All:
> >
> > So, in addition to the stability issues raised at the PostgresXC summit,
> > I need to raise something which is a deficiency of both XC and XL and
> > should be (in my opinion) our #2 priority after stability. And that's
> > node/shard redundancy.
> >
> > Right now, if single node fails, the cluster is frozen for writes ...
> > and fails some reads ... until the node is replaced by the user from a
> > replica.  It's also not clear that we *can* actually replace a node from
> > a replica because the replica will be async rep, and thus not at exactly
> > the same GXID as the rest of the cluster.  This makes XC a
> > low-availability solution.
> >
> > The answer for this is to do the same thing which every other clustering
> > system has done: write each shard to multiple locations.  Default would
> > be two.  If each shard is present on two different nodes, then losing a
> > node is just a performance problem, not a downtime event.
> >
> > Thoughts?
> >
> > --
> > Josh Berkus
> > PostgreSQL Experts Inc.
> > http://pgexperts.com
> >
> >
> ------------------------------------------------------------------------------
> > "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> > Instantly run your Selenium tests across 300+ browser/OS combos.
> > Get unparalleled scalability from the best Selenium testing platform
> available
> > Simple to use. Nothing to install. Get started now for free."
> > http://p.sf.net/sfu/SauceLabs
> > _______________________________________________
> > Postgres-xc-general mailing list
> > Pos...@li...
> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
>
> ------------------------------------------------------------------------------
> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> Instantly run your Selenium tests across 300+ browser/OS combos.
> Get unparalleled scalability from the best Selenium testing platform
> available
> Simple to use. Nothing to install. Get started now for free."
> http://p.sf.net/sfu/SauceLabs
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>

On Tue, May 27, 2014 at 3:55 AM, Josh Berkus <jo...@ag...> wrote:
> All:
>
> The meeting notes from last week's meeting are up on the wiki.  If you
> presented slides, please link them from the wiki page.
With a link it is better:
https://wiki.postgresql.org/wiki/Pgcon2014PostgresXCmeeting
I cannot really believe that XC has 470k of diffs with upstream
Postgres. With my last estimations it was closer to 30k.
-- 
Michael

All:

The meeting notes from last week's meeting are up on the wiki.  If you
presented slides, please link them from the wiki page.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On 05/24/2014 04:02 PM, Koichi Suzuki wrote:
> 2014-05-24 17:10 GMT-04:00 Josh Berkus <jo...@ag...>:
>> Koichi,
>>
>>> 1. To allow async., when a node fails, fall back whole cluster status
>>> to the latest consistent state, such as pointed by a barrier.   I can
>>> provide some detailed thought on this if interesting.
>>
>> This is not interesting to me.  If I have to accept major data loss for
>> a single node failure, then I can use solutions which do not require an GTM.
>>
>>> 2. Allow to have a copy of shards to another node at planner/executor level.
>>
>> Yes.  This should be at the executor level, in my opinion.  All writes
>> go to all shards and do not complete until they all succeed or the shard
>> times out (and then is marked disabled).
>>
>> What to do with reads is more nuanced.  If we load-balance reads, then
>> we are increasing throughput of the cluster.  If we send each read to
>> all duplicate shards, then we are improving response times while
>> decreasing throughput.  I think that deserves some testing.
> 
> Planner needs some more to choose the best one which pushdown is the
> best path to do.   Also, to handle conflicting writes in different
> cooI'rdinators, we may need to define node priority where to go first.

I guess I'm not clear on how we could have a conflict in the first place?

As far as reads are concerned, I can only see two options:

1) Push read down to one random shard -- maximizes throughput

2) Push read down to all shards, take first response -- minimizes
response time

The choice of (1) or (2) is application-specific, so ultimately I think
we will need to implement both and allow the user to choose, maybe as a
config option.  The best functionality, of course, would be to provide
the user an option to choose as a userset GUC, so that they could switch
on a per-query basis.

>>
>>> 3. Implement another replication better for XC using BDR, just for
>>> distributed tables, for example.
>>
>> This has the same problems as solution #1.
> 
> We can implement better synchronization suitable for XC need.   Also,
> only shards can be replicated to reduce the overhead.   I think this
> has better potential than streaming replication.

Ah, ok.  When you get back home, maybe you can sketch out what you're
thinking with BDR?


-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com