postgres-xc-general Mailing List for Postgres-XC

thanks Ashutosh/Mason/Koichi. Really appreciate your quick and detailed reply.
 
I understand the HA aspect now. Btw, what is the role of XCM/xcwatch components? And which release are they planned for. For ex: is it being designed to not need things like corosync/pacemaker for HA?
 
>>> 4) Is GTM used for repeatable reads as well, apart from the Write transactions?
>Even when reading, one needs an XID and a snapshot obtained from GTM.

Here (for read-only xactions), is the XID obtained from GTM? Or generated locally?

Btw, as I understand it (the MVCC based approach in PG/XC to provide isolation) so far, commited-reads can be provided using local TM's, right? i.e a global-xaction gets local-xid's on the nodes it runs. So, GTM is required only to provide repeatable-reads. Pls clarify/correct.
Can you also share if you have any detailed document explaining the motivation/requirements for GTM and its design.

thanks a lot,
-Prasad
PS: see what we are doing everyday.. Stop animal cruelty: http://www.hsus.org/farm/multimedia/
http://www.vegsoc.org/info/
 

________________________________
 From: Mason <ma...@us...>
To: Ashutosh Bapat <ash...@en...> 
Cc: pos...@li...; va...@ro... 
Sent: Wednesday, February 29, 2012 7:50 PM
Subject: Re: [Postgres-xc-general] Few queries from Varprasad
  
On Wed, Feb 29, 2012 at 4:22 AM, Ashutosh Bapat
<ash...@en...> wrote:
> Hi All,
> One of my colleague in Sybase, came across my article in Linux For You
> magazine and sent following queries. I thought it better to discuss on XC
> general forum.
>
> Varprasad was earlier working with Sybase and has long experience with HA,
> Shared disk cluster and RDBMS in general.
>
> 1) what are the HA aspects of various component failures? What is current HA
> recommendations for various failures (coo, datanode or gtm etc)?

There is a GTM standby component, but HA is something that will be
worked on in the future. For now you could add replication to the
nodes, but you'll have to roll your own failover.

> 2) why GTM is desinged as centralized server? Is it possible to make it work
> in distributed manner so as to remove the bottleneck and SPOF?

It is centralized so that transaction ids and snapshots can be
assigned centrally.

Yes, it can eventually become a bottleneck. At what point, I don't
think we can say definitively. One thing that somewhat mitigates the
issue is that snapshots are grouped together and shared for multiple
snapshot requests from the same GTM Proxy. Different from PostgreSQL
is that the snapshot will contain the requestors' XIDs.

In PG 9.2 some work was done to reduce locking when getting snapshots.
We may want to try and apply the same here and see if it helps.

Anyway, even with more work done here to improve this, we will end up
with really large snapshots as more nodes are added. As the number of
connections increases and the number of open transactions increase,
when checking tuple visibility it will potentially have to look at
many more XIDs, slowing down performance some on all nodes. What
probably also has an impact is the global xmin.  If it is low because
of a long running transaction on one or two nodes, it is going to slow
down everybody because of more visibility checking on all nodes. We
have not measured this though, so I cannot say what the exact impact
is. We have done some things to minimize the impact of a long running
vacuum on one of the nodes.

As far as SPOF goes, there is a GTM Standby to help. It requires a GTM
Proxy though, so we could do something to make sure GTM proxy on
restarts if it crashes (daemonize or watchdog).

> 3) what is the expacted scale-up for very large number of nodes 50-100, esp
> considering that the scale-up reduces with larger number of nodes.

More research is needed.

> 4) Is GTM used for repeatable reads as well, apart from the Write
> transactions?

Even when reading, one needs an XID and a snapshot obtained from GTM.

> 5) Do all coordinators need to be alive to get this setup working esp for
> DDLs. We observed that connection pooling component gives error when the DDL
> is fired after a coordinator crashes.

I think more work could be done to make this flexible. We intended to
make coordinators not as critical of a component as the others. That
is, try and reach the point where if you lose some coordinators but
others are up, that is ok, we should be able to continue processing.
Also, we want to make it easy to spin up a new coordinator. We aren't
quite there yet though.

Regards,

Mason


>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Enterprise Postgres Company
>
>
> ------------------------------------------------------------------------------
> Virtualization & Cloud Management Using Capacity Planning
> Cloud computing makes use of virtualization - but cloud computing
> also focuses on allowing computing to be delivered as a service.
> http://www.accelacomm.com/jaw/sfnl/114/51521223/
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>

Streaming replication is available for each node of XC, other than
GTM.   Integration with specific HA middleware will be a future work.
  Also, GTM has its own backup to failover when GTM crashes.

GTM delivers global transaction id (GXID) to each transaction in XC,
as well as global snapshot (a set of GXIDs of the live transaction).
This is the essential mechanism to run each XC component in globally
consistent way and this is why GTM is implemented in a centralized
manner.     So far, I don't see how more than one GTM can work in
distributed way, as well as provide global transaction feature.

I agree that we should look into new locking and sharing snapshot in
9.2.   At least it will improve each node performance.   I'm not sure
if it is helpful to improve GTM inside.   GTM is build based upon
thread so locking mechanism is based upon pthread lock (not spinlock
and semaphore as done in PG).     Anyway, I do hope that we can use
many of PG9.2 improvements.      In fact, with GTM-proxy, XC backend
shares snapshot if snapshot requirements are obtained by GTM-Proxy in
a single scan.    Although we've not measured how many backends shares
a snapshot, because GTM network I/O becomes almost half with GTM-Proxy
involvement, I think at least two or more backends share snapshot in
average in DBT-1 benchmark with ten coordinators and datanodes.

As for the snapshot size, this is not XC specific.   This is an issue
of current MVCC algorithm and the issue shares its background with
PostgreSQL.     Snapshot size, for XC, could be compressed if the
value range of GXIDs in a snapshot is small enough, less than 256 for
example, hopefully.

Regards;
----------
Koichi Suzuki



2012/3/1 Mason <ma...@us...>:
> On Wed, Feb 29, 2012 at 4:22 AM, Ashutosh Bapat
> <ash...@en...> wrote:
>> Hi All,
>> One of my colleague in Sybase, came across my article in Linux For You
>> magazine and sent following queries. I thought it better to discuss on XC
>> general forum.
>>
>> Varprasad was earlier working with Sybase and has long experience with HA,
>> Shared disk cluster and RDBMS in general.
>>
>> 1) what are the HA aspects of various component failures? What is current HA
>> recommendations for various failures (coo, datanode or gtm etc)?
>
> There is a GTM standby component, but HA is something that will be
> worked on in the future. For now you could add replication to the
> nodes, but you'll have to roll your own failover.
>
>> 2) why GTM is desinged as centralized server? Is it possible to make it work
>> in distributed manner so as to remove the bottleneck and SPOF?
>
> It is centralized so that transaction ids and snapshots can be
> assigned centrally.
>
> Yes, it can eventually become a bottleneck. At what point, I don't
> think we can say definitively. One thing that somewhat mitigates the
> issue is that snapshots are grouped together and shared for multiple
> snapshot requests from the same GTM Proxy. Different from PostgreSQL
> is that the snapshot will contain the requestors' XIDs.
>
> In PG 9.2 some work was done to reduce locking when getting snapshots.
> We may want to try and apply the same here and see if it helps.
>
> Anyway, even with more work done here to improve this, we will end up
> with really large snapshots as more nodes are added. As the number of
> connections increases and the number of open transactions increase,
> when checking tuple visibility it will potentially have to look at
> many more XIDs, slowing down performance some on all nodes. What
> probably also has an impact is the global xmin.  If it is low because
> of a long running transaction on one or two nodes, it is going to slow
> down everybody because of more visibility checking on all nodes. We
> have not measured this though, so I cannot say what the exact impact
> is. We have done some things to minimize the impact of a long running
> vacuum on one of the nodes.
>
> As far as SPOF goes, there is a GTM Standby to help. It requires a GTM
> Proxy though, so we could do something to make sure GTM proxy on
> restarts if it crashes (daemonize or watchdog).
>
>> 3) what is the expacted scale-up for very large number of nodes 50-100, esp
>> considering that the scale-up reduces with larger number of nodes.
>
> More research is needed.
>
>> 4) Is GTM used for repeatable reads as well, apart from the Write
>> transactions?
>
> Even when reading, one needs an XID and a snapshot obtained from GTM.
>
>> 5) Do all coordinators need to be alive to get this setup working esp for
>> DDLs. We observed that connection pooling component gives error when the DDL
>> is fired after a coordinator crashes.
>
> I think more work could be done to make this flexible. We intended to
> make coordinators not as critical of a component as the others. That
> is, try and reach the point where if you lose some coordinators but
> others are up, that is ok, we should be able to continue processing.
> Also, we want to make it easy to spin up a new coordinator. We aren't
> quite there yet though.
>
> Regards,
>
> Mason
>
>
>>
>> --
>> Best Wishes,
>> Ashutosh Bapat
>> EntepriseDB Corporation
>> The Enterprise Postgres Company
>>
>>
>> ------------------------------------------------------------------------------
>> Virtualization & Cloud Management Using Capacity Planning
>> Cloud computing makes use of virtualization - but cloud computing
>> also focuses on allowing computing to be delivered as a service.
>> http://www.accelacomm.com/jaw/sfnl/114/51521223/
>> _______________________________________________
>> Postgres-xc-general mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>
>
> ------------------------------------------------------------------------------
> Virtualization & Cloud Management Using Capacity Planning
> Cloud computing makes use of virtualization - but cloud computing
> also focuses on allowing computing to be delivered as a service.
> http://www.accelacomm.com/jaw/sfnl/114/51521223/
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

On Wed, Feb 29, 2012 at 4:22 AM, Ashutosh Bapat
<ash...@en...> wrote:
> Hi All,
> One of my colleague in Sybase, came across my article in Linux For You
> magazine and sent following queries. I thought it better to discuss on XC
> general forum.
>
> Varprasad was earlier working with Sybase and has long experience with HA,
> Shared disk cluster and RDBMS in general.
>
> 1) what are the HA aspects of various component failures? What is current HA
> recommendations for various failures (coo, datanode or gtm etc)?

There is a GTM standby component, but HA is something that will be
worked on in the future. For now you could add replication to the
nodes, but you'll have to roll your own failover.

> 2) why GTM is desinged as centralized server? Is it possible to make it work
> in distributed manner so as to remove the bottleneck and SPOF?

It is centralized so that transaction ids and snapshots can be
assigned centrally.

Yes, it can eventually become a bottleneck. At what point, I don't
think we can say definitively. One thing that somewhat mitigates the
issue is that snapshots are grouped together and shared for multiple
snapshot requests from the same GTM Proxy. Different from PostgreSQL
is that the snapshot will contain the requestors' XIDs.

In PG 9.2 some work was done to reduce locking when getting snapshots.
We may want to try and apply the same here and see if it helps.

Anyway, even with more work done here to improve this, we will end up
with really large snapshots as more nodes are added. As the number of
connections increases and the number of open transactions increase,
when checking tuple visibility it will potentially have to look at
many more XIDs, slowing down performance some on all nodes. What
probably also has an impact is the global xmin.  If it is low because
of a long running transaction on one or two nodes, it is going to slow
down everybody because of more visibility checking on all nodes. We
have not measured this though, so I cannot say what the exact impact
is. We have done some things to minimize the impact of a long running
vacuum on one of the nodes.

As far as SPOF goes, there is a GTM Standby to help. It requires a GTM
Proxy though, so we could do something to make sure GTM proxy on
restarts if it crashes (daemonize or watchdog).

> 3) what is the expacted scale-up for very large number of nodes 50-100, esp
> considering that the scale-up reduces with larger number of nodes.

More research is needed.

> 4) Is GTM used for repeatable reads as well, apart from the Write
> transactions?

Even when reading, one needs an XID and a snapshot obtained from GTM.

> 5) Do all coordinators need to be alive to get this setup working esp for
> DDLs. We observed that connection pooling component gives error when the DDL
> is fired after a coordinator crashes.

I think more work could be done to make this flexible. We intended to
make coordinators not as critical of a component as the others. That
is, try and reach the point where if you lose some coordinators but
others are up, that is ok, we should be able to continue processing.
Also, we want to make it easy to spin up a new coordinator. We aren't
quite there yet though.

Regards,

Mason


>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Enterprise Postgres Company
>
>
> ------------------------------------------------------------------------------
> Virtualization & Cloud Management Using Capacity Planning
> Cloud computing makes use of virtualization - but cloud computing
> also focuses on allowing computing to be delivered as a service.
> http://www.accelacomm.com/jaw/sfnl/114/51521223/
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>