postgres-xc-general Mailing List for Postgres-XC

On 25/10/12 08:37, Vladimir Stavrinov wrote:
> On Thu, Oct 25, 2012 at 2:05 AM, Vladimir Stavrinov
> <vst...@gm...> wrote:
>> On Wed, Oct 24, 2012 at 11:18:59PM +0300, Andrei Martsinchyk wrote:
>>>    one of those solutions. Everybody wins.  If XC integrates one
>>>    approach it will lose flexibility in this area.
>> and gain much more users.
> OK. Paulo don't wants more users, because he don't like easy ways and
> simple things. But we all want flexibility. Flexibility is good thing
> and here is example.

I didn't say "I don't want more users". I just believe, based on my
experience, that subjects as advanced as the ones we're discussing don't
come easy. And they shouldn't in the sense that people should really
learn/know about what they're doing, regarding clustering, HA, etc.!

>
> We have cluster consists of 4 nodes. Nodes organized in groups. All
> data distributed between groups and every group contains the identical
> data, i.e. replicas. In this case with such model we have 3 options:
>
> 1. Read scalability only with 4 replicas in group.
> 2. Read and write scalability with 2 replicas per group.
> 3. Write scalability only with 1 replica per group.
>
> It is obvious: with more nodes we have more options, i.e. more
> flexibility. It means here the trade off between read and write
> scalability. And we don't need for this "CREATE TABLE ... DISTRIBUTE
> BY ..." I think it is enough for most cases.
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_sfd2d_oct
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

-- 
Paulo Pires

I feel like the discussion is senseless. Everything costs its price. If
your need HA you pay with performance. If you need both HA and performance
you pay for more powerful hardware.
XC is for those, who want more TPS per dollar, under the circumstances HA
is not a first priority definitely. If you know how to implement HA
solution that does not affect performance please tell us.
There are a lot of useful features (like ability to start when server
starts, schedule backups, failover to standby system) which are out of the
core. If you want any of these your need to set it up or have someone do
that for you. If you do not need them you can go without them pretty well.

2012/10/25 Vladimir Stavrinov <vst...@gm...>

> On Thu, Oct 25, 2012 at 12:18 AM, Andrei Martsinchyk
> <and...@gm...> wrote:
>
> > I think your test was incorrect. It works.
>
> No, it is exactly what this thread started from and what indicated in
> its subject. See very first answer of developer: it is not even a bug,
> it is by design. Sounds like anecdote, but it is true.
>
> > performance scalability. They could use XC as is. If there is demand of
> HA
> > on market, other developers may create XC-based solutions, more or less
>
> Do You really have question about this? I think High Availability is
> priority number one because we are not very happy sitting in
> Rolls-Royce that can not move.
>

Nice. Rolls-Royce requires road, fuel, driver, service. If you do not
provide all these, you will be sitting in car that can not move. Why you
purchased it then?

-- 
Andrei Martsinchyk

StormDB - http://www.stormdb.com
The Database Cloud

On Thu, Oct 25, 2012 at 2:05 AM, Vladimir Stavrinov
<vst...@gm...> wrote:
> On Wed, Oct 24, 2012 at 11:18:59PM +0300, Andrei Martsinchyk wrote:

>>    one of those solutions. Everybody wins.  If XC integrates one
>>    approach it will lose flexibility in this area.
>
> and gain much more users.

OK. Paulo don't wants more users, because he don't like easy ways and
simple things. But we all want flexibility. Flexibility is good thing
and here is example.

We have cluster consists of 4 nodes. Nodes organized in groups. All
data distributed between groups and every group contains the identical
data, i.e. replicas. In this case with such model we have 3 options:

1. Read scalability only with 4 replicas in group.
2. Read and write scalability with 2 replicas per group.
3. Write scalability only with 1 replica per group.

It is obvious: with more nodes we have more options, i.e. more
flexibility. It means here the trade off between read and write
scalability. And we don't need for this "CREATE TABLE ... DISTRIBUTE
BY ..." I think it is enough for most cases.

On Thu, Oct 25, 2012 at 12:18 AM, Andrei Martsinchyk
<and...@gm...> wrote:

> I think your test was incorrect. It works.

No, it is exactly what this thread started from and what indicated in
its subject. See very first answer of developer: it is not even a bug,
it is by design. Sounds like anecdote, but it is true.

> performance scalability. They could use XC as is. If there is demand of HA
> on market, other developers may create XC-based solutions, more or less

Do You really have question about this? I think High Availability is
priority number one because we are not very happy sitting in
Rolls-Royce that can not move.

On Thu, Oct 25, 2012 at 5:43 AM, Michael Paquier
<mic...@gm...>wrote:

> On Thu, Oct 25, 2012 at 5:41 AM, David Hofstee <pg...@c0...> wrote:
>
>> **
>>
>> Hi,
>>
>> I've been reading the '*ERROR: Failed to get pooled connections*' thread
>> about what XC should and should not do. I opted to start a new thread
>> (instead of replying) about how I would like XC to be.
>>
>> Some background. I work for a SaaS company (mostly dev, some ops) which
>> has to be online 24/7. We are now running apache/tomcat/mysql for each set
>> of customers on about 30 nodes and we want to centralize and make our
>> application more robust, efficient and simple. It basically means creating
>> layers: LB, webservers, application servers, database cluster. Some easy
>> parts are already done (haproxy, nginx). Our 'platform' is pretty complex
>> and I have so many tasks, I prefer to *not* dig into details. We are now
>> discussing the db issue (mysql cluster is not that great).
>>
>> My dream DB cluster:
>>
> Scalability - that means read and write scalability. XC should do that
>> right now. Nice.
>>
>> High availability - a node can go offline and it should not hinder
>> availability (only processing capacity)
>>
>> Maintainability - Since maintenance/change is our primary cause of
>> downtime, it should be possible to kill a node and add it later. This can
>> be because the VM is being moved, the OS is updated/upgraded, etc. Also,
>> think about how a cluster is updated from major version to major version
>> (lets say 9.x to 10.x). Maybe that is not an issue (but I don't know about
>> it yet).
>>
>> Simplicity - It would be nice if the default package+config file is all I
>> need. If it is too complex I cannot go on holidays. Some points:
>>
>>    - I read that *'...even the stock postgresql.conf configuration file
>>    is pretty conservative and users tweak it as per their requirements...
>>    *'. For me that translates as 'if you are new to Postgres it works
>>    bad'. Not simple (for e.g. some of our dev-ers).
>>    - For HA* '...Like Postgres, you need an external application to
>>    provide it'*. When using a cluster I think HA is very often wanted. I
>>    need to explain all this to every ops-colleague of mine and some are not
>>    very accurate. Not simple again.
>>
>> XC is a fork of Postgres and we try to share the same philosophy as the
> parent project about being really conservative on the things that should or
> should not be added in core.
> For example, let's take the case of HA. It is of course possible to
> implement an HA solution directly in the core of XC, but there are 2 things
> that would go against that:
> 1) It is not our goal to oblige the users to user an HA solution or
> another, and I do not believe that it is the role of core people to
> integrate directly in XC core a solution that might be good for a certain
> type of applications, without caring of the other types of applications.
> Postgres is popular because it lets all the users free to use what they
> want, and depending on the application people want to use with XC, they
> might prefer an HA solution or another.
> 2) If in the future Postgres integrates a native HA solution (I do not
> believe it will be the case as the community is really conservative, but
> let's assume), and if XC had a some point integrated an HA solution
> directly in its core, we would certainly have to drop the XC solution and
> rely on the Postgres solution as XC is a fork of Postgres. This would be a
> waste of time for the core people who integrated the HA solution, and
> people merging Postgres code with XC. One of the reasons explaining that XC
> is able to keep up with Postgres code pace easily is that we avoid to
> implement solutions in core that might impact unnecessarily its
> interactions with Postgres.
>

+10. I totally agree with Michael here. We would like to keep XC's
footprint as small as possible. XC would add features for distributed
computing that will not be present in PG. Rest features would come from PG.
At the same time, we lack in terms of resources; and hence choose only few
things that look to be important from XC's perspective.

>
>>
>> Quick setup - I want to setup an NxM cluster quickly (N times duplication
>> for HA, M times distributed writes for performance). I prefer to setup a
>> single node with a given config file, add nodes and be ready to go. Maybe
>> an hour in case of disaster recovery?
>>
> There are already tools about that like this one written in Ruby:
>
> https://sourceforge.net/projects/postgres-xc/files/misc/pgxc_config_v0_9_3.tar.gz/download
> It is not maintained since 0.9.3 as this is not honestly a part of core.
> You might have a look at it.
>
>> Managability - I want to manage a cluster easily (add node, remove node,
>> spare nodes, monitoring, ...). It cannot be simple enough.
>>
> Sure. I don't know about any utilities able to do that, but if you could
> build a utility like this running on top of XC and sell it, well you might
> be able to make some money if XC becomes popular, what is not really the
> case now ;)
>
>> Backup - I'm not familiar with running backups on Postgres but we
>> currently run a blocking backup on the mysql, for consistency, and it
>> causes issues. We use Bacula on a file level. Which brings up a question:
>> How do you backup a cluster (if you don't know which nodes are hot)?
>>
> In the case of XC, you might directly take a dump from a Coordinator with
> pg_dump, and then restore the dump file with pg_restore. You might want to
> use archive files.
> There are many ways to accomplish that, like in Postgres. The only
> difference in the case of XC is that you need to do that for each node as
> architecture is shared nothing.
>
 Logging - Yes...
>>
>> Some may respond that things are not that simple. I know. But I still
>> want it to be simple. It would make PGXC a no-brainer for everyone. Thanks
>> for listening and keep up the good work! I appreciate it.
>>
> There are already utilities implemented for Postgres that can work
> natively with XC, like for logging you might want to use log analyzers like
> pgbadger.
> You should have a look at that first for each thing you want to do, then
> evaluate the effort necessary to achieve each of your goals.
>
> Thanks,
> --
> Michael Paquier
> http://michael.otacoo.com
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_sfd2d_oct
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
>


-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

On Wed, Oct 24, 2012 at 11:27:25PM +0100, Paulo Pires wrote:

> FYI there is technology that deprecates the need of rebooting a machine
> following a kernel update, such as ksplice (bought by Oracle a couple
> years ago).

There is such debian package but it is not commonly used.

> I believe you can add new machinery (new coordinators, new data-nodes)
> and deprecate old hardware. Am I being to simplistic thinking this way?
> Anyway, changing a cluster hardware every two years seems overkill to
> me. But of course, it depends on your app growth

We don't speak about upgrade here, it is about scalability, do you
remember?

> Yes, internal is (supposedly) easier or as you say "transparent" - I'd
> use the word "seamless". But you'll need to learn it and take care of it
> somehow, the same way you'd do with external solutions, such as haproxy
> or keepalived. I don't think HA/Clustering/LB is for the "heart faint".
> Whether you know what you're doing, or leave this matter alone! You'll
> save your sanity in the medium term..

If You know how automobile works it doesn't means You want to build it
just for Your own usage. But in our context, remember again, extra
complexity means not only extra software, but extra infrastructure, i.e.
extra hardware as well. I am using corosync, pacemaker, ipvs,
ldirectord, drbd and keepalvied. But here we are discussing database
cluster and it needs some other approach. I want to use some of such
tools for distributing requests between coordinators and for failover of
ipvs point of distribution and gtm.  But I don't want standby data
nodes. All nodes should be under load and there are should be enough
redundancy to survive any one node lost. Health monitoring and failover
should be done internally by XC in this case.

> I don't understand why you keep citing MySQL as an example. *Don't take
> me wrong here*, but if you feel it to be the right tool, just go with it

I've already explained this here twice: it is not right tool, because it
is in-memory database. But it has right clustering model and that is why
I cite it here as good exemplar.

> and leave the ones who think the same about Postgres-XC alone.

This is good tool to close any discussion about anything.

> Do you know anyone putting up a database cluster without
> HA/Clustering/LB knowledge? If you do, please ask them to stop.

This questing is not for me. Look cites above.

> If at least this was a "who has more users" competition, that would
> make sense.  The best tools I use in my day-to-day job didn't come
> easy! I don't agree with you on this, at all.

But I agree with You at this point. But it is not about "easy way" or
"more users". I don't think we should lose flexibility with clustering
model where distribution scheme defined on cluster level. I believe it
can include distribution on table level. So it may be default setting
issue.

Well designed complex things easy to use with default setting, but
still provides enough flexibility.

> I *only* had to change my biggest app DDL (which is generated by some
> Java JPA tool) in order to test DISTRIBUTE BY. But I'm good with 100%
> replication.. for now. In the end I made *zero* changes!

I don't see how this story helps in production environment.


***************************
###  Vladimir Stavrinov
###  vst...@gm...
***************************

On Thu, Oct 25, 2012 at 6:40 AM, Paulo Pires <pj...@ub...> wrote:

> Hi,
>
> Summing, I've found Postgres-XC to be quite easy to install and
> configure in a 3 coordinators + 3 data-nodes (GTM all over them and
> GTM-Proxy handling HA). A little Google and command-line did the trick
> in *a couple hours*!
>
> Now, the only downside for me is that Postgres-XC doesn't have a
> built-in way of load-balancing between coordinators. If the coordinator
> your app is pointing to goes down, your app goes down - your application
> can target all of them, but in my experience, your application will
> *always* target a host. So, ATM my solution is:
>   1) Define a DNS FQDN like coordinator.mydomain pointing to an IP
> (i.e., 10.0.0.1)
>   2) Point my app to work with that FQDN
>   3) On every coordinator, configure keepalived with one shared-IP
> (10.0.0.1)
>   4) Install haproxy in every coordinator and have it load-balance with
> the other coordinators
>
> This way, keep-alived will always choose the first coordinator (based on
> its priority) and then haproxy (running on that machine) will
> load-balance with others. If this coordinator goes down, the second host
> in keepalived priority list will replace it and not only is it a valid
> coordinator, but also it will be able to load-balance with other
> coordinators.
>
This looks like a possible solution trying to achieve load balancing easily
at Coordinator level. You could also publish a small utility for the XC
community based in your experience. That is only a suggestion to help
community, please understand that I do not force you publishing anything of
course.


> My only doubt is, if you get a data-node offline and then bring it up,
> will the data in that data-node be synchronized?
>
If the Datanode becomes offline for a certain reason, all the transactions
that should have run on it will fail at Coordinator level, so there is no
worries here about data synchronization normally.
It is btw recommended to have a standby node behing the one that failed if
the Datanode that failed cannot be recovered for a reason or another.


> And that's it. I'm in now way a DB-expert and I felt quite confused by
> reading the previous thread. But as a developer, Postgres-XC has been a
> huge upgrade for me. (Now, if only RETURN ID was to be implemented, mr.
> Abbas ;-)).
>
+1. Looking forward to seeing this feature ;-o


> Sorry for being a little off-topic, but wanted to share my _little_
> experience with this wonderful piece of software.
>
Thanks, I am convinced it is helpful for a lot of people
-- 
Michael Paquier
http://michael.otacoo.com

On Thu, Oct 25, 2012 at 5:41 AM, David Hofstee <pg...@c0...> wrote:

> **
>
> Hi,
>
> I've been reading the '*ERROR: Failed to get pooled connections*' thread
> about what XC should and should not do. I opted to start a new thread
> (instead of replying) about how I would like XC to be.
>
> Some background. I work for a SaaS company (mostly dev, some ops) which
> has to be online 24/7. We are now running apache/tomcat/mysql for each set
> of customers on about 30 nodes and we want to centralize and make our
> application more robust, efficient and simple. It basically means creating
> layers: LB, webservers, application servers, database cluster. Some easy
> parts are already done (haproxy, nginx). Our 'platform' is pretty complex
> and I have so many tasks, I prefer to *not* dig into details. We are now
> discussing the db issue (mysql cluster is not that great).
>
> My dream DB cluster:
>
Scalability - that means read and write scalability. XC should do that
> right now. Nice.
>
> High availability - a node can go offline and it should not hinder
> availability (only processing capacity)
>
> Maintainability - Since maintenance/change is our primary cause of
> downtime, it should be possible to kill a node and add it later. This can
> be because the VM is being moved, the OS is updated/upgraded, etc. Also,
> think about how a cluster is updated from major version to major version
> (lets say 9.x to 10.x). Maybe that is not an issue (but I don't know about
> it yet).
>
> Simplicity - It would be nice if the default package+config file is all I
> need. If it is too complex I cannot go on holidays. Some points:
>
>    - I read that *'...even the stock postgresql.conf configuration file
>    is pretty conservative and users tweak it as per their requirements...*'.
>    For me that translates as 'if you are new to Postgres it works bad'. Not
>    simple (for e.g. some of our dev-ers).
>    - For HA* '...Like Postgres, you need an external application to
>    provide it'*. When using a cluster I think HA is very often wanted. I
>    need to explain all this to every ops-colleague of mine and some are not
>    very accurate. Not simple again.
>
> XC is a fork of Postgres and we try to share the same philosophy as the
parent project about being really conservative on the things that should or
should not be added in core.
For example, let's take the case of HA. It is of course possible to
implement an HA solution directly in the core of XC, but there are 2 things
that would go against that:
1) It is not our goal to oblige the users to user an HA solution or
another, and I do not believe that it is the role of core people to
integrate directly in XC core a solution that might be good for a certain
type of applications, without caring of the other types of applications.
Postgres is popular because it lets all the users free to use what they
want, and depending on the application people want to use with XC, they
might prefer an HA solution or another.
2) If in the future Postgres integrates a native HA solution (I do not
believe it will be the case as the community is really conservative, but
let's assume), and if XC had a some point integrated an HA solution
directly in its core, we would certainly have to drop the XC solution and
rely on the Postgres solution as XC is a fork of Postgres. This would be a
waste of time for the core people who integrated the HA solution, and
people merging Postgres code with XC. One of the reasons explaining that XC
is able to keep up with Postgres code pace easily is that we avoid to
implement solutions in core that might impact unnecessarily its
interactions with Postgres.

>
>
> Quick setup - I want to setup an NxM cluster quickly (N times duplication
> for HA, M times distributed writes for performance). I prefer to setup a
> single node with a given config file, add nodes and be ready to go. Maybe
> an hour in case of disaster recovery?
>
There are already tools about that like this one written in Ruby:
https://sourceforge.net/projects/postgres-xc/files/misc/pgxc_config_v0_9_3.tar.gz/download
It is not maintained since 0.9.3 as this is not honestly a part of core.
You might have a look at it.

> Managability - I want to manage a cluster easily (add node, remove node,
> spare nodes, monitoring, ...). It cannot be simple enough.
>
Sure. I don't know about any utilities able to do that, but if you could
build a utility like this running on top of XC and sell it, well you might
be able to make some money if XC becomes popular, what is not really the
case now ;)

> Backup - I'm not familiar with running backups on Postgres but we
> currently run a blocking backup on the mysql, for consistency, and it
> causes issues. We use Bacula on a file level. Which brings up a question:
> How do you backup a cluster (if you don't know which nodes are hot)?
>
In the case of XC, you might directly take a dump from a Coordinator with
pg_dump, and then restore the dump file with pg_restore. You might want to
use archive files.
There are many ways to accomplish that, like in Postgres. The only
difference in the case of XC is that you need to do that for each node as
architecture is shared nothing.

> Logging - Yes...
>
> Some may respond that things are not that simple. I know. But I still want
> it to be simple. It would make PGXC a no-brainer for everyone. Thanks for
> listening and keep up the good work! I appreciate it.
>
There are already utilities implemented for Postgres that can work natively
with XC, like for logging you might want to use log analyzers like pgbadger.
You should have a look at that first for each thing you want to do, then
evaluate the effort necessary to achieve each of your goals.

Thanks,
-- 
Michael Paquier
http://michael.otacoo.com