postgres-xc-general Mailing List for Postgres-XC (Page 4)

What they are looking for in the end is high-availability and load balancing. They want either system to be able to receive read and write requests, and for either one to be able to take over all the work if the other is down. My understanding is that postgres-xc is capable of this, it’s just a matter of getting there from where they are. All of this would be easier if their current database server weren’t running on one of the machines that needs to be in the two-machine cluster when we’re done.

And yes, minimizing downtime is a priority, too.

From: Mason Sharp 
Sent: Thursday, August 21, 2014 4:02 PM
To: Joseph Mays 
Cc: mailto:pos...@li... 
Subject: Re: [Postgres-xc-general] Move from a single-cluster machine to a multi cluster machine.





On Thu, Aug 21, 2014 at 3:04 PM, Joseph Mays <ma...@wi...> wrote:

  Okay, I have a postgres-xc “cluster” operating with only one machine in the cluster. What I want to do is load the database from the standard postgres cluster into it, then switch over to make it operational as the postgres server. Then I want to rebuild the original standard postgres machine to be another postgres-xc cluster machine and add it as a second machine to the cluster.
  There are still several things about this I have to figure out, but one question I don’t know the answer to is how I bring the two machines into “sync” at the database. Is there some way to add the second machine into the cluster and have it synchronize its database to the first machine in the cluster, or do I have to dump the database, get them running in a two machine cluster with a blank database, and the restore the dump into the two machine cluster?
  If the latter, if one machine goes down or something, so that the other is carrying all the load for a bit, how I do bring the two machines back into sync when the failed machine comes back up?

It sounds like you are trying to minimize downtime while you switchover.

If you were running on PostgreSQL 9.4, this may have been easier with the help of logical streaming replication. In the meantime, you may want to use a trigger-based scheme or Slony, where you enable triggers to track the changes, dump the data, load into XC and then play back those changes that were captured.

Regarding adding the other server back, if the load can handle it, you could already define it as a datanode and have data be loaded there. If you have a spare system and recent version of PostgreSQL, you could use streaming replication to it, promote it to being the master, then configure the other two systems.

Regards,


-- 

Mason Sharp

TransLattice - http://www.translattice.com
Distributed and Clustered Database Solutions

On Thu, Aug 21, 2014 at 3:04 PM, Joseph Mays <ma...@wi...> wrote:

>   Okay, I have a postgres-xc “cluster” operating with only one machine in
> the cluster. What I want to do is load the database from the standard
> postgres cluster into it, then switch over to make it operational as the
> postgres server. Then I want to rebuild the original standard postgres
> machine to be another postgres-xc cluster machine and add it as a second
> machine to the cluster.
>
>  There are still several things about this I have to figure out, but one
> question I don’t know the answer to is how I bring the two machines into
> “sync” at the database. Is there some way to add the second machine into
> the cluster and have it synchronize its database to the first machine in
> the cluster, or do I have to dump the database, get them running in a two
> machine cluster with a blank database, and the restore the dump into the
> two machine cluster?
>
>  If the latter, if one machine goes down or something, so that the other
> is carrying all the load for a bit, how I do bring the two machines back
> into sync when the failed machine comes back up?
>
>

It sounds like you are trying to minimize downtime while you switchover.

If you were running on PostgreSQL 9.4, this may have been easier with the
help of logical streaming replication. In the meantime, you may want to use
a trigger-based scheme or Slony, where you enable triggers to track the
changes, dump the data, load into XC and then play back those changes that
were captured.

Regarding adding the other server back, if the load can handle it, you
could already define it as a datanode and have data be loaded there. If you
have a spare system and recent version of PostgreSQL, you could use
streaming replication to it, promote it to being the master, then configure
the other two systems.

Regards,


-- 
Mason Sharp

TransLattice - http://www.translattice.com
Distributed and Clustered Database Solutions

Okay, I have a postgres-xc “cluster” operating with only one machine in the cluster. What I want to do is load the database from the standard postgres cluster into it, then switch over to make it operational as the postgres server. Then I want to rebuild the original standard postgres machine to be another postgres-xc cluster machine and add it as a second machine to the cluster.
There are still several things about this I have to figure out, but one question I don’t know the answer to is how I bring the two machines into “sync” at the database. Is there some way to add the second machine into the cluster and have it synchronize its database to the first machine in the cluster, or do I have to dump the database, get them running in a two machine cluster with a blank database, and the restore the dump into the two machine cluster?
If the latter, if one machine goes down or something, so that the other is carrying all the load for a bit, how I do bring the two machines back into sync when the failed machine comes back up?
Thank you for your patience and help.
Joe Mays

Thanks for your help! I did a dumpall out of the existing server and loaded it into the new “cluster”. It seemed to work fine, but I did not get this message until after, so I did not add any “Distribute By” statements to the CREATE TABLE commands. I will have to do the import again before turning up the server for real. I will try to do this on the next important.

My last two questions are about adding a server to the cluster, and startup commands.

First off, after I get the first server running as a clustered machine, I will be transferring all our sql client software over to it. Then I will be taking what was the original, reconfiguring it to be another machine in the cluster, and adding it to the cluster for a two-machine cluster.

Unless I am looking in the wrong place I have found very little info on how to add a machine to the cluster. Everything seems to assume that all the machines in the cluster are being created together. Does anyone know where I can find info on adding a machine, including configuring the second machine, adding it so it talks to the other, bring the databases into sync with each other, etc.?

Lastly, I am running under redhat, and I haven’t found anything about how to set up the software to start under chkconfig. It’s tricky, especially since you have 3 daemons to start (gtm, postgres datanode, and postgres coord) and I believe it matters what order they are started in.

From: Mason Sharp 
Sent: Monday, August 11, 2014 4:45 PM
To: Joseph Mays 
Subject: Re: [Postgres-xc-general] Install postgres-xc on redhat

Hi Joseph,




On Wed, Aug 6, 2014 at 4:47 PM, Joseph Mays <ma...@wi...> wrote:

  I did several searches of the entire file system for it, to no avail. I finally downloaded the source to 1.0.4 and built the binaries from the source code and installed, and initgtm as well as the other pieces are there now.

  What I am trying to do is turned a what was a slaved streaming backup of another pgsql server into a server in a postgres-xc cluster. When I have it operational, and have the database running on it, I’ll point things at it instead of the primary, then make what used to be the primary into a second server in this cluster.

  So, I a couple of questions. Is there any problem with using databases created under a standard postgres server on a postgres-xc cluster?

You would have to export it and then import it into Postgres-XC (or Postgres-XL).  You should spend some time on your CREATE TABLE statements to add the appropriate DISTRIBUTE BY clause; it may make a big difference in performance.


  Also it says gtm should not be run on the same server with the other components. Is this correct? 

You should be fine running it together if the number of servers is small and it meets your performance requirements.

  I have only two servers here, and I assumed I was going to be running gtm, as well as the other software, on both, so either could function as the primary or secondary at will. Is this a problem?

You can do that-- have datanode masters on server 1 and server 2, have server 1's datanode slave(s) on server 2, and vice versa.

Good luck!

-- 

Mason Sharp

TransLattice - http://www.translattice.com
Distributed and Clustered Database Solutions

1.1 has better performance and bug fixes compared to the previous release.


On Thu, Aug 7, 2014 at 5:13 PM, Joseph Mays <ma...@wi...> wrote:

>   Okay, I eventually got a working 1-server “cluster” running using
> postgres-xc 1.04. I think probably could 1.1 working now, but all the
> documentation, everything that exists, seems to be aimed at 1.04. The
> existence of 1.1 is barely acknowledge. Is there any reason to choose 1.1
> over 1.0.4? If so, should switch to that?
>
>

Regards,

-- 
Mason Sharp

TransLattice - http://www.translattice.com
Distributed and Clustered Database Solutions

Okay, I eventually got a working 1-server “cluster” running using postgres-xc 1.04. I think probably could 1.1 working now, but all the documentation, everything that exists, seems to be aimed at 1.04. The existence of 1.1 is barely acknowledge. Is there any reason to choose 1.1 over 1.0.4? If so, should switch to that?

On Wed, Aug 6, 2014 at 1:51 PM, Joseph Mays <ma...@wi...> wrote:

> I was able to download the binaries and install them; the install seemed to
> run fine, all libraries seemed to run fine, then I went to take the first
> step in the docs, running initgtm, only to find it's nowhere in the system.
>
> Here is the list of packages I installed. I don't think I missed anything
> --
> I installed every postgres-xc 1.1 package that existed, but can someone
> tell
> me if there is something that is not here?
>
> postgres-xc-1.1-10.1.x86_64.rpm
> postgres-xc-contrib-1.1-10.1.x86_64.rpm
> postgres-xc-debuginfo-1.1-10.1.x86_64.rpm
> postgres-xc-devel-1.1-10.1.x86_64.rpm
> postgres-xc-docs-1.1-10.1.x86_64.rpm
> postgres-xc-libs-1.1-10.1.x86_64.rpm
> postgres-xc-plperl-1.1-10.1.x86_64.rpm
> postgres-xc-plpython-1.1-10.1.x86_64.rpm
> postgres-xc-pltcl-1.1-10.1.x86_64.rpm
> postgres-xc-server-1.1-10.1.x86_64.rpm
>

My guess it is probably there but not in your $PATH. IIRC, it is somewhere
under /usr.

-- 
Mason Sharp

TransLattice - http://www.translattice.com
Distributed and Clustered Database Solutions

I was able to download the binaries and install them; the install seemed to 
run fine, all libraries seemed to run fine, then I went to take the first 
step in the docs, running initgtm, only to find it's nowhere in the system.

Here is the list of packages I installed. I don't think I missed anything --  
I installed every postgres-xc 1.1 package that existed, but can someone tell 
me if there is something that is not here?

postgres-xc-1.1-10.1.x86_64.rpm
postgres-xc-contrib-1.1-10.1.x86_64.rpm
postgres-xc-debuginfo-1.1-10.1.x86_64.rpm
postgres-xc-devel-1.1-10.1.x86_64.rpm
postgres-xc-docs-1.1-10.1.x86_64.rpm
postgres-xc-libs-1.1-10.1.x86_64.rpm
postgres-xc-plperl-1.1-10.1.x86_64.rpm
postgres-xc-plpython-1.1-10.1.x86_64.rpm
postgres-xc-pltcl-1.1-10.1.x86_64.rpm
postgres-xc-server-1.1-10.1.x86_64.rpm

-----Original Message----- 
From: Koichi Suzuki
Sent: Thursday, July 17, 2014 9:07 PM
To: Joseph Mays
Cc: Postgres-XC mailing list
Subject: Re: [Postgres-xc-general] Install postgres-xc on redhat

There's a page you can download XC rpm.   They provide only 1.0.x rpm 
though.

Try http://www.stormdb.com/content/postgres-xc-rpms-available

For later versions, you need to build binaries manually.

Regards;
---
Koichi Suzuki


2014-07-18 3:28 GMT+09:00 Joseph Mays <ma...@wi...>:
> Hi! Looking to install a postgres-xc cluster on two RHEL servers. Is there 
> a
> repo I can use to install the binaries from yum?
>
>
>
> ------------------------------------------------------------------------------
> Want fast and easy access to all the code in your enterprise? Index and
> search up to 200,000 lines of code with a free copy of Black Duck
> Code Sight - the same software that powers the world's largest code
> search on Ohloh, the Black Duck Open Hub! Try it now.
> http://p.sf.net/sfu/bds
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> 

Not sure about any RHEL repo.
You can download rpm from stromdb site.

RPM LINK: http://yum.stormdb.com/repos/Postgres-XC/

Regards,
Amul Sul

There's a page you can download XC rpm.   They provide only 1.0.x rpm though.

Try http://www.stormdb.com/content/postgres-xc-rpms-available

For later versions, you need to build binaries manually.

Regards;
---
Koichi Suzuki


2014-07-18 3:28 GMT+09:00 Joseph Mays <ma...@wi...>:
> Hi! Looking to install a postgres-xc cluster on two RHEL servers. Is there a
> repo I can use to install the binaries from yum?
>
>
>
> ------------------------------------------------------------------------------
> Want fast and easy access to all the code in your enterprise? Index and
> search up to 200,000 lines of code with a free copy of Black Duck
> Code Sight - the same software that powers the world's largest code
> search on Ohloh, the Black Duck Open Hub! Try it now.
> http://p.sf.net/sfu/bds
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>

Hi! Looking to install a postgres-xc cluster on two RHEL servers. Is there a repo I can use to install the binaries from yum?

Thank you Mason for the input.

I’d like to add one more.   If you have N datanodes and would like to allow M max connections to each coordinator, you should setup
N x M max connection to each datanode.

Regards;
---
Koichi Suzuki

2014/07/02 8:29、Mason Sharp <ms...@tr...<mailto:ms...@tr...>> のメール：



On Tue, Jul 1, 2014 at 6:59 AM, Wilkerson, Daniel <dwi...@fu...<mailto:dwi...@fu...>> wrote:
Hey everyone. I have my cluster up and running using pgxc_ctl instead of manually configuring (XC 1.1; Ubuntu 13.04). It's definitely easier to setup and control the cluster this way for sure. I have HAProxy balancing the requests to the servers. It's working fine also.

All of the nodes startup fine and everything seems ok on the surface; that is until I run the DDL script to create my database across all the nodes. During script execution I get, "ERROR: sorry, too many clients already". I've tried upping the number of available connections but still get the same issue across the cluster. I am executing the DDL against my first coordinator.

Could this be a mismatch in my config of the proportions between shared mem and max connections? I've queried the database and it doesn't report tons of connections. I'm not sure where to look in my config to suss out the problem.

I appreciate any light you all could shed on this. I'm fairly certain it's something I need to tweak in my config somewhere, but also want to make sure I've not run up against a bug.

Thanks!
dw


Connection values entered into the pgxc_ctl cluster config:
max_connections = 200
max_pool_size = 200

These should be appended in the individual nodes' postgresql.conf files.  I would double-check there, adding them manually if necessary.



This is a snippet from the script erring out.
…
CREATE TABLE
CREATE SEQUENCE
ALTER SEQUENCE
CREATE TABLE
CREATE SEQUENCE
ALTER SEQUENCE
CREATE TABLE
CREATE SEQUENCE
ALTER SEQUENCE
psql:db.sql:2654: WARNING:  unexpected EOF on datanode connection
psql:db.sql:2654: WARNING:  unexpected EOF on datanode connection
psql:db.sql:2654: ERROR:  sorry, too many clients already


This snippet from ps –ef | grep post on my first coordinator/datanode combo shows a hung create table statement:

postgre+  3608  3588  0 09:39 pts/1    00:00:00 psql -p 20004 -d postgres
postgre+  3616  2980  0 09:40 ?        00:00:00 postgres: postgres-xc db [local] CREATE TABLE
postgre+  3644  3291  0 09:41 ?        00:00:00 postgres: checkpointer process
postgre+  3645  3291  0 09:41 ?        00:00:00 postgres: writer process
postgre+  3646  3291  0 09:41 ?        00:00:00 postgres: wal writer process
postgre+  3647  3291  0 09:41 ?        00:00:00 postgres: autovacuum launcher process
postgre+  3648  3291  0 09:41 ?        00:00:00 postgres: stats collector process
postgre+  3705  2553  0 09:48 pts/0    00:00:00 ps -ef
postgre+  3706  2553  0 09:48 pts/0    00:00:00 grep post

OTOH, this does not look like very much activity at all... there should be plenty of connections.


--
Mason Sharp

TransLattice - http://www.translattice.com<http://www.translattice.com/>
Distributed and Clustered Database Solutions


------------------------------------------------------------------------------
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft_______________________________________________
Postgres-xc-general mailing list
Pos...@li...
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

On Tue, Jul 1, 2014 at 6:59 AM, Wilkerson, Daniel <dwi...@fu...>
wrote:

> Hey everyone. I have my cluster up and running using pgxc_ctl instead of
> manually configuring (XC 1.1; Ubuntu 13.04). It's definitely easier to
> setup and control the cluster this way for sure. I have HAProxy balancing
> the requests to the servers. It's working fine also.
>
> All of the nodes startup fine and everything seems ok on the surface; that
> is until I run the DDL script to create my database across all the nodes.
> During script execution I get, "ERROR: sorry, too many clients already".
> I've tried upping the number of available connections but still get the
> same issue across the cluster. I am executing the DDL against my first
> coordinator.
>
> Could this be a mismatch in my config of the proportions between shared
> mem and max connections? I've queried the database and it doesn't report
> tons of connections. I'm not sure where to look in my config to suss out
> the problem.
>
> I appreciate any light you all could shed on this. I'm fairly certain it's
> something I need to tweak in my config somewhere, but also want to make
> sure I've not run up against a bug.
>
> Thanks!
> dw
>
>
> Connection values entered into the pgxc_ctl cluster config:
> max_connections = 200
> max_pool_size = 200
>

These should be appended in the individual nodes' postgresql.conf files.  I
would double-check there, adding them manually if necessary.



>
> This is a snippet from the script erring out.
> …
> CREATE TABLE
> CREATE SEQUENCE
> ALTER SEQUENCE
> CREATE TABLE
> CREATE SEQUENCE
> ALTER SEQUENCE
> CREATE TABLE
> CREATE SEQUENCE
> ALTER SEQUENCE
> psql:db.sql:2654: WARNING:  unexpected EOF on datanode connection
> psql:db.sql:2654: WARNING:  unexpected EOF on datanode connection
> psql:db.sql:2654: ERROR:  sorry, too many clients already
>
>
> This snippet from ps –ef | grep post on my first coordinator/datanode
> combo shows a hung create table statement:
>
> postgre+  3608  3588  0 09:39 pts/1    00:00:00 psql -p 20004 -d postgres
> postgre+  3616  2980  0 09:40 ?        00:00:00 postgres: postgres-xc db
> [local] CREATE TABLE
> postgre+  3644  3291  0 09:41 ?        00:00:00 postgres: checkpointer
> process
> postgre+  3645  3291  0 09:41 ?        00:00:00 postgres: writer process
> postgre+  3646  3291  0 09:41 ?        00:00:00 postgres: wal writer
> process
> postgre+  3647  3291  0 09:41 ?        00:00:00 postgres: autovacuum
> launcher process
> postgre+  3648  3291  0 09:41 ?        00:00:00 postgres: stats collector
> process
> postgre+  3705  2553  0 09:48 pts/0    00:00:00 ps -ef
> postgre+  3706  2553  0 09:48 pts/0    00:00:00 grep post
>

OTOH, this does not look like very much activity at all... there should be
plenty of connections.


-- 
Mason Sharp

TransLattice - http://www.translattice.com
Distributed and Clustered Database Solutions

Hey everyone. I have my cluster up and running using pgxc_ctl instead of manually configuring (XC 1.1; Ubuntu 13.04). It's definitely easier to setup and control the cluster this way for sure. I have HAProxy balancing the requests to the servers. It's working fine also.

All of the nodes startup fine and everything seems ok on the surface; that is until I run the DDL script to create my database across all the nodes. During script execution I get, "ERROR: sorry, too many clients already". I've tried upping the number of available connections but still get the same issue across the cluster. I am executing the DDL against my first coordinator.

Could this be a mismatch in my config of the proportions between shared mem and max connections? I've queried the database and it doesn't report tons of connections. I'm not sure where to look in my config to suss out the problem.

I appreciate any light you all could shed on this. I'm fairly certain it's something I need to tweak in my config somewhere, but also want to make sure I've not run up against a bug.

Thanks!
dw


Connection values entered into the pgxc_ctl cluster config:
max_connections = 200
max_pool_size = 200

This is a snippet from the script erring out.
…
CREATE TABLE
CREATE SEQUENCE
ALTER SEQUENCE
CREATE TABLE
CREATE SEQUENCE
ALTER SEQUENCE
CREATE TABLE
CREATE SEQUENCE
ALTER SEQUENCE
psql:db.sql:2654: WARNING:  unexpected EOF on datanode connection
psql:db.sql:2654: WARNING:  unexpected EOF on datanode connection
psql:db.sql:2654: ERROR:  sorry, too many clients already


This snippet from ps –ef | grep post on my first coordinator/datanode combo shows a hung create table statement:

postgre+  3608  3588  0 09:39 pts/1    00:00:00 psql -p 20004 -d postgres
postgre+  3616  2980  0 09:40 ?        00:00:00 postgres: postgres-xc db [local] CREATE TABLE
postgre+  3644  3291  0 09:41 ?        00:00:00 postgres: checkpointer process
postgre+  3645  3291  0 09:41 ?        00:00:00 postgres: writer process
postgre+  3646  3291  0 09:41 ?        00:00:00 postgres: wal writer process
postgre+  3647  3291  0 09:41 ?        00:00:00 postgres: autovacuum launcher process
postgre+  3648  3291  0 09:41 ?        00:00:00 postgres: stats collector process
postgre+  3705  2553  0 09:48 pts/0    00:00:00 ps -ef
postgre+  3706  2553  0 09:48 pts/0    00:00:00 grep post

Hi Koichi,

Got it! Thanks a lot!

Thanks
Julian

> Date: Mon, 30 Jun 2014 14:06:21 +0900
> Subject: Re: [Postgres-xc-general] Which benchmark could be run successfully on PGXC?
> From: koi...@gm...
> To: jul...@ou...
> CC: pos...@li...
> 
> DBT-1 is available.   Please try
> https://sourceforge.net/projects/postgres-xc/files/dbt1/
> 
> Its GIT repository is at
> https://sourceforge.net/p/postgres-xc/dbt1/ci/master/tree/
> 
> Also, pgbench is available as a part of XC release.
> 
> Someone ran DBT-2 against XC but I think their code is not open yet.
> 
> Good luck;
> ---
> Koichi Suzuki
> 
> 
> 2014-06-30 12:31 GMT+09:00 ZhangJulian <jul...@ou...>:
> > Hi All,
> >
> > We are evaluating PGXC based on a general benchmark in database domain.
> >
> > First, we tested TPCC by using the tool BenchmarkSQL and HammerDB, but some
> > strange bugs block us.
> > We plan to use pgbench, do you know if it can be tesed smoothly on PGXC?
> > We also plan to use SwingBench, but it seems mostly used on Oracle. Anybody
> > has the experience on its testing on PGXC?
> >
> > And do you know others benchmark which could be tested on PGXC?
> >
> > Thanks
> > Julian
> >
> > ------------------------------------------------------------------------------
> > Open source business process management suite built on Java and Eclipse
> > Turn processes into business applications with Bonita BPM Community Edition
> > Quickly connect people, data, and systems into organized workflows
> > Winner of BOSSIE, CODIE, OW2 and Gartner awards
> > http://p.sf.net/sfu/Bonitasoft
> > _______________________________________________
> > Postgres-xc-general mailing list
> > Pos...@li...
> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> >
 		 	   		  

DBT-1 is available.   Please try
https://sourceforge.net/projects/postgres-xc/files/dbt1/

Its GIT repository is at
https://sourceforge.net/p/postgres-xc/dbt1/ci/master/tree/

Also, pgbench is available as a part of XC release.

Someone ran DBT-2 against XC but I think their code is not open yet.

Good luck;
---
Koichi Suzuki


2014-06-30 12:31 GMT+09:00 ZhangJulian <jul...@ou...>:
> Hi All,
>
> We are evaluating PGXC based on a general benchmark in database domain.
>
> First, we tested TPCC by using the tool BenchmarkSQL and HammerDB, but some
> strange bugs block us.
> We plan to use pgbench, do you know if it can be tesed smoothly on PGXC?
> We also plan to use SwingBench, but it seems mostly used on Oracle. Anybody
> has the experience on its testing on PGXC?
>
> And do you know others benchmark which could be tested on PGXC?
>
> Thanks
> Julian
>
> ------------------------------------------------------------------------------
> Open source business process management suite built on Java and Eclipse
> Turn processes into business applications with Bonita BPM Community Edition
> Quickly connect people, data, and systems into organized workflows
> Winner of BOSSIE, CODIE, OW2 and Gartner awards
> http://p.sf.net/sfu/Bonitasoft
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>

Hi All,

We are evaluating PGXC based on a general benchmark in database domain. 

First, we tested TPCC by using the tool BenchmarkSQL and HammerDB, but some strange bugs block us. 
We plan to use pgbench, do you know if it can be tesed smoothly on PGXC?
We also plan to use SwingBench, but it seems mostly used on Oracle. Anybody has the experience on its testing on PGXC? 

And do you know others benchmark which could be tested on PGXC? 

Thanks 
Julian
 		 	   		  

You can run EXECUTE statements and run cursors for fetching data.

Best Regards,
Johan Arve


________________________________________
Från: Thibault Marquand <thi...@ec...>
Skickat: den 20 juni 2014 10:25
Till: Masataka Saito
Kopia: Postgres-XC mailing list
Ämne: Re: [Postgres-xc-general] consistency check on SPI tuple count failed & PL/pgsql

Is there any way to work around with this ?
Like Execute statement or any thing else ?

Le 20/06/2014 03:49, Masataka Saito a écrit :
> Unfortunately it's a restriction.
>
> DML cannot be used in plpgsql functions.
>   * http://postgres-xc.sourceforge.net/docs/1_2/release-xc-1-2.html#AEN126656
>
> Regards.


------------------------------------------------------------------------------
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems
_______________________________________________
Postgres-xc-general mailing list
Pos...@li...
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

Is there any way to work around with this ?
Like Execute statement or any thing else ?

Le 20/06/2014 03:49, Masataka Saito a écrit :
> Unfortunately it's a restriction.
>
> DML cannot be used in plpgsql functions.
>   * http://postgres-xc.sourceforge.net/docs/1_2/release-xc-1-2.html#AEN126656
>
> Regards.

Unfortunately it's a restriction.

DML cannot be used in plpgsql functions.
 * http://postgres-xc.sourceforge.net/docs/1_2/release-xc-1-2.html#AEN126656

Regards.

On 19 June 2014 22:00, Thibault Marquand
<thi...@ec...> wrote:
> Hi,
> I have got some PL/pgsql script which are working on PostgreSQL 9.1.
>
> But on PGXC 1.2.1, I have got the error message : consistency check on
> SPI tuple count failed
> I read the oldest post on mail list about this error but in my case,
> T_TABLE_EXAMPLE is empty so could we talk about inconstencies ?
> I am working on replicated tables.
>
> DO $$
> DECLARE
>      CPA_ID BIGINT;
> [...]
> BEGIN
>
>      SELECT "CPA_ID" INTO CPA_ID FROM "T_TABLE_EXAMPLE" WHERE
> "CPA_FILENAME" = 'example_string' AND "CPA_DELETE" IS NULL;
>
>      IF CPA_ID IS NULL
>      THEN
>        INSERT INTO "T_TABLE" (version, "CPA_FILENAME") VALUES (0,
> 'example_string') RETURNING "CPA_ID" INTO CPA_ID;
>      END IF;
> [...]
> END$$;
>
> The entire error is :
> ERROR:  consistency check on SPI tuple count failed
> CONTEXT:  SQL statement "INSERT INTO "T_TABLE" (version, "CPA_FILENAME")
> VALUES (0, 'example_string') RETURNING "CPA_ID""
>      PL/pgSQL function inline_code_block line 20 at SQL statement
>
> Line 20 is the empty line just under SELECT query.
>
> And when I execute the SQL statement "INSERT INTO "T_TABLE" (version,
> "CPA_FILENAME") VALUES (0, 'example_string') RETURNING "CPA_ID""
> directly, it works. I can read T_TABLE_EXAMPLE and my data are on both
> datanodes.
>
> Thibault
>
> ------------------------------------------------------------------------------
> HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
> Find What Matters Most in Your Big Data with HPCC Systems
> Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
> Leverages Graph Analysis for Fast Processing & Easy Data Exploration
> http://p.sf.net/sfu/hpccsystems
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

Hi,
I have got some PL/pgsql script which are working on PostgreSQL 9.1.

But on PGXC 1.2.1, I have got the error message : consistency check on 
SPI tuple count failed
I read the oldest post on mail list about this error but in my case, 
T_TABLE_EXAMPLE is empty so could we talk about inconstencies ?
I am working on replicated tables.

DO $$
DECLARE
     CPA_ID BIGINT;
[...]
BEGIN

     SELECT "CPA_ID" INTO CPA_ID FROM "T_TABLE_EXAMPLE" WHERE 
"CPA_FILENAME" = 'example_string' AND "CPA_DELETE" IS NULL;

     IF CPA_ID IS NULL
     THEN
       INSERT INTO "T_TABLE" (version, "CPA_FILENAME") VALUES (0, 
'example_string') RETURNING "CPA_ID" INTO CPA_ID;
     END IF;
[...]
END$$;

The entire error is :
ERROR:  consistency check on SPI tuple count failed
CONTEXT:  SQL statement "INSERT INTO "T_TABLE" (version, "CPA_FILENAME") 
VALUES (0, 'example_string') RETURNING "CPA_ID""
     PL/pgSQL function inline_code_block line 20 at SQL statement

Line 20 is the empty line just under SELECT query.

And when I execute the SQL statement "INSERT INTO "T_TABLE" (version, 
"CPA_FILENAME") VALUES (0, 'example_string') RETURNING "CPA_ID"" 
directly, it works. I can read T_TABLE_EXAMPLE and my data are on both 
datanodes.

Thibault

EXECUTE DIRECT ON (datanode1) ‘select count(*) from app.someTable’;

Parens and single quotes.


On Wed, Jun 18, 2014 at 2:31 PM, Aaron Jackson <aja...@re...>
wrote:

>  Sorry if this seems awefully mundane, but I’ve tried this several ways
> and it simply doesn’t seem to work.
>
>
>
> EXECUTE DIRECT ON datanode1 select count(*) from app.someTable;
>
>
>
> ERROR:  syntax error at or near “datanode1” – I’ve tried variations..
>
>
>
> EXECUTE DIRECT ON NODE datanode1 select count(*) from app.someTable;
>
> EXECUTE DIRECT ON NODE datanode1 ‘select count(*) from app.someTable’;
>
> EXECUTE DIRECT ON datanode1 ‘select count(*) from app.someTable’;
>
>
>
> Those should be single quotes if they come across wrong.  I’d venture to
> guess there is a BNF parser somewhere that processed this request and if
> so, I’ll go dig down into that code to see if it’s a bison parsing issue.
> Any suggestions or pointers would be helpful, or maybe I just screwed up
> the syntax.
>
>
>
> Aaron
>
>
> ------------------------------------------------------------------------------
> HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
> Find What Matters Most in Your Big Data with HPCC Systems
> Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
> Leverages Graph Analysis for Fast Processing & Easy Data Exploration
> http://p.sf.net/sfu/hpccsystems
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
>


-- 
Mason Sharp

StormDB - http://www.stormdb.com
The Database Cloud
Postgres-XC Support and Services

Sorry if this seems awefully mundane, but I've tried this several ways and it simply doesn't seem to work.

EXECUTE DIRECT ON datanode1 select count(*) from app.someTable;

ERROR:  syntax error at or near "datanode1" - I've tried variations..

EXECUTE DIRECT ON NODE datanode1 select count(*) from app.someTable;
EXECUTE DIRECT ON NODE datanode1 'select count(*) from app.someTable';
EXECUTE DIRECT ON datanode1 'select count(*) from app.someTable';

Those should be single quotes if they come across wrong.  I'd venture to guess there is a BNF parser somewhere that processed this request and if so, I'll go dig down into that code to see if it's a bison parsing issue.  Any suggestions or pointers would be helpful, or maybe I just screwed up the syntax.

Aaron

Not a problem, that's why we test these things (and take backups).

Aaron
________________________________
From: 鈴木 幸市 [ko...@in...]
Sent: Tuesday, June 17, 2014 7:45 PM
To: Aaron Jackson
Cc: Postgres-XC mailing list
Subject: Re: [Postgres-xc-general] Problem with taking database dump with pgxc-1.2.1

Hmm,  I’m afraid GTM have falled back local XID somehow.   Could you let me test this.   May need a bit.

Thank you.
---
Koichi Suzuki

2014/06/18 3:46、Aaron Jackson <aja...@re...<mailto:aja...@re...>> のメール：

So, it would appear that after applying this patch, that pg_dump works the first time.  However, I prematurely terminated pg_dump (SIGINT on the terminal) and subsequently issued a count on one of my tables.  The query hung and eventually returned with “WARNING: Xid is invalid.” – the call to psql never actually finished.

I ended up restarting the gtm_proxy, data node and coordinator on host 1.  I think at this point, I’ll do a dump and rebuild the cluster since it seems unstable.

Aaron

From: 鈴木 幸市 [mailto:ko...@in...]
Sent: Monday, June 16, 2014 8:05 PM
To: Aaron Jackson
Cc: Postgres-XC mailing list
Subject: Re: [Postgres-xc-general] Problem with taking database dump with pgxc-1.2.1

Year, it’s XC bug.   Because of the sequence handling difference from PG, transaction should not be read-only.   Please try the attached patch.   It will go to 1.2 and master tree if it works.

Hope it helps.

Regards;
---
Koichi Suzuki

2014/06/17 4:25、Aaron Jackson <aja...@re...<mailto:aja...@re...>> のメール：


So, I tried the same thing and applied this patch.  I even verified that it was setting the transaction mode to "serializable, read only, deferrable" - yet the problem persists.  I believe my execution is similar to the OP.

pg_dump -d analysisdb --serializable-deferrable

pg_dump: [archiver (db)] query failed: ERROR:  cannot execute nextval() in a read-only transaction
pg_dump: [archiver (db)] query was: SELECT pg_catalog.nextval('customtype_customtypeid_seq');

Did I miss something else?

Aaron

________________________________
From: Juned Khan [jkh...@gm...<mailto:jkh...@gm...>]
Sent: Friday, April 11, 2014 2:05 AM
To: Pavan Deolasee
Cc: Postgres-XC mailing list
Subject: Re: [Postgres-xc-general] Problem with taking database dump with pgxc-1.2.1
Thanks to Pavan and Koichi for your inputs.
So i think as of now i do not need to worry about
this


On Fri, Apr 11, 2014 at 12:31 PM, Pavan Deolasee <pav...@gm...<mailto:pav...@gm...>> wrote:
On Fri, Apr 11, 2014 at 12:07 PM, Juned Khan <jkh...@gm...<mailto:jkh...@gm...>> wrote:
As of now this fix worked fine for me. is there any side effect of this
patch ?

IMV there won't be any significant side effect of the patch. pg_dump used to run in RW mode till we fixed it in 9.3.  If it has worked for so many years, it may work for some more time too :-) Marking pg_dump READ ONLY was exactly to catch bugs like these. But other than that, I think its OK.

Thanks,
Pavan

--
Pavan Deolasee
http://www.linkedin.com/in/pavandeolasee



--
Thanks,
Juned Khan
iNextrix Technologies Pvt Ltd.
www.inextrix.com<http://www.inextrix.com/>
------------------------------------------------------------------------------
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems_______________________________________________
Postgres-xc-general mailing list
Pos...@li...<mailto:Pos...@li...>
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

Hmm,  I’m afraid GTM have falled back local XID somehow.   Could you let me test this.   May need a bit.

Thank you.
---
Koichi Suzuki

2014/06/18 3:46、Aaron Jackson <aja...@re...<mailto:aja...@re...>> のメール：

So, it would appear that after applying this patch, that pg_dump works the first time.  However, I prematurely terminated pg_dump (SIGINT on the terminal) and subsequently issued a count on one of my tables.  The query hung and eventually returned with “WARNING: Xid is invalid.” – the call to psql never actually finished.

I ended up restarting the gtm_proxy, data node and coordinator on host 1.  I think at this point, I’ll do a dump and rebuild the cluster since it seems unstable.

Aaron

From: 鈴木 幸市 [mailto:ko...@in...]
Sent: Monday, June 16, 2014 8:05 PM
To: Aaron Jackson
Cc: Postgres-XC mailing list
Subject: Re: [Postgres-xc-general] Problem with taking database dump with pgxc-1.2.1

Year, it’s XC bug.   Because of the sequence handling difference from PG, transaction should not be read-only.   Please try the attached patch.   It will go to 1.2 and master tree if it works.

Hope it helps.

Regards;
---
Koichi Suzuki

2014/06/17 4:25、Aaron Jackson <aja...@re...<mailto:aja...@re...>> のメール：


So, I tried the same thing and applied this patch.  I even verified that it was setting the transaction mode to "serializable, read only, deferrable" - yet the problem persists.  I believe my execution is similar to the OP.

pg_dump -d analysisdb --serializable-deferrable

pg_dump: [archiver (db)] query failed: ERROR:  cannot execute nextval() in a read-only transaction
pg_dump: [archiver (db)] query was: SELECT pg_catalog.nextval('customtype_customtypeid_seq');

Did I miss something else?

Aaron

________________________________
From: Juned Khan [jkh...@gm...<mailto:jkh...@gm...>]
Sent: Friday, April 11, 2014 2:05 AM
To: Pavan Deolasee
Cc: Postgres-XC mailing list
Subject: Re: [Postgres-xc-general] Problem with taking database dump with pgxc-1.2.1
Thanks to Pavan and Koichi for your inputs.
So i think as of now i do not need to worry about
this


On Fri, Apr 11, 2014 at 12:31 PM, Pavan Deolasee <pav...@gm...<mailto:pav...@gm...>> wrote:
On Fri, Apr 11, 2014 at 12:07 PM, Juned Khan <jkh...@gm...<mailto:jkh...@gm...>> wrote:
As of now this fix worked fine for me. is there any side effect of this
patch ?

IMV there won't be any significant side effect of the patch. pg_dump used to run in RW mode till we fixed it in 9.3.  If it has worked for so many years, it may work for some more time too :-) Marking pg_dump READ ONLY was exactly to catch bugs like these. But other than that, I think its OK.

Thanks,
Pavan

--
Pavan Deolasee
http://www.linkedin.com/in/pavandeolasee



--
Thanks,
Juned Khan
iNextrix Technologies Pvt Ltd.
www.inextrix.com<http://www.inextrix.com/>
------------------------------------------------------------------------------
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems_______________________________________________
Postgres-xc-general mailing list
Pos...@li...<mailto:Pos...@li...>
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general