postgres-xc-general Mailing List for Postgres-XC (Page 2)

Hello Mr. Bapat,

I modified the output a little, but only the fields selected,
                                                                                                                                                                                                                                                                                                                                                                QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Hash Join  (cost=1.64..5.35 rows=236 width=10726)
   Output: t.id, t.seq, t.text, t.date_updated
   Hash Cond: (t.id = l.id)
   ->  Append  (cost=0.00..0.00 rows=36000 width=5638)
         ->  Data Node Scan on table "_REMOTE_TABLE_QUERY_"  (cost=0.00..0.00 rows=1000 width=5638)
               Output: t.id, t.seq, t.text, t.date_updated
               Node/s: datanode01d, datanode02d, datanode03d, datanode04d, datanode05d, datanode06d, datanode07d, datanode08d, datanode09d, datanode10d, datanode11d, datanode12d, datanode13d, datanode14d, datanode15d, datanode16d, datanode17d, datanode18d, datanode19d, datanode20d
               Remote query: SELECT id, seq, text, date_updated FROM ONLY table t WHERE true

+ 10-20 similar

   ->  Hash  (cost=0.00..0.00 rows=131000 width=5088)
         Output: l.id, l.date_updated, l.seq, l.text
         ->  Append  (cost=0.00..0.00 rows=131000 width=5088)
               ->  Data Node Scan on table_2 "_REMOTE_TABLE_QUERY_"  (cost=0.00..0.00 rows=1000 width=5088)
                     Output: l.id, l.date_updated, l.seq, l.text
                     Node/s: datanode01d, datanode02d, datanode03d, datanode04d, datanode05d, datanode06d, datanode07d, datanode08d, datanode09d, datanode10d, datanode11d, datanode12d, datanode13d, datanode14d, datanode15d, datanode16d, datanode17d, datanode18d, datanode19d, datanode20d
                     Remote query: SELECT id, date_updated, seq, text FROM ONLY table_2 l WHERE ((date_updated >= '2012-12-15 00:00:00-05'::timestamp with time zone) AND (date_updated < '2013-01-01 00:00:00-05'::timestamp with time zone))

                + 30-50 Similar

Thank you for your time,

From: Ashutosh Bapat [mailto:ash...@en...]
Sent: Friday, March 22, 2013 12:28 AM
To: Arni Sumarlidason
Cc: Koichi Suzuki; pos...@li...
Subject: Re: [Postgres-xc-general] Planner join logic

Hi Arni,
Can you please send an explain verbose output of this query?
On Thu, Mar 21, 2013 at 5:57 PM, Arni Sumarlidason <Arn...@md...<mailto:Arn...@md...>> wrote:
As always, thank you for your responses, appreciate it!

We are using trunk from around Feb. 18th.

Here are some tables that replicate the issue,
create table table1(
  id bigint,
  seq integer,
  text data(256),
  created_at timestamp with time zone,
  PRIMARY KEY(id, seq)) DISTRIBUTE BY HASH(id);

create table table2(
  id bigint,
  seq integer,
  text data_complex(256),
  update_time timestamp with time zone,
  PRIMARY KEY(id, seq)) DISTRIBUTE BY HASH(id);

Select * from table1 t1, table2 t2 where t2.update_time >= '2012-12-15' and t2.update_time < '2013-01-01' and t2.id<https://console.mxlogic.com/redir/?2eosod7bNEVd79JYsOMOyqejhOr01PjWi1K_cKnLuZXTKrKrSxdKtq4-8qJIMlY9_5qcGuwgVsSYYyCrhpspd7arz3XxJxwS2_id41Fr2Rqj8N-xsfIn8_3UWuqnjh05zZx3UCy0hHa1EwfGhBoTfM-u0USyrjdIIczxNEVvsdVuuN3I-tei91>=t1.id<https://console.mxlogic.com/redir/?zC763hOYqejhOrv7cIcECzAQsCM0sQuAwrLPbBXTLuZXCXCZEjrDmxfy6Hrc5v2vNmzaDE4endLf8FCQmn6jhOCUM-UroodwLQzh0qmMJmAOcvEn3X5OfM-eDCBQQg1o_og-9Ew4qOwq83WApmdPYfDwedFCQPrb38UsqenT3qiXMGWXj>

Thank you,
Arni



--
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

Hi Arni,
Can you please send an explain verbose output of this query?

On Thu, Mar 21, 2013 at 5:57 PM, Arni Sumarlidason <
Arn...@md...> wrote:

>  As always, thank you for your responses, appreciate it! ****
>
> ** **
>
> We are using trunk from around Feb. 18th.****
>
> ** **
>
> Here are some tables that replicate the issue,****
>
> create table table1(****
>
>   id bigint,****
>
>   seq integer,****
>
>   text data(256),****
>
>   created_at timestamp with time zone,****
>
>   PRIMARY KEY(id, seq)) DISTRIBUTE BY HASH(id);****
>
> ** **
>
> create table table2(****
>
>   id bigint,****
>
>   seq integer,****
>
>   text data_complex(256),****
>
>   update_time timestamp with time zone,****
>
>   PRIMARY KEY(id, seq)) DISTRIBUTE BY HASH(id);****
>
> ** **
>
> Select * from table1 t1, table2 t2 where t2.update_time >= ‘2012-12-15’
> and t2.update_time < ‘2013-01-01’ and t2.id=t1.id****
>
> ** **
>
> Thank you,****
>
> Arni****
>



-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

You can try pgxc_pool_reload() and pgxc_pool_check() as found at
http://postgres-xc.sourceforge.net/docs/1_0_2/functions-admin.html

Another means is CLEAN CONNECTION statement,
http://postgres-xc.sourceforge.net/docs/1_0_2/sql-cleanconnection.html

Regards;
----------
Koichi Suzuki


2013/3/21 Chris Brown <cb...@in...>:
> Hi all,
>
> I have previously run successful tests with 2 datanodes and I'm trying to
> run the same test with 3 datanodes.  I am able to create/drop tables and
> insert/select rows from all 3 coordinators.  The test runs if I point it
> at a single coordinator (doesn't matter which one) and set the number of
> clients to 32.  Each datanode box is running
> datanode+coordinator(master)+gtm_proxy and gtm is running on a 4th box.
>
> I have upped max_pool_size on the coordinator and max_connections on the
> datanode to 400.  I haven't upped max_prepared_transactions however I
> don't think any of these single-row-and-table inserts will required a
> prepared transaction unless _all_ transactions use them (do they? And if
> they do, why does my 2-datanode test work?).
>
>
> Problem 1: If I use a single client, inserts seem to get stuck after about
> 12 (running a batch file with 6014 individual insert statements).  The
> process doing inserts (psql) just stops doing inserts and I don't see any
> output anywhere.
>
> Problem 2: If I use 64 clients connecting to all 3 coordinators (round
> robin so approx. 21 clients each) I get 'Failed to get pooled connection'
> errors.
>
> Problem 3: If I get 'Failed to get pooled connection' errors, the cluster
> is in a broken state from then on.  The only solution I've found is to
> dropdb then createdb which usually can't be accomplished without stopping
> the entire cluster and restarting it.
>
> What am I doing wrong?
>
> Chris...
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_mar
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

Hi Arni,
There are various scenarios under which we push join, it's difficult to
elaborate all of them. Between 1.0 and current head there have been a whole
bunch of changes in join push-down. You will find that in current head,
even the non-shippable joins are very fast, per my measurement and
Suzuki-san's measurement 100 times), owing to merge/hash join instead of
nested loop join. Which version are you trying out? Can you please try the
current head (for testing only, not production).

It will help you can specify your table structures, and queries.
On Thu, Mar 21, 2013 at 8:39 AM, Koichi Suzuki <koi...@gm...>wrote:

> Hi,
>
> XC pushes down join when join column of both table are distribution
> column if both tables are distributed by hash or modulo.   If one of
> the tables is replicated table or both tables are replicated table,
> join will be pushed down too.   We have much more variety of the
> statement context.   If you share your table structure (create table
> etc.), statement and its plan, it will be helpful if there's something
> wrong or if there's something helping.
>
> I hope Ashutosh will answer the question better.
>
> Regards;
> ----------
> Koichi Suzuki
>
>
> 2013/3/21 Arni Sumarlidason <Arn...@md...>:
> > What kind of logic does pgxc use to determine if a join can be can at
> the data node level vs the coordinator node level?
> >
> > My joins are occurring at the coordinator and the data is overwhelming
> the coordinator. I asked a similar question before, I thought this time I'd
> be more generic and focus on the core problem.
> >
> > Arni
> > Sent from my iPhone
> >
> ------------------------------------------------------------------------------
> > Everyone hates slow websites. So do we.
> > Make your web apps faster with AppDynamics
> > Download AppDynamics Lite for free today:
> > http://p.sf.net/sfu/appdyn_d2d_mar
> > _______________________________________________
> > Postgres-xc-general mailing list
> > Pos...@li...
> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_mar
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>



-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

Hi,

XC pushes down join when join column of both table are distribution
column if both tables are distributed by hash or modulo.   If one of
the tables is replicated table or both tables are replicated table,
join will be pushed down too.   We have much more variety of the
statement context.   If you share your table structure (create table
etc.), statement and its plan, it will be helpful if there's something
wrong or if there's something helping.

I hope Ashutosh will answer the question better.

Regards;
----------
Koichi Suzuki


2013/3/21 Arni Sumarlidason <Arn...@md...>:
> What kind of logic does pgxc use to determine if a join can be can at the data node level vs the coordinator node level?
>
> My joins are occurring at the coordinator and the data is overwhelming the coordinator. I asked a similar question before, I thought this time I'd be more generic and focus on the core problem.
>
> Arni
> Sent from my iPhone
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_mar
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

What kind of logic does pgxc use to determine if a join can be can at the data node level vs the coordinator node level?

My joins are occurring at the coordinator and the data is overwhelming the coordinator. I asked a similar question before, I thought this time I'd be more generic and focus on the core problem.

Arni
Sent from my iPhone

Hi all,

I have previously run successful tests with 2 datanodes and I'm trying to
run the same test with 3 datanodes.  I am able to create/drop tables and
insert/select rows from all 3 coordinators.  The test runs if I point it
at a single coordinator (doesn't matter which one) and set the number of
clients to 32.  Each datanode box is running
datanode+coordinator(master)+gtm_proxy and gtm is running on a 4th box.

I have upped max_pool_size on the coordinator and max_connections on the
datanode to 400.  I haven't upped max_prepared_transactions however I
don't think any of these single-row-and-table inserts will required a
prepared transaction unless _all_ transactions use them (do they? And if
they do, why does my 2-datanode test work?).


Problem 1: If I use a single client, inserts seem to get stuck after about
12 (running a batch file with 6014 individual insert statements).  The
process doing inserts (psql) just stops doing inserts and I don't see any
output anywhere.

Problem 2: If I use 64 clients connecting to all 3 coordinators (round
robin so approx. 21 clients each) I get 'Failed to get pooled connection'
errors.

Problem 3: If I get 'Failed to get pooled connection' errors, the cluster
is in a broken state from then on.  The only solution I've found is to
dropdb then createdb which usually can't be accomplished without stopping
the entire cluster and restarting it.

What am I doing wrong?

Chris...

My guess, looking at the graphs, is that you're checkpointing in the
middle of the test. If you set your checkpoint_timeout and
checkout_segments high enough and do a manual checkpoint before the
test, you shouldn't checkpoint during the test.

checkpoint_segments=256
checkpoint_timeout=30min

You'll need to restart for those settings to take effect. The from psql, run:
postgres=# checkpoint;


On Tue, Mar 19, 2013 at 6:41 PM, Chris Brown <cb...@in...> wrote:
> Ahhh, you were right, the pgxc (64 clients without gtm_proxy) has almost exactly the same usage characteristics as pg (32 clients non-xc).
>
> I find this puzzling.  If each of the two boxes is doing slightly more than half the work, why does it appear that they're both still nearly bottlenecked on disk?  I can understand the CPU if the coordinator/datanode design isn't optimized yet.
>
> Chris...
> ________________________________________
> From: Chris Brown
> Sent: March 19, 2013 3:01 PM
> To: Jim Mlodgenski
> Cc: pos...@li...
> Subject: RE: [Postgres-xc-general] PG-XC Performance
>
> I managed to get some more instrumentation results...
>
> PG waffles between disk-bound and CPU bound.  CPU wavers around 90% across all CPUs and the disk spikes up and down from 25% to 95%.
>
> PG-XC (using GTM-proxy) seems to be CPU bound with around 95% CPU throughout the test.  One CPU core is consistently nearly 100% throughout the test on both datanodes.  Disk is high but not as high or as spikey as PG, ranging from 20% (most of the time) to 80%.
>
> I haven't been able to figure out how much memory the raid card has.  It is an HPSA 410i which comes in at least 3 variants.
>
> I will run the fsync-off test asap but that probably won't be 'til tomorrow (deadlines).  Honestly, though, I don't care that NDB is faster - the difference between PG and NDB is meaningless since I know that's apples-to-oranges (I need it as a known baseline for a presentation so I included it here).  I'm also going to run this test using 3 datanode servers to see how the bottlenecks change.
>
> Chris...
>
> ________________________________________
> From: Jim Mlodgenski [ji...@gm...]
> Sent: March 19, 2013 10:23 AM
> To: Chris Brown
> Cc: pos...@li...
> Subject: Re: [Postgres-xc-general] PG-XC Performance
>
> How big is the RAID controller's cache? You're probably pushing
> several gig of data through it on you're XC and PostgreSQL tests so
> you're likely flooding it to the point where it needs to wait to flush
> to disk even when you're adding the second XC datanode.
>
> If you wanted to have a more apples-to-apples comparison to NDB, you
> can shut off some of the durability requirements that PostgreSQL has
> on by default.
> http://www.postgresql.org/docs/9.1/static/non-durability.html
>
> fsync=off
> full_page_writes=off
> synchronous_commit=off
> checkpoint_segments=64
>
> You wouldn't run with these setting in production unless you are OK
> with potentially corrupted data, but it does get you close to what NDB
> does under the covers.
>
> On Tue, Mar 19, 2013 at 1:07 PM, Chris Brown <cb...@in...> wrote:
>> Yes, by round robin so it's not perfect, but it's probably good enough.
>>
>> On 13-03-19 10:06 AM, "Jim Mlodgenski" <ji...@gm...> wrote:
>>
>>>For your XC test, are you splitting your connections across the
>>>coordinators with half of your psql sessions pointing to q1 and half
>>>to q2?
>>>
>>>On Tue, Mar 19, 2013 at 1:01 PM, Chris Brown <cb...@in...> wrote:
>>>> For PG (no-XC) it looks like I'm pegged on disk (still getting better
>>>> instrumentation).  I would expect that my disk usage would be nearly
>>>> halved when striping data - still gathering info though.
>>>>
>>>> On 13-03-19 8:40 AM, "Jim Mlodgenski" <ji...@gm...> wrote:
>>>>
>>>>>It may be easier to just concentrate on the PostgreSQL test to find
>>>>>the bottleneck. The fact that the XC test (without GTM Proxy) and the
>>>>>PostgreSQL test is so similar, is leading me to believe you're hitting
>>>>>the same bottleneck.
>>>>>
>>>>>On Tue, Mar 19, 2013 at 11:35 AM, Chris Brown <cb...@in...>
>>>>>wrote:
>>>>>> I assume I'm I/O bound.  The CPU on the datanode/coordinator/gtm_proxy
>>>>>>boxes gets to around 85%.  I'll need some more instrumentation to be
>>>>>>sure - I'll get back to you with details.
>>>>>>
>>>>>> Chris...
>>>>>>
>>>>>> ________________________________________
>>>>>> From: Jim Mlodgenski [ji...@gm...]
>>>>>> Sent: March 19, 2013 8:32 AM
>>>>>> To: Chris Brown
>>>>>> Cc: pos...@li...
>>>>>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>>>>>
>>>>>> When running the test against straight PostgreSQL, where is the
>>>>>> bottleneck? Are you CPU bound or IO bound?
>>>>>>
>>>>>> On Tue, Mar 19, 2013 at 10:59 AM, Chris Brown <cb...@in...>
>>>>>>wrote:
>>>>>>> I also just discovered that my assumption about postgres (non xc)
>>>>>>>being 9.2 was incorrect:
>>>>>>>   postgresql-server-9.1.7-1.fc16.x86_64
>>>>>>> Oops, sorry about the misinformation.
>>>>>>>
>>>>>>> Chris...
>>>>>>> ________________________________________
>>>>>>> From: Chris Brown [cb...@in...]
>>>>>>> Sent: March 19, 2013 7:45 AM
>>>>>>> To: Jim Mlodgenski
>>>>>>> Cc: pos...@li...
>>>>>>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>>>>>>
>>>>>>> Hi Jim,
>>>>>>>
>>>>>>> I believe I'm doing an fsync (fdatasync) for each commit.  I'm doing
>>>>>>>the inserts as actual inserts performed by psql.  They are not batched
>>>>>>>- there is one insert per line and each file contains about 10,000
>>>>>>>inserts.  This was a conscious choice since I was testing expected
>>>>>>>transaction rate as opposed to import rate.
>>>>>>>
>>>>>>> I haven't done much tuning to postgres.  Datanodes received this:
>>>>>>>   checkpoint_segements=32
>>>>>>>   effective_cache_size=4GB
>>>>>>>   shared_buffers=4GB
>>>>>>>   random_page_cost=3.0
>>>>>>> For the coordinators, no tuning was done.
>>>>>>> For xc, I also added:
>>>>>>>   max_prepared_transactions = 10
>>>>>>>   gtm_host = 'localhost'  # or the remote host, as appropriate on
>>>>>>>coordinators
>>>>>>>   gtm_port = 6666
>>>>>>> Otherwise the postgresql.conf file is the default one provided by
>>>>>>>initdb.  If I understand this, then in linux (3.1.9-1.fc16.x86_64) I
>>>>>>>am
>>>>>>>using the defaults of wal_level(minimum), fsync(on),
>>>>>>>wal_sync_method(fdatasync).
>>>>>>>
>>>>>>> I reran the 60 client test with explicit DISTRIBUTE BY HASH and
>>>>>>>MODULO, confirming the obj_MODULO and obj_HASH in the tables:
>>>>>>>     "table_11_obj_MODULO_idx" btree ("obj_MODULO")
>>>>>>> It didn't make an appreciable difference (~10 seconds on a 5m50s
>>>>>>>test)
>>>>>>>which leads me to believe that the 5m50s number is slightly in error.
>>>>>>>
>>>>>>> Chris...
>>>>>>>
>>>>>>> ________________________________________
>>>>>>> From: Jim Mlodgenski [ji...@gm...]
>>>>>>> Sent: March 19, 2013 7:26 AM
>>>>>>> To: Chris Brown
>>>>>>> Cc: Koichi Suzuki; pos...@li...
>>>>>>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>>>>>>
>>>>>>> Can you send along some more details of the test? I find it odd that
>>>>>>> in every scenario, the load times are consistent despite the change
>>>>>>>in
>>>>>>> the number of threads. Are you committing your transaction after each
>>>>>>> insert? If so, your limiting factor is probably the fsync to disk
>>>>>>> which would account for why PostgreSQL 9.2 is about the same speed as
>>>>>>> XC 1.0.
>>>>>>>
>>>>>>> On Tue, Mar 19, 2013 at 10:06 AM, Chris Brown <cb...@in...>
>>>>>>>wrote:
>>>>>>>> Here's a sample table:
>>>>>>>>
>>>>>>>> CREATE TABLE table_11 (
>>>>>>>>   "part" int not null,
>>>>>>>>   "key1" bigint not null,
>>>>>>>>   "key2" bigint not null,
>>>>>>>>   "col_1" varchar(23),
>>>>>>>>   "col_2" varchar(5),
>>>>>>>>   "col_3" varchar(32),
>>>>>>>>   "col_4" varchar(5),
>>>>>>>>   PRIMARY KEY(part,key1,key2)
>>>>>>>> );
>>>>>>>>
>>>>>>>> I guess I'm not specifying a distribution key.  I assumed, perhaps
>>>>>>>>incorrectly, that hash on the first element of the primary key would
>>>>>>>>be the default.  I will try a run with an explicit DISTRIBUTE BY
>>>>>>>>HASH(part).
>>>>>>>>
>>>>>>>> Chris...
>>>>>>>> ________________________________________
>>>>>>>> From: Koichi Suzuki [koi...@gm...]
>>>>>>>> Sent: March 18, 2013 10:47 PM
>>>>>>>> To: Chris Brown
>>>>>>>> Cc: pos...@li...
>>>>>>>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>>>>>>>
>>>>>>>> What kind of distribution did you use in your case?   If all the
>>>>>>>> tables are replicated, all the inserts are associated with implicit
>>>>>>>> 2PCs.   Frequently-updated tables should be distributed (hash will
>>>>>>>>be
>>>>>>>> better if the table has a suitable column).
>>>>>>>>
>>>>>>>> Regards;
>>>>>>>> ----------
>>>>>>>> Koichi Suzuki
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/3/19 Chris Brown <cb...@in...>:
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> I've been doing some performance testing of PostgreSQL-XC.  My
>>>>>>>>>workload is
>>>>>>>>> all insert in this case.  I was expecting PG-XC to perform worse
>>>>>>>>>than the
>>>>>>>>> alternatives but I was shocked by how much worse it was than
>>>>>>>>>PostgreSQL.
>>>>>>>>> I'm well aware that NDB has durability and features issues so I
>>>>>>>>>expected
>>>>>>>>> it to perform better - I don't consider the 0.8 to be particularly
>>>>>>>>> interesting.  I'd like anyone's thoughts on this.
>>>>>>>>>
>>>>>>>>> Chris...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Results:
>>>>>>>>>
>>>>>>>>> All results are very approximate!
>>>>>>>>> CPU is an estimate on the number of cores used by server processes
>>>>>>>>> (ndbmtd, ndb_mgmd, postgresql, gtm, gtm_proxy).
>>>>>>>>> TC  is a comparison of (Time*CPU) using PostgreSQL as the baseline
>>>>>>>>>and is
>>>>>>>>> intended to represent a comparative CPU cost for a single insert
>>>>>>>>> transaction.
>>>>>>>>>
>>>>>>>>> Client   PostgreSQL               PostgreSQL-XC             MySQL
>>>>>>>>>Cluster
>>>>>>>>>           PostgreSQL-XC-2
>>>>>>>>> Threads  Time   CPU        TC     Time   CPU         TC     Time
>>>>>>>>>CPU
>>>>>>>>>    TC     Time   CPU         TC
>>>>>>>>> 16       5m08s  4.3 cores  1.0
>>>>>>>>> 24       4m56s  4.6 cores  1.0    6m10s  10.7 cores  2.9
>>>>>>>>> 32       4m48s  4.7 cores  1.0    5m26s  11.9 cores  2.9    2m27s
>>>>>>>>>7.8
>>>>>>>>> cores  0.8
>>>>>>>>> 48       4m49s  4.7 cores  1.0    5m01s  12.6 cores  2.8    2m16s
>>>>>>>>>8.0
>>>>>>>>> cores  0.8    6m06s  10.2 cores  2.7
>>>>>>>>> 64                                5m01s  13.1 cores  2.9    2m11s
>>>>>>>>>8.3
>>>>>>>>> cores  0.8    5m50s  10.5 cores  2.7
>>>>>>>>> 96                                                          2m13s
>>>>>>>>>8.4
>>>>>>>>> cores  0.8
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Configurations:
>>>>>>>>>
>>>>>>>>> PostgreSQL:
>>>>>>>>> Datanodes        q1
>>>>>>>>> Version: 9.2.x (redhat)
>>>>>>>>>
>>>>>>>>> PostgreSQL-XC:
>>>>>>>>> Datanodes        q1-q2 (striped)
>>>>>>>>> Coordinators     q1-q2
>>>>>>>>> GTM      q3
>>>>>>>>> Version 9.1.7 (tar)
>>>>>>>>>
>>>>>>>>> PostgreSQL-XC-2:
>>>>>>>>> Datanodes        q1-q2 (striped)
>>>>>>>>> Coordinators     q1-q2
>>>>>>>>> GTM Proxy        q1-q2
>>>>>>>>> GTM      q3
>>>>>>>>> Version 9.1.7 (tar)
>>>>>>>>>
>>>>>>>>> MySQL Cluster:
>>>>>>>>> ndbmtd   q1-q2 (striped)
>>>>>>>>> mysqld   q1-q2
>>>>>>>>> ndb_mgmd         q1
>>>>>>>>> Version 7.1.17 (oracle rhel 5)
>>>>>>>>>
>>>>>>>>> q1, q2, q3 are identical boxes each with:
>>>>>>>>>   24g ram, 1 cpu (6 cores), middle-tier raid with battery backed
>>>>>>>>>ram
>>>>>>>>>and 3
>>>>>>>>> platters in raid 5.
>>>>>>>>> They are lan local with a vlan-aware switch (I don't know which one
>>>>>>>>>and I
>>>>>>>>> didn't check to see if it was saturated).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Tables & Data:
>>>>>>>>>
>>>>>>>>> 161 tables with primary key ("part" int not null, "key1" bigint not
>>>>>>>>>null,
>>>>>>>>> "key2" bigint not null).  Each table has a selection of varchar
>>>>>>>>>columns
>>>>>>>>> typically less than varchar(80).  All non-primary-key columns have
>>>>>>>>> non-unique indexes.  5.8 million rows were imported with a
>>>>>>>>>different
>>>>>>>>> number of rows to each table.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>--------------------------------------------------------------------
>>>>>>>>>--
>>>>>>>>>--------
>>>>>>>>> Everyone hates slow websites. So do we.
>>>>>>>>> Make your web apps faster with AppDynamics
>>>>>>>>> Download AppDynamics Lite for free today:
>>>>>>>>> http://p.sf.net/sfu/appdyn_d2d_mar
>>>>>>>>> _______________________________________________
>>>>>>>>> Postgres-xc-general mailing list
>>>>>>>>> Pos...@li...
>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>>>>>>
>>>>>>>>
>>>>>>>>---------------------------------------------------------------------
>>>>>>>>--
>>>>>>>>-------
>>>>>>>> Everyone hates slow websites. So do we.
>>>>>>>> Make your web apps faster with AppDynamics
>>>>>>>> Download AppDynamics Lite for free today:
>>>>>>>> http://p.sf.net/sfu/appdyn_d2d_mar
>>>>>>>> _______________________________________________
>>>>>>>> Postgres-xc-general mailing list
>>>>>>>> Pos...@li...
>>>>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>>>>>
>>>>>>>
>>>>>>>----------------------------------------------------------------------
>>>>>>>--
>>>>>>>------
>>>>>>> Everyone hates slow websites. So do we.
>>>>>>> Make your web apps faster with AppDynamics
>>>>>>> Download AppDynamics Lite for free today:
>>>>>>> http://p.sf.net/sfu/appdyn_d2d_mar
>>>>>>> _______________________________________________
>>>>>>> Postgres-xc-general mailing list
>>>>>>> Pos...@li...
>>>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>>
>>

Ahhh, you were right, the pgxc (64 clients without gtm_proxy) has almost exactly the same usage characteristics as pg (32 clients non-xc).

I find this puzzling.  If each of the two boxes is doing slightly more than half the work, why does it appear that they're both still nearly bottlenecked on disk?  I can understand the CPU if the coordinator/datanode design isn't optimized yet.

Chris...
________________________________________
From: Chris Brown
Sent: March 19, 2013 3:01 PM
To: Jim Mlodgenski
Cc: pos...@li...
Subject: RE: [Postgres-xc-general] PG-XC Performance

I managed to get some more instrumentation results...

PG waffles between disk-bound and CPU bound.  CPU wavers around 90% across all CPUs and the disk spikes up and down from 25% to 95%.

PG-XC (using GTM-proxy) seems to be CPU bound with around 95% CPU throughout the test.  One CPU core is consistently nearly 100% throughout the test on both datanodes.  Disk is high but not as high or as spikey as PG, ranging from 20% (most of the time) to 80%.

I haven't been able to figure out how much memory the raid card has.  It is an HPSA 410i which comes in at least 3 variants.

I will run the fsync-off test asap but that probably won't be 'til tomorrow (deadlines).  Honestly, though, I don't care that NDB is faster - the difference between PG and NDB is meaningless since I know that's apples-to-oranges (I need it as a known baseline for a presentation so I included it here).  I'm also going to run this test using 3 datanode servers to see how the bottlenecks change.

Chris...

________________________________________
From: Jim Mlodgenski [ji...@gm...]
Sent: March 19, 2013 10:23 AM
To: Chris Brown
Cc: pos...@li...
Subject: Re: [Postgres-xc-general] PG-XC Performance

How big is the RAID controller's cache? You're probably pushing
several gig of data through it on you're XC and PostgreSQL tests so
you're likely flooding it to the point where it needs to wait to flush
to disk even when you're adding the second XC datanode.

If you wanted to have a more apples-to-apples comparison to NDB, you
can shut off some of the durability requirements that PostgreSQL has
on by default.
http://www.postgresql.org/docs/9.1/static/non-durability.html

fsync=off
full_page_writes=off
synchronous_commit=off
checkpoint_segments=64

You wouldn't run with these setting in production unless you are OK
with potentially corrupted data, but it does get you close to what NDB
does under the covers.

On Tue, Mar 19, 2013 at 1:07 PM, Chris Brown <cb...@in...> wrote:
> Yes, by round robin so it's not perfect, but it's probably good enough.
>
> On 13-03-19 10:06 AM, "Jim Mlodgenski" <ji...@gm...> wrote:
>
>>For your XC test, are you splitting your connections across the
>>coordinators with half of your psql sessions pointing to q1 and half
>>to q2?
>>
>>On Tue, Mar 19, 2013 at 1:01 PM, Chris Brown <cb...@in...> wrote:
>>> For PG (no-XC) it looks like I'm pegged on disk (still getting better
>>> instrumentation).  I would expect that my disk usage would be nearly
>>> halved when striping data - still gathering info though.
>>>
>>> On 13-03-19 8:40 AM, "Jim Mlodgenski" <ji...@gm...> wrote:
>>>
>>>>It may be easier to just concentrate on the PostgreSQL test to find
>>>>the bottleneck. The fact that the XC test (without GTM Proxy) and the
>>>>PostgreSQL test is so similar, is leading me to believe you're hitting
>>>>the same bottleneck.
>>>>
>>>>On Tue, Mar 19, 2013 at 11:35 AM, Chris Brown <cb...@in...>
>>>>wrote:
>>>>> I assume I'm I/O bound.  The CPU on the datanode/coordinator/gtm_proxy
>>>>>boxes gets to around 85%.  I'll need some more instrumentation to be
>>>>>sure - I'll get back to you with details.
>>>>>
>>>>> Chris...
>>>>>
>>>>> ________________________________________
>>>>> From: Jim Mlodgenski [ji...@gm...]
>>>>> Sent: March 19, 2013 8:32 AM
>>>>> To: Chris Brown
>>>>> Cc: pos...@li...
>>>>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>>>>
>>>>> When running the test against straight PostgreSQL, where is the
>>>>> bottleneck? Are you CPU bound or IO bound?
>>>>>
>>>>> On Tue, Mar 19, 2013 at 10:59 AM, Chris Brown <cb...@in...>
>>>>>wrote:
>>>>>> I also just discovered that my assumption about postgres (non xc)
>>>>>>being 9.2 was incorrect:
>>>>>>   postgresql-server-9.1.7-1.fc16.x86_64
>>>>>> Oops, sorry about the misinformation.
>>>>>>
>>>>>> Chris...
>>>>>> ________________________________________
>>>>>> From: Chris Brown [cb...@in...]
>>>>>> Sent: March 19, 2013 7:45 AM
>>>>>> To: Jim Mlodgenski
>>>>>> Cc: pos...@li...
>>>>>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>>>>>
>>>>>> Hi Jim,
>>>>>>
>>>>>> I believe I'm doing an fsync (fdatasync) for each commit.  I'm doing
>>>>>>the inserts as actual inserts performed by psql.  They are not batched
>>>>>>- there is one insert per line and each file contains about 10,000
>>>>>>inserts.  This was a conscious choice since I was testing expected
>>>>>>transaction rate as opposed to import rate.
>>>>>>
>>>>>> I haven't done much tuning to postgres.  Datanodes received this:
>>>>>>   checkpoint_segements=32
>>>>>>   effective_cache_size=4GB
>>>>>>   shared_buffers=4GB
>>>>>>   random_page_cost=3.0
>>>>>> For the coordinators, no tuning was done.
>>>>>> For xc, I also added:
>>>>>>   max_prepared_transactions = 10
>>>>>>   gtm_host = 'localhost'  # or the remote host, as appropriate on
>>>>>>coordinators
>>>>>>   gtm_port = 6666
>>>>>> Otherwise the postgresql.conf file is the default one provided by
>>>>>>initdb.  If I understand this, then in linux (3.1.9-1.fc16.x86_64) I
>>>>>>am
>>>>>>using the defaults of wal_level(minimum), fsync(on),
>>>>>>wal_sync_method(fdatasync).
>>>>>>
>>>>>> I reran the 60 client test with explicit DISTRIBUTE BY HASH and
>>>>>>MODULO, confirming the obj_MODULO and obj_HASH in the tables:
>>>>>>     "table_11_obj_MODULO_idx" btree ("obj_MODULO")
>>>>>> It didn't make an appreciable difference (~10 seconds on a 5m50s
>>>>>>test)
>>>>>>which leads me to believe that the 5m50s number is slightly in error.
>>>>>>
>>>>>> Chris...
>>>>>>
>>>>>> ________________________________________
>>>>>> From: Jim Mlodgenski [ji...@gm...]
>>>>>> Sent: March 19, 2013 7:26 AM
>>>>>> To: Chris Brown
>>>>>> Cc: Koichi Suzuki; pos...@li...
>>>>>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>>>>>
>>>>>> Can you send along some more details of the test? I find it odd that
>>>>>> in every scenario, the load times are consistent despite the change
>>>>>>in
>>>>>> the number of threads. Are you committing your transaction after each
>>>>>> insert? If so, your limiting factor is probably the fsync to disk
>>>>>> which would account for why PostgreSQL 9.2 is about the same speed as
>>>>>> XC 1.0.
>>>>>>
>>>>>> On Tue, Mar 19, 2013 at 10:06 AM, Chris Brown <cb...@in...>
>>>>>>wrote:
>>>>>>> Here's a sample table:
>>>>>>>
>>>>>>> CREATE TABLE table_11 (
>>>>>>>   "part" int not null,
>>>>>>>   "key1" bigint not null,
>>>>>>>   "key2" bigint not null,
>>>>>>>   "col_1" varchar(23),
>>>>>>>   "col_2" varchar(5),
>>>>>>>   "col_3" varchar(32),
>>>>>>>   "col_4" varchar(5),
>>>>>>>   PRIMARY KEY(part,key1,key2)
>>>>>>> );
>>>>>>>
>>>>>>> I guess I'm not specifying a distribution key.  I assumed, perhaps
>>>>>>>incorrectly, that hash on the first element of the primary key would
>>>>>>>be the default.  I will try a run with an explicit DISTRIBUTE BY
>>>>>>>HASH(part).
>>>>>>>
>>>>>>> Chris...
>>>>>>> ________________________________________
>>>>>>> From: Koichi Suzuki [koi...@gm...]
>>>>>>> Sent: March 18, 2013 10:47 PM
>>>>>>> To: Chris Brown
>>>>>>> Cc: pos...@li...
>>>>>>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>>>>>>
>>>>>>> What kind of distribution did you use in your case?   If all the
>>>>>>> tables are replicated, all the inserts are associated with implicit
>>>>>>> 2PCs.   Frequently-updated tables should be distributed (hash will
>>>>>>>be
>>>>>>> better if the table has a suitable column).
>>>>>>>
>>>>>>> Regards;
>>>>>>> ----------
>>>>>>> Koichi Suzuki
>>>>>>>
>>>>>>>
>>>>>>> 2013/3/19 Chris Brown <cb...@in...>:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> I've been doing some performance testing of PostgreSQL-XC.  My
>>>>>>>>workload is
>>>>>>>> all insert in this case.  I was expecting PG-XC to perform worse
>>>>>>>>than the
>>>>>>>> alternatives but I was shocked by how much worse it was than
>>>>>>>>PostgreSQL.
>>>>>>>> I'm well aware that NDB has durability and features issues so I
>>>>>>>>expected
>>>>>>>> it to perform better - I don't consider the 0.8 to be particularly
>>>>>>>> interesting.  I'd like anyone's thoughts on this.
>>>>>>>>
>>>>>>>> Chris...
>>>>>>>>
>>>>>>>>
>>>>>>>> Results:
>>>>>>>>
>>>>>>>> All results are very approximate!
>>>>>>>> CPU is an estimate on the number of cores used by server processes
>>>>>>>> (ndbmtd, ndb_mgmd, postgresql, gtm, gtm_proxy).
>>>>>>>> TC  is a comparison of (Time*CPU) using PostgreSQL as the baseline
>>>>>>>>and is
>>>>>>>> intended to represent a comparative CPU cost for a single insert
>>>>>>>> transaction.
>>>>>>>>
>>>>>>>> Client   PostgreSQL               PostgreSQL-XC             MySQL
>>>>>>>>Cluster
>>>>>>>>           PostgreSQL-XC-2
>>>>>>>> Threads  Time   CPU        TC     Time   CPU         TC     Time
>>>>>>>>CPU
>>>>>>>>    TC     Time   CPU         TC
>>>>>>>> 16       5m08s  4.3 cores  1.0
>>>>>>>> 24       4m56s  4.6 cores  1.0    6m10s  10.7 cores  2.9
>>>>>>>> 32       4m48s  4.7 cores  1.0    5m26s  11.9 cores  2.9    2m27s
>>>>>>>>7.8
>>>>>>>> cores  0.8
>>>>>>>> 48       4m49s  4.7 cores  1.0    5m01s  12.6 cores  2.8    2m16s
>>>>>>>>8.0
>>>>>>>> cores  0.8    6m06s  10.2 cores  2.7
>>>>>>>> 64                                5m01s  13.1 cores  2.9    2m11s
>>>>>>>>8.3
>>>>>>>> cores  0.8    5m50s  10.5 cores  2.7
>>>>>>>> 96                                                          2m13s
>>>>>>>>8.4
>>>>>>>> cores  0.8
>>>>>>>>
>>>>>>>>
>>>>>>>> Configurations:
>>>>>>>>
>>>>>>>> PostgreSQL:
>>>>>>>> Datanodes        q1
>>>>>>>> Version: 9.2.x (redhat)
>>>>>>>>
>>>>>>>> PostgreSQL-XC:
>>>>>>>> Datanodes        q1-q2 (striped)
>>>>>>>> Coordinators     q1-q2
>>>>>>>> GTM      q3
>>>>>>>> Version 9.1.7 (tar)
>>>>>>>>
>>>>>>>> PostgreSQL-XC-2:
>>>>>>>> Datanodes        q1-q2 (striped)
>>>>>>>> Coordinators     q1-q2
>>>>>>>> GTM Proxy        q1-q2
>>>>>>>> GTM      q3
>>>>>>>> Version 9.1.7 (tar)
>>>>>>>>
>>>>>>>> MySQL Cluster:
>>>>>>>> ndbmtd   q1-q2 (striped)
>>>>>>>> mysqld   q1-q2
>>>>>>>> ndb_mgmd         q1
>>>>>>>> Version 7.1.17 (oracle rhel 5)
>>>>>>>>
>>>>>>>> q1, q2, q3 are identical boxes each with:
>>>>>>>>   24g ram, 1 cpu (6 cores), middle-tier raid with battery backed
>>>>>>>>ram
>>>>>>>>and 3
>>>>>>>> platters in raid 5.
>>>>>>>> They are lan local with a vlan-aware switch (I don't know which one
>>>>>>>>and I
>>>>>>>> didn't check to see if it was saturated).
>>>>>>>>
>>>>>>>>
>>>>>>>> Tables & Data:
>>>>>>>>
>>>>>>>> 161 tables with primary key ("part" int not null, "key1" bigint not
>>>>>>>>null,
>>>>>>>> "key2" bigint not null).  Each table has a selection of varchar
>>>>>>>>columns
>>>>>>>> typically less than varchar(80).  All non-primary-key columns have
>>>>>>>> non-unique indexes.  5.8 million rows were imported with a
>>>>>>>>different
>>>>>>>> number of rows to each table.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>--------------------------------------------------------------------
>>>>>>>>--
>>>>>>>>--------
>>>>>>>> Everyone hates slow websites. So do we.
>>>>>>>> Make your web apps faster with AppDynamics
>>>>>>>> Download AppDynamics Lite for free today:
>>>>>>>> http://p.sf.net/sfu/appdyn_d2d_mar
>>>>>>>> _______________________________________________
>>>>>>>> Postgres-xc-general mailing list
>>>>>>>> Pos...@li...
>>>>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>>>>>
>>>>>>>
>>>>>>>---------------------------------------------------------------------
>>>>>>>--
>>>>>>>-------
>>>>>>> Everyone hates slow websites. So do we.
>>>>>>> Make your web apps faster with AppDynamics
>>>>>>> Download AppDynamics Lite for free today:
>>>>>>> http://p.sf.net/sfu/appdyn_d2d_mar
>>>>>>> _______________________________________________
>>>>>>> Postgres-xc-general mailing list
>>>>>>> Pos...@li...
>>>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>>>>
>>>>>>
>>>>>>----------------------------------------------------------------------
>>>>>>--
>>>>>>------
>>>>>> Everyone hates slow websites. So do we.
>>>>>> Make your web apps faster with AppDynamics
>>>>>> Download AppDynamics Lite for free today:
>>>>>> http://p.sf.net/sfu/appdyn_d2d_mar
>>>>>> _______________________________________________
>>>>>> Postgres-xc-general mailing list
>>>>>> Pos...@li...
>>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>
>

I managed to get some more instrumentation results...

PG waffles between disk-bound and CPU bound.  CPU wavers around 90% across all CPUs and the disk spikes up and down from 25% to 95%.

PG-XC (using GTM-proxy) seems to be CPU bound with around 95% CPU throughout the test.  One CPU core is consistently nearly 100% throughout the test on both datanodes.  Disk is high but not as high or as spikey as PG, ranging from 20% (most of the time) to 80%.

I haven't been able to figure out how much memory the raid card has.  It is an HPSA 410i which comes in at least 3 variants.

I will run the fsync-off test asap but that probably won't be 'til tomorrow (deadlines).  Honestly, though, I don't care that NDB is faster - the difference between PG and NDB is meaningless since I know that's apples-to-oranges (I need it as a known baseline for a presentation so I included it here).  I'm also going to run this test using 3 datanode servers to see how the bottlenecks change.

Chris...

________________________________________
From: Jim Mlodgenski [ji...@gm...]
Sent: March 19, 2013 10:23 AM
To: Chris Brown
Cc: pos...@li...
Subject: Re: [Postgres-xc-general] PG-XC Performance

How big is the RAID controller's cache? You're probably pushing
several gig of data through it on you're XC and PostgreSQL tests so
you're likely flooding it to the point where it needs to wait to flush
to disk even when you're adding the second XC datanode.

If you wanted to have a more apples-to-apples comparison to NDB, you
can shut off some of the durability requirements that PostgreSQL has
on by default.
http://www.postgresql.org/docs/9.1/static/non-durability.html

fsync=off
full_page_writes=off
synchronous_commit=off
checkpoint_segments=64

You wouldn't run with these setting in production unless you are OK
with potentially corrupted data, but it does get you close to what NDB
does under the covers.

On Tue, Mar 19, 2013 at 1:07 PM, Chris Brown <cb...@in...> wrote:
> Yes, by round robin so it's not perfect, but it's probably good enough.
>
> On 13-03-19 10:06 AM, "Jim Mlodgenski" <ji...@gm...> wrote:
>
>>For your XC test, are you splitting your connections across the
>>coordinators with half of your psql sessions pointing to q1 and half
>>to q2?
>>
>>On Tue, Mar 19, 2013 at 1:01 PM, Chris Brown <cb...@in...> wrote:
>>> For PG (no-XC) it looks like I'm pegged on disk (still getting better
>>> instrumentation).  I would expect that my disk usage would be nearly
>>> halved when striping data - still gathering info though.
>>>
>>> On 13-03-19 8:40 AM, "Jim Mlodgenski" <ji...@gm...> wrote:
>>>
>>>>It may be easier to just concentrate on the PostgreSQL test to find
>>>>the bottleneck. The fact that the XC test (without GTM Proxy) and the
>>>>PostgreSQL test is so similar, is leading me to believe you're hitting
>>>>the same bottleneck.
>>>>
>>>>On Tue, Mar 19, 2013 at 11:35 AM, Chris Brown <cb...@in...>
>>>>wrote:
>>>>> I assume I'm I/O bound.  The CPU on the datanode/coordinator/gtm_proxy
>>>>>boxes gets to around 85%.  I'll need some more instrumentation to be
>>>>>sure - I'll get back to you with details.
>>>>>
>>>>> Chris...
>>>>>
>>>>> ________________________________________
>>>>> From: Jim Mlodgenski [ji...@gm...]
>>>>> Sent: March 19, 2013 8:32 AM
>>>>> To: Chris Brown
>>>>> Cc: pos...@li...
>>>>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>>>>
>>>>> When running the test against straight PostgreSQL, where is the
>>>>> bottleneck? Are you CPU bound or IO bound?
>>>>>
>>>>> On Tue, Mar 19, 2013 at 10:59 AM, Chris Brown <cb...@in...>
>>>>>wrote:
>>>>>> I also just discovered that my assumption about postgres (non xc)
>>>>>>being 9.2 was incorrect:
>>>>>>   postgresql-server-9.1.7-1.fc16.x86_64
>>>>>> Oops, sorry about the misinformation.
>>>>>>
>>>>>> Chris...
>>>>>> ________________________________________
>>>>>> From: Chris Brown [cb...@in...]
>>>>>> Sent: March 19, 2013 7:45 AM
>>>>>> To: Jim Mlodgenski
>>>>>> Cc: pos...@li...
>>>>>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>>>>>
>>>>>> Hi Jim,
>>>>>>
>>>>>> I believe I'm doing an fsync (fdatasync) for each commit.  I'm doing
>>>>>>the inserts as actual inserts performed by psql.  They are not batched
>>>>>>- there is one insert per line and each file contains about 10,000
>>>>>>inserts.  This was a conscious choice since I was testing expected
>>>>>>transaction rate as opposed to import rate.
>>>>>>
>>>>>> I haven't done much tuning to postgres.  Datanodes received this:
>>>>>>   checkpoint_segements=32
>>>>>>   effective_cache_size=4GB
>>>>>>   shared_buffers=4GB
>>>>>>   random_page_cost=3.0
>>>>>> For the coordinators, no tuning was done.
>>>>>> For xc, I also added:
>>>>>>   max_prepared_transactions = 10
>>>>>>   gtm_host = 'localhost'  # or the remote host, as appropriate on
>>>>>>coordinators
>>>>>>   gtm_port = 6666
>>>>>> Otherwise the postgresql.conf file is the default one provided by
>>>>>>initdb.  If I understand this, then in linux (3.1.9-1.fc16.x86_64) I
>>>>>>am
>>>>>>using the defaults of wal_level(minimum), fsync(on),
>>>>>>wal_sync_method(fdatasync).
>>>>>>
>>>>>> I reran the 60 client test with explicit DISTRIBUTE BY HASH and
>>>>>>MODULO, confirming the obj_MODULO and obj_HASH in the tables:
>>>>>>     "table_11_obj_MODULO_idx" btree ("obj_MODULO")
>>>>>> It didn't make an appreciable difference (~10 seconds on a 5m50s
>>>>>>test)
>>>>>>which leads me to believe that the 5m50s number is slightly in error.
>>>>>>
>>>>>> Chris...
>>>>>>
>>>>>> ________________________________________
>>>>>> From: Jim Mlodgenski [ji...@gm...]
>>>>>> Sent: March 19, 2013 7:26 AM
>>>>>> To: Chris Brown
>>>>>> Cc: Koichi Suzuki; pos...@li...
>>>>>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>>>>>
>>>>>> Can you send along some more details of the test? I find it odd that
>>>>>> in every scenario, the load times are consistent despite the change
>>>>>>in
>>>>>> the number of threads. Are you committing your transaction after each
>>>>>> insert? If so, your limiting factor is probably the fsync to disk
>>>>>> which would account for why PostgreSQL 9.2 is about the same speed as
>>>>>> XC 1.0.
>>>>>>
>>>>>> On Tue, Mar 19, 2013 at 10:06 AM, Chris Brown <cb...@in...>
>>>>>>wrote:
>>>>>>> Here's a sample table:
>>>>>>>
>>>>>>> CREATE TABLE table_11 (
>>>>>>>   "part" int not null,
>>>>>>>   "key1" bigint not null,
>>>>>>>   "key2" bigint not null,
>>>>>>>   "col_1" varchar(23),
>>>>>>>   "col_2" varchar(5),
>>>>>>>   "col_3" varchar(32),
>>>>>>>   "col_4" varchar(5),
>>>>>>>   PRIMARY KEY(part,key1,key2)
>>>>>>> );
>>>>>>>
>>>>>>> I guess I'm not specifying a distribution key.  I assumed, perhaps
>>>>>>>incorrectly, that hash on the first element of the primary key would
>>>>>>>be the default.  I will try a run with an explicit DISTRIBUTE BY
>>>>>>>HASH(part).
>>>>>>>
>>>>>>> Chris...
>>>>>>> ________________________________________
>>>>>>> From: Koichi Suzuki [koi...@gm...]
>>>>>>> Sent: March 18, 2013 10:47 PM
>>>>>>> To: Chris Brown
>>>>>>> Cc: pos...@li...
>>>>>>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>>>>>>
>>>>>>> What kind of distribution did you use in your case?   If all the
>>>>>>> tables are replicated, all the inserts are associated with implicit
>>>>>>> 2PCs.   Frequently-updated tables should be distributed (hash will
>>>>>>>be
>>>>>>> better if the table has a suitable column).
>>>>>>>
>>>>>>> Regards;
>>>>>>> ----------
>>>>>>> Koichi Suzuki
>>>>>>>
>>>>>>>
>>>>>>> 2013/3/19 Chris Brown <cb...@in...>:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> I've been doing some performance testing of PostgreSQL-XC.  My
>>>>>>>>workload is
>>>>>>>> all insert in this case.  I was expecting PG-XC to perform worse
>>>>>>>>than the
>>>>>>>> alternatives but I was shocked by how much worse it was than
>>>>>>>>PostgreSQL.
>>>>>>>> I'm well aware that NDB has durability and features issues so I
>>>>>>>>expected
>>>>>>>> it to perform better - I don't consider the 0.8 to be particularly
>>>>>>>> interesting.  I'd like anyone's thoughts on this.
>>>>>>>>
>>>>>>>> Chris...
>>>>>>>>
>>>>>>>>
>>>>>>>> Results:
>>>>>>>>
>>>>>>>> All results are very approximate!
>>>>>>>> CPU is an estimate on the number of cores used by server processes
>>>>>>>> (ndbmtd, ndb_mgmd, postgresql, gtm, gtm_proxy).
>>>>>>>> TC  is a comparison of (Time*CPU) using PostgreSQL as the baseline
>>>>>>>>and is
>>>>>>>> intended to represent a comparative CPU cost for a single insert
>>>>>>>> transaction.
>>>>>>>>
>>>>>>>> Client   PostgreSQL               PostgreSQL-XC             MySQL
>>>>>>>>Cluster
>>>>>>>>           PostgreSQL-XC-2
>>>>>>>> Threads  Time   CPU        TC     Time   CPU         TC     Time
>>>>>>>>CPU
>>>>>>>>    TC     Time   CPU         TC
>>>>>>>> 16       5m08s  4.3 cores  1.0
>>>>>>>> 24       4m56s  4.6 cores  1.0    6m10s  10.7 cores  2.9
>>>>>>>> 32       4m48s  4.7 cores  1.0    5m26s  11.9 cores  2.9    2m27s
>>>>>>>>7.8
>>>>>>>> cores  0.8
>>>>>>>> 48       4m49s  4.7 cores  1.0    5m01s  12.6 cores  2.8    2m16s
>>>>>>>>8.0
>>>>>>>> cores  0.8    6m06s  10.2 cores  2.7
>>>>>>>> 64                                5m01s  13.1 cores  2.9    2m11s
>>>>>>>>8.3
>>>>>>>> cores  0.8    5m50s  10.5 cores  2.7
>>>>>>>> 96                                                          2m13s
>>>>>>>>8.4
>>>>>>>> cores  0.8
>>>>>>>>
>>>>>>>>
>>>>>>>> Configurations:
>>>>>>>>
>>>>>>>> PostgreSQL:
>>>>>>>> Datanodes        q1
>>>>>>>> Version: 9.2.x (redhat)
>>>>>>>>
>>>>>>>> PostgreSQL-XC:
>>>>>>>> Datanodes        q1-q2 (striped)
>>>>>>>> Coordinators     q1-q2
>>>>>>>> GTM      q3
>>>>>>>> Version 9.1.7 (tar)
>>>>>>>>
>>>>>>>> PostgreSQL-XC-2:
>>>>>>>> Datanodes        q1-q2 (striped)
>>>>>>>> Coordinators     q1-q2
>>>>>>>> GTM Proxy        q1-q2
>>>>>>>> GTM      q3
>>>>>>>> Version 9.1.7 (tar)
>>>>>>>>
>>>>>>>> MySQL Cluster:
>>>>>>>> ndbmtd   q1-q2 (striped)
>>>>>>>> mysqld   q1-q2
>>>>>>>> ndb_mgmd         q1
>>>>>>>> Version 7.1.17 (oracle rhel 5)
>>>>>>>>
>>>>>>>> q1, q2, q3 are identical boxes each with:
>>>>>>>>   24g ram, 1 cpu (6 cores), middle-tier raid with battery backed
>>>>>>>>ram
>>>>>>>>and 3
>>>>>>>> platters in raid 5.
>>>>>>>> They are lan local with a vlan-aware switch (I don't know which one
>>>>>>>>and I
>>>>>>>> didn't check to see if it was saturated).
>>>>>>>>
>>>>>>>>
>>>>>>>> Tables & Data:
>>>>>>>>
>>>>>>>> 161 tables with primary key ("part" int not null, "key1" bigint not
>>>>>>>>null,
>>>>>>>> "key2" bigint not null).  Each table has a selection of varchar
>>>>>>>>columns
>>>>>>>> typically less than varchar(80).  All non-primary-key columns have
>>>>>>>> non-unique indexes.  5.8 million rows were imported with a
>>>>>>>>different
>>>>>>>> number of rows to each table.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>--------------------------------------------------------------------
>>>>>>>>--
>>>>>>>>--------
>>>>>>>> Everyone hates slow websites. So do we.
>>>>>>>> Make your web apps faster with AppDynamics
>>>>>>>> Download AppDynamics Lite for free today:
>>>>>>>> http://p.sf.net/sfu/appdyn_d2d_mar
>>>>>>>> _______________________________________________
>>>>>>>> Postgres-xc-general mailing list
>>>>>>>> Pos...@li...
>>>>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>>>>>
>>>>>>>
>>>>>>>---------------------------------------------------------------------
>>>>>>>--
>>>>>>>-------
>>>>>>> Everyone hates slow websites. So do we.
>>>>>>> Make your web apps faster with AppDynamics
>>>>>>> Download AppDynamics Lite for free today:
>>>>>>> http://p.sf.net/sfu/appdyn_d2d_mar
>>>>>>> _______________________________________________
>>>>>>> Postgres-xc-general mailing list
>>>>>>> Pos...@li...
>>>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>>>>
>>>>>>
>>>>>>----------------------------------------------------------------------
>>>>>>--
>>>>>>------
>>>>>> Everyone hates slow websites. So do we.
>>>>>> Make your web apps faster with AppDynamics
>>>>>> Download AppDynamics Lite for free today:
>>>>>> http://p.sf.net/sfu/appdyn_d2d_mar
>>>>>> _______________________________________________
>>>>>> Postgres-xc-general mailing list
>>>>>> Pos...@li...
>>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>
>

How big is the RAID controller's cache? You're probably pushing
several gig of data through it on you're XC and PostgreSQL tests so
you're likely flooding it to the point where it needs to wait to flush
to disk even when you're adding the second XC datanode.

If you wanted to have a more apples-to-apples comparison to NDB, you
can shut off some of the durability requirements that PostgreSQL has
on by default.
http://www.postgresql.org/docs/9.1/static/non-durability.html

fsync=off
full_page_writes=off
synchronous_commit=off
checkpoint_segments=64

You wouldn't run with these setting in production unless you are OK
with potentially corrupted data, but it does get you close to what NDB
does under the covers.

On Tue, Mar 19, 2013 at 1:07 PM, Chris Brown <cb...@in...> wrote:
> Yes, by round robin so it's not perfect, but it's probably good enough.
>
> On 13-03-19 10:06 AM, "Jim Mlodgenski" <ji...@gm...> wrote:
>
>>For your XC test, are you splitting your connections across the
>>coordinators with half of your psql sessions pointing to q1 and half
>>to q2?
>>
>>On Tue, Mar 19, 2013 at 1:01 PM, Chris Brown <cb...@in...> wrote:
>>> For PG (no-XC) it looks like I'm pegged on disk (still getting better
>>> instrumentation).  I would expect that my disk usage would be nearly
>>> halved when striping data - still gathering info though.
>>>
>>> On 13-03-19 8:40 AM, "Jim Mlodgenski" <ji...@gm...> wrote:
>>>
>>>>It may be easier to just concentrate on the PostgreSQL test to find
>>>>the bottleneck. The fact that the XC test (without GTM Proxy) and the
>>>>PostgreSQL test is so similar, is leading me to believe you're hitting
>>>>the same bottleneck.
>>>>
>>>>On Tue, Mar 19, 2013 at 11:35 AM, Chris Brown <cb...@in...>
>>>>wrote:
>>>>> I assume I'm I/O bound.  The CPU on the datanode/coordinator/gtm_proxy
>>>>>boxes gets to around 85%.  I'll need some more instrumentation to be
>>>>>sure - I'll get back to you with details.
>>>>>
>>>>> Chris...
>>>>>
>>>>> ________________________________________
>>>>> From: Jim Mlodgenski [ji...@gm...]
>>>>> Sent: March 19, 2013 8:32 AM
>>>>> To: Chris Brown
>>>>> Cc: pos...@li...
>>>>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>>>>
>>>>> When running the test against straight PostgreSQL, where is the
>>>>> bottleneck? Are you CPU bound or IO bound?
>>>>>
>>>>> On Tue, Mar 19, 2013 at 10:59 AM, Chris Brown <cb...@in...>
>>>>>wrote:
>>>>>> I also just discovered that my assumption about postgres (non xc)
>>>>>>being 9.2 was incorrect:
>>>>>>   postgresql-server-9.1.7-1.fc16.x86_64
>>>>>> Oops, sorry about the misinformation.
>>>>>>
>>>>>> Chris...
>>>>>> ________________________________________
>>>>>> From: Chris Brown [cb...@in...]
>>>>>> Sent: March 19, 2013 7:45 AM
>>>>>> To: Jim Mlodgenski
>>>>>> Cc: pos...@li...
>>>>>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>>>>>
>>>>>> Hi Jim,
>>>>>>
>>>>>> I believe I'm doing an fsync (fdatasync) for each commit.  I'm doing
>>>>>>the inserts as actual inserts performed by psql.  They are not batched
>>>>>>- there is one insert per line and each file contains about 10,000
>>>>>>inserts.  This was a conscious choice since I was testing expected
>>>>>>transaction rate as opposed to import rate.
>>>>>>
>>>>>> I haven't done much tuning to postgres.  Datanodes received this:
>>>>>>   checkpoint_segements=32
>>>>>>   effective_cache_size=4GB
>>>>>>   shared_buffers=4GB
>>>>>>   random_page_cost=3.0
>>>>>> For the coordinators, no tuning was done.
>>>>>> For xc, I also added:
>>>>>>   max_prepared_transactions = 10
>>>>>>   gtm_host = 'localhost'  # or the remote host, as appropriate on
>>>>>>coordinators
>>>>>>   gtm_port = 6666
>>>>>> Otherwise the postgresql.conf file is the default one provided by
>>>>>>initdb.  If I understand this, then in linux (3.1.9-1.fc16.x86_64) I
>>>>>>am
>>>>>>using the defaults of wal_level(minimum), fsync(on),
>>>>>>wal_sync_method(fdatasync).
>>>>>>
>>>>>> I reran the 60 client test with explicit DISTRIBUTE BY HASH and
>>>>>>MODULO, confirming the obj_MODULO and obj_HASH in the tables:
>>>>>>     "table_11_obj_MODULO_idx" btree ("obj_MODULO")
>>>>>> It didn't make an appreciable difference (~10 seconds on a 5m50s
>>>>>>test)
>>>>>>which leads me to believe that the 5m50s number is slightly in error.
>>>>>>
>>>>>> Chris...
>>>>>>
>>>>>> ________________________________________
>>>>>> From: Jim Mlodgenski [ji...@gm...]
>>>>>> Sent: March 19, 2013 7:26 AM
>>>>>> To: Chris Brown
>>>>>> Cc: Koichi Suzuki; pos...@li...
>>>>>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>>>>>
>>>>>> Can you send along some more details of the test? I find it odd that
>>>>>> in every scenario, the load times are consistent despite the change
>>>>>>in
>>>>>> the number of threads. Are you committing your transaction after each
>>>>>> insert? If so, your limiting factor is probably the fsync to disk
>>>>>> which would account for why PostgreSQL 9.2 is about the same speed as
>>>>>> XC 1.0.
>>>>>>
>>>>>> On Tue, Mar 19, 2013 at 10:06 AM, Chris Brown <cb...@in...>
>>>>>>wrote:
>>>>>>> Here's a sample table:
>>>>>>>
>>>>>>> CREATE TABLE table_11 (
>>>>>>>   "part" int not null,
>>>>>>>   "key1" bigint not null,
>>>>>>>   "key2" bigint not null,
>>>>>>>   "col_1" varchar(23),
>>>>>>>   "col_2" varchar(5),
>>>>>>>   "col_3" varchar(32),
>>>>>>>   "col_4" varchar(5),
>>>>>>>   PRIMARY KEY(part,key1,key2)
>>>>>>> );
>>>>>>>
>>>>>>> I guess I'm not specifying a distribution key.  I assumed, perhaps
>>>>>>>incorrectly, that hash on the first element of the primary key would
>>>>>>>be the default.  I will try a run with an explicit DISTRIBUTE BY
>>>>>>>HASH(part).
>>>>>>>
>>>>>>> Chris...
>>>>>>> ________________________________________
>>>>>>> From: Koichi Suzuki [koi...@gm...]
>>>>>>> Sent: March 18, 2013 10:47 PM
>>>>>>> To: Chris Brown
>>>>>>> Cc: pos...@li...
>>>>>>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>>>>>>
>>>>>>> What kind of distribution did you use in your case?   If all the
>>>>>>> tables are replicated, all the inserts are associated with implicit
>>>>>>> 2PCs.   Frequently-updated tables should be distributed (hash will
>>>>>>>be
>>>>>>> better if the table has a suitable column).
>>>>>>>
>>>>>>> Regards;
>>>>>>> ----------
>>>>>>> Koichi Suzuki
>>>>>>>
>>>>>>>
>>>>>>> 2013/3/19 Chris Brown <cb...@in...>:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> I've been doing some performance testing of PostgreSQL-XC.  My
>>>>>>>>workload is
>>>>>>>> all insert in this case.  I was expecting PG-XC to perform worse
>>>>>>>>than the
>>>>>>>> alternatives but I was shocked by how much worse it was than
>>>>>>>>PostgreSQL.
>>>>>>>> I'm well aware that NDB has durability and features issues so I
>>>>>>>>expected
>>>>>>>> it to perform better - I don't consider the 0.8 to be particularly
>>>>>>>> interesting.  I'd like anyone's thoughts on this.
>>>>>>>>
>>>>>>>> Chris...
>>>>>>>>
>>>>>>>>
>>>>>>>> Results:
>>>>>>>>
>>>>>>>> All results are very approximate!
>>>>>>>> CPU is an estimate on the number of cores used by server processes
>>>>>>>> (ndbmtd, ndb_mgmd, postgresql, gtm, gtm_proxy).
>>>>>>>> TC  is a comparison of (Time*CPU) using PostgreSQL as the baseline
>>>>>>>>and is
>>>>>>>> intended to represent a comparative CPU cost for a single insert
>>>>>>>> transaction.
>>>>>>>>
>>>>>>>> Client   PostgreSQL               PostgreSQL-XC             MySQL
>>>>>>>>Cluster
>>>>>>>>           PostgreSQL-XC-2
>>>>>>>> Threads  Time   CPU        TC     Time   CPU         TC     Time
>>>>>>>>CPU
>>>>>>>>    TC     Time   CPU         TC
>>>>>>>> 16       5m08s  4.3 cores  1.0
>>>>>>>> 24       4m56s  4.6 cores  1.0    6m10s  10.7 cores  2.9
>>>>>>>> 32       4m48s  4.7 cores  1.0    5m26s  11.9 cores  2.9    2m27s
>>>>>>>>7.8
>>>>>>>> cores  0.8
>>>>>>>> 48       4m49s  4.7 cores  1.0    5m01s  12.6 cores  2.8    2m16s
>>>>>>>>8.0
>>>>>>>> cores  0.8    6m06s  10.2 cores  2.7
>>>>>>>> 64                                5m01s  13.1 cores  2.9    2m11s
>>>>>>>>8.3
>>>>>>>> cores  0.8    5m50s  10.5 cores  2.7
>>>>>>>> 96                                                          2m13s
>>>>>>>>8.4
>>>>>>>> cores  0.8
>>>>>>>>
>>>>>>>>
>>>>>>>> Configurations:
>>>>>>>>
>>>>>>>> PostgreSQL:
>>>>>>>> Datanodes        q1
>>>>>>>> Version: 9.2.x (redhat)
>>>>>>>>
>>>>>>>> PostgreSQL-XC:
>>>>>>>> Datanodes        q1-q2 (striped)
>>>>>>>> Coordinators     q1-q2
>>>>>>>> GTM      q3
>>>>>>>> Version 9.1.7 (tar)
>>>>>>>>
>>>>>>>> PostgreSQL-XC-2:
>>>>>>>> Datanodes        q1-q2 (striped)
>>>>>>>> Coordinators     q1-q2
>>>>>>>> GTM Proxy        q1-q2
>>>>>>>> GTM      q3
>>>>>>>> Version 9.1.7 (tar)
>>>>>>>>
>>>>>>>> MySQL Cluster:
>>>>>>>> ndbmtd   q1-q2 (striped)
>>>>>>>> mysqld   q1-q2
>>>>>>>> ndb_mgmd         q1
>>>>>>>> Version 7.1.17 (oracle rhel 5)
>>>>>>>>
>>>>>>>> q1, q2, q3 are identical boxes each with:
>>>>>>>>   24g ram, 1 cpu (6 cores), middle-tier raid with battery backed
>>>>>>>>ram
>>>>>>>>and 3
>>>>>>>> platters in raid 5.
>>>>>>>> They are lan local with a vlan-aware switch (I don't know which one
>>>>>>>>and I
>>>>>>>> didn't check to see if it was saturated).
>>>>>>>>
>>>>>>>>
>>>>>>>> Tables & Data:
>>>>>>>>
>>>>>>>> 161 tables with primary key ("part" int not null, "key1" bigint not
>>>>>>>>null,
>>>>>>>> "key2" bigint not null).  Each table has a selection of varchar
>>>>>>>>columns
>>>>>>>> typically less than varchar(80).  All non-primary-key columns have
>>>>>>>> non-unique indexes.  5.8 million rows were imported with a
>>>>>>>>different
>>>>>>>> number of rows to each table.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>--------------------------------------------------------------------
>>>>>>>>--
>>>>>>>>--------
>>>>>>>> Everyone hates slow websites. So do we.
>>>>>>>> Make your web apps faster with AppDynamics
>>>>>>>> Download AppDynamics Lite for free today:
>>>>>>>> http://p.sf.net/sfu/appdyn_d2d_mar
>>>>>>>> _______________________________________________
>>>>>>>> Postgres-xc-general mailing list
>>>>>>>> Pos...@li...
>>>>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>>>>>
>>>>>>>
>>>>>>>---------------------------------------------------------------------
>>>>>>>--
>>>>>>>-------
>>>>>>> Everyone hates slow websites. So do we.
>>>>>>> Make your web apps faster with AppDynamics
>>>>>>> Download AppDynamics Lite for free today:
>>>>>>> http://p.sf.net/sfu/appdyn_d2d_mar
>>>>>>> _______________________________________________
>>>>>>> Postgres-xc-general mailing list
>>>>>>> Pos...@li...
>>>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>>>>
>>>>>>
>>>>>>----------------------------------------------------------------------
>>>>>>--
>>>>>>------
>>>>>> Everyone hates slow websites. So do we.
>>>>>> Make your web apps faster with AppDynamics
>>>>>> Download AppDynamics Lite for free today:
>>>>>> http://p.sf.net/sfu/appdyn_d2d_mar
>>>>>> _______________________________________________
>>>>>> Postgres-xc-general mailing list
>>>>>> Pos...@li...
>>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>
>

Yes, by round robin so it's not perfect, but it's probably good enough.

On 13-03-19 10:06 AM, "Jim Mlodgenski" <ji...@gm...> wrote:

>For your XC test, are you splitting your connections across the
>coordinators with half of your psql sessions pointing to q1 and half
>to q2?
>
>On Tue, Mar 19, 2013 at 1:01 PM, Chris Brown <cb...@in...> wrote:
>> For PG (no-XC) it looks like I'm pegged on disk (still getting better
>> instrumentation).  I would expect that my disk usage would be nearly
>> halved when striping data - still gathering info though.
>>
>> On 13-03-19 8:40 AM, "Jim Mlodgenski" <ji...@gm...> wrote:
>>
>>>It may be easier to just concentrate on the PostgreSQL test to find
>>>the bottleneck. The fact that the XC test (without GTM Proxy) and the
>>>PostgreSQL test is so similar, is leading me to believe you're hitting
>>>the same bottleneck.
>>>
>>>On Tue, Mar 19, 2013 at 11:35 AM, Chris Brown <cb...@in...>
>>>wrote:
>>>> I assume I'm I/O bound.  The CPU on the datanode/coordinator/gtm_proxy
>>>>boxes gets to around 85%.  I'll need some more instrumentation to be
>>>>sure - I'll get back to you with details.
>>>>
>>>> Chris...
>>>>
>>>> ________________________________________
>>>> From: Jim Mlodgenski [ji...@gm...]
>>>> Sent: March 19, 2013 8:32 AM
>>>> To: Chris Brown
>>>> Cc: pos...@li...
>>>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>>>
>>>> When running the test against straight PostgreSQL, where is the
>>>> bottleneck? Are you CPU bound or IO bound?
>>>>
>>>> On Tue, Mar 19, 2013 at 10:59 AM, Chris Brown <cb...@in...>
>>>>wrote:
>>>>> I also just discovered that my assumption about postgres (non xc)
>>>>>being 9.2 was incorrect:
>>>>>   postgresql-server-9.1.7-1.fc16.x86_64
>>>>> Oops, sorry about the misinformation.
>>>>>
>>>>> Chris...
>>>>> ________________________________________
>>>>> From: Chris Brown [cb...@in...]
>>>>> Sent: March 19, 2013 7:45 AM
>>>>> To: Jim Mlodgenski
>>>>> Cc: pos...@li...
>>>>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>>>>
>>>>> Hi Jim,
>>>>>
>>>>> I believe I'm doing an fsync (fdatasync) for each commit.  I'm doing
>>>>>the inserts as actual inserts performed by psql.  They are not batched
>>>>>- there is one insert per line and each file contains about 10,000
>>>>>inserts.  This was a conscious choice since I was testing expected
>>>>>transaction rate as opposed to import rate.
>>>>>
>>>>> I haven't done much tuning to postgres.  Datanodes received this:
>>>>>   checkpoint_segements=32
>>>>>   effective_cache_size=4GB
>>>>>   shared_buffers=4GB
>>>>>   random_page_cost=3.0
>>>>> For the coordinators, no tuning was done.
>>>>> For xc, I also added:
>>>>>   max_prepared_transactions = 10
>>>>>   gtm_host = 'localhost'  # or the remote host, as appropriate on
>>>>>coordinators
>>>>>   gtm_port = 6666
>>>>> Otherwise the postgresql.conf file is the default one provided by
>>>>>initdb.  If I understand this, then in linux (3.1.9-1.fc16.x86_64) I
>>>>>am
>>>>>using the defaults of wal_level(minimum), fsync(on),
>>>>>wal_sync_method(fdatasync).
>>>>>
>>>>> I reran the 60 client test with explicit DISTRIBUTE BY HASH and
>>>>>MODULO, confirming the obj_MODULO and obj_HASH in the tables:
>>>>>     "table_11_obj_MODULO_idx" btree ("obj_MODULO")
>>>>> It didn't make an appreciable difference (~10 seconds on a 5m50s
>>>>>test)
>>>>>which leads me to believe that the 5m50s number is slightly in error.
>>>>>
>>>>> Chris...
>>>>>
>>>>> ________________________________________
>>>>> From: Jim Mlodgenski [ji...@gm...]
>>>>> Sent: March 19, 2013 7:26 AM
>>>>> To: Chris Brown
>>>>> Cc: Koichi Suzuki; pos...@li...
>>>>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>>>>
>>>>> Can you send along some more details of the test? I find it odd that
>>>>> in every scenario, the load times are consistent despite the change
>>>>>in
>>>>> the number of threads. Are you committing your transaction after each
>>>>> insert? If so, your limiting factor is probably the fsync to disk
>>>>> which would account for why PostgreSQL 9.2 is about the same speed as
>>>>> XC 1.0.
>>>>>
>>>>> On Tue, Mar 19, 2013 at 10:06 AM, Chris Brown <cb...@in...>
>>>>>wrote:
>>>>>> Here's a sample table:
>>>>>>
>>>>>> CREATE TABLE table_11 (
>>>>>>   "part" int not null,
>>>>>>   "key1" bigint not null,
>>>>>>   "key2" bigint not null,
>>>>>>   "col_1" varchar(23),
>>>>>>   "col_2" varchar(5),
>>>>>>   "col_3" varchar(32),
>>>>>>   "col_4" varchar(5),
>>>>>>   PRIMARY KEY(part,key1,key2)
>>>>>> );
>>>>>>
>>>>>> I guess I'm not specifying a distribution key.  I assumed, perhaps
>>>>>>incorrectly, that hash on the first element of the primary key would
>>>>>>be the default.  I will try a run with an explicit DISTRIBUTE BY
>>>>>>HASH(part).
>>>>>>
>>>>>> Chris...
>>>>>> ________________________________________
>>>>>> From: Koichi Suzuki [koi...@gm...]
>>>>>> Sent: March 18, 2013 10:47 PM
>>>>>> To: Chris Brown
>>>>>> Cc: pos...@li...
>>>>>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>>>>>
>>>>>> What kind of distribution did you use in your case?   If all the
>>>>>> tables are replicated, all the inserts are associated with implicit
>>>>>> 2PCs.   Frequently-updated tables should be distributed (hash will
>>>>>>be
>>>>>> better if the table has a suitable column).
>>>>>>
>>>>>> Regards;
>>>>>> ----------
>>>>>> Koichi Suzuki
>>>>>>
>>>>>>
>>>>>> 2013/3/19 Chris Brown <cb...@in...>:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I've been doing some performance testing of PostgreSQL-XC.  My
>>>>>>>workload is
>>>>>>> all insert in this case.  I was expecting PG-XC to perform worse
>>>>>>>than the
>>>>>>> alternatives but I was shocked by how much worse it was than
>>>>>>>PostgreSQL.
>>>>>>> I'm well aware that NDB has durability and features issues so I
>>>>>>>expected
>>>>>>> it to perform better - I don't consider the 0.8 to be particularly
>>>>>>> interesting.  I'd like anyone's thoughts on this.
>>>>>>>
>>>>>>> Chris...
>>>>>>>
>>>>>>>
>>>>>>> Results:
>>>>>>>
>>>>>>> All results are very approximate!
>>>>>>> CPU is an estimate on the number of cores used by server processes
>>>>>>> (ndbmtd, ndb_mgmd, postgresql, gtm, gtm_proxy).
>>>>>>> TC  is a comparison of (Time*CPU) using PostgreSQL as the baseline
>>>>>>>and is
>>>>>>> intended to represent a comparative CPU cost for a single insert
>>>>>>> transaction.
>>>>>>>
>>>>>>> Client   PostgreSQL               PostgreSQL-XC             MySQL
>>>>>>>Cluster
>>>>>>>           PostgreSQL-XC-2
>>>>>>> Threads  Time   CPU        TC     Time   CPU         TC     Time
>>>>>>>CPU
>>>>>>>    TC     Time   CPU         TC
>>>>>>> 16       5m08s  4.3 cores  1.0
>>>>>>> 24       4m56s  4.6 cores  1.0    6m10s  10.7 cores  2.9
>>>>>>> 32       4m48s  4.7 cores  1.0    5m26s  11.9 cores  2.9    2m27s
>>>>>>>7.8
>>>>>>> cores  0.8
>>>>>>> 48       4m49s  4.7 cores  1.0    5m01s  12.6 cores  2.8    2m16s
>>>>>>>8.0
>>>>>>> cores  0.8    6m06s  10.2 cores  2.7
>>>>>>> 64                                5m01s  13.1 cores  2.9    2m11s
>>>>>>>8.3
>>>>>>> cores  0.8    5m50s  10.5 cores  2.7
>>>>>>> 96                                                          2m13s
>>>>>>>8.4
>>>>>>> cores  0.8
>>>>>>>
>>>>>>>
>>>>>>> Configurations:
>>>>>>>
>>>>>>> PostgreSQL:
>>>>>>> Datanodes        q1
>>>>>>> Version: 9.2.x (redhat)
>>>>>>>
>>>>>>> PostgreSQL-XC:
>>>>>>> Datanodes        q1-q2 (striped)
>>>>>>> Coordinators     q1-q2
>>>>>>> GTM      q3
>>>>>>> Version 9.1.7 (tar)
>>>>>>>
>>>>>>> PostgreSQL-XC-2:
>>>>>>> Datanodes        q1-q2 (striped)
>>>>>>> Coordinators     q1-q2
>>>>>>> GTM Proxy        q1-q2
>>>>>>> GTM      q3
>>>>>>> Version 9.1.7 (tar)
>>>>>>>
>>>>>>> MySQL Cluster:
>>>>>>> ndbmtd   q1-q2 (striped)
>>>>>>> mysqld   q1-q2
>>>>>>> ndb_mgmd         q1
>>>>>>> Version 7.1.17 (oracle rhel 5)
>>>>>>>
>>>>>>> q1, q2, q3 are identical boxes each with:
>>>>>>>   24g ram, 1 cpu (6 cores), middle-tier raid with battery backed
>>>>>>>ram
>>>>>>>and 3
>>>>>>> platters in raid 5.
>>>>>>> They are lan local with a vlan-aware switch (I don't know which one
>>>>>>>and I
>>>>>>> didn't check to see if it was saturated).
>>>>>>>
>>>>>>>
>>>>>>> Tables & Data:
>>>>>>>
>>>>>>> 161 tables with primary key ("part" int not null, "key1" bigint not
>>>>>>>null,
>>>>>>> "key2" bigint not null).  Each table has a selection of varchar
>>>>>>>columns
>>>>>>> typically less than varchar(80).  All non-primary-key columns have
>>>>>>> non-unique indexes.  5.8 million rows were imported with a
>>>>>>>different
>>>>>>> number of rows to each table.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>--------------------------------------------------------------------
>>>>>>>--
>>>>>>>--------
>>>>>>> Everyone hates slow websites. So do we.
>>>>>>> Make your web apps faster with AppDynamics
>>>>>>> Download AppDynamics Lite for free today:
>>>>>>> http://p.sf.net/sfu/appdyn_d2d_mar
>>>>>>> _______________________________________________
>>>>>>> Postgres-xc-general mailing list
>>>>>>> Pos...@li...
>>>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>>>>
>>>>>>
>>>>>>---------------------------------------------------------------------
>>>>>>--
>>>>>>-------
>>>>>> Everyone hates slow websites. So do we.
>>>>>> Make your web apps faster with AppDynamics
>>>>>> Download AppDynamics Lite for free today:
>>>>>> http://p.sf.net/sfu/appdyn_d2d_mar
>>>>>> _______________________________________________
>>>>>> Postgres-xc-general mailing list
>>>>>> Pos...@li...
>>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>>>
>>>>>
>>>>>----------------------------------------------------------------------
>>>>>--
>>>>>------
>>>>> Everyone hates slow websites. So do we.
>>>>> Make your web apps faster with AppDynamics
>>>>> Download AppDynamics Lite for free today:
>>>>> http://p.sf.net/sfu/appdyn_d2d_mar
>>>>> _______________________________________________
>>>>> Postgres-xc-general mailing list
>>>>> Pos...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>

For your XC test, are you splitting your connections across the
coordinators with half of your psql sessions pointing to q1 and half
to q2?

On Tue, Mar 19, 2013 at 1:01 PM, Chris Brown <cb...@in...> wrote:
> For PG (no-XC) it looks like I'm pegged on disk (still getting better
> instrumentation).  I would expect that my disk usage would be nearly
> halved when striping data - still gathering info though.
>
> On 13-03-19 8:40 AM, "Jim Mlodgenski" <ji...@gm...> wrote:
>
>>It may be easier to just concentrate on the PostgreSQL test to find
>>the bottleneck. The fact that the XC test (without GTM Proxy) and the
>>PostgreSQL test is so similar, is leading me to believe you're hitting
>>the same bottleneck.
>>
>>On Tue, Mar 19, 2013 at 11:35 AM, Chris Brown <cb...@in...> wrote:
>>> I assume I'm I/O bound.  The CPU on the datanode/coordinator/gtm_proxy
>>>boxes gets to around 85%.  I'll need some more instrumentation to be
>>>sure - I'll get back to you with details.
>>>
>>> Chris...
>>>
>>> ________________________________________
>>> From: Jim Mlodgenski [ji...@gm...]
>>> Sent: March 19, 2013 8:32 AM
>>> To: Chris Brown
>>> Cc: pos...@li...
>>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>>
>>> When running the test against straight PostgreSQL, where is the
>>> bottleneck? Are you CPU bound or IO bound?
>>>
>>> On Tue, Mar 19, 2013 at 10:59 AM, Chris Brown <cb...@in...>
>>>wrote:
>>>> I also just discovered that my assumption about postgres (non xc)
>>>>being 9.2 was incorrect:
>>>>   postgresql-server-9.1.7-1.fc16.x86_64
>>>> Oops, sorry about the misinformation.
>>>>
>>>> Chris...
>>>> ________________________________________
>>>> From: Chris Brown [cb...@in...]
>>>> Sent: March 19, 2013 7:45 AM
>>>> To: Jim Mlodgenski
>>>> Cc: pos...@li...
>>>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>>>
>>>> Hi Jim,
>>>>
>>>> I believe I'm doing an fsync (fdatasync) for each commit.  I'm doing
>>>>the inserts as actual inserts performed by psql.  They are not batched
>>>>- there is one insert per line and each file contains about 10,000
>>>>inserts.  This was a conscious choice since I was testing expected
>>>>transaction rate as opposed to import rate.
>>>>
>>>> I haven't done much tuning to postgres.  Datanodes received this:
>>>>   checkpoint_segements=32
>>>>   effective_cache_size=4GB
>>>>   shared_buffers=4GB
>>>>   random_page_cost=3.0
>>>> For the coordinators, no tuning was done.
>>>> For xc, I also added:
>>>>   max_prepared_transactions = 10
>>>>   gtm_host = 'localhost'  # or the remote host, as appropriate on
>>>>coordinators
>>>>   gtm_port = 6666
>>>> Otherwise the postgresql.conf file is the default one provided by
>>>>initdb.  If I understand this, then in linux (3.1.9-1.fc16.x86_64) I am
>>>>using the defaults of wal_level(minimum), fsync(on),
>>>>wal_sync_method(fdatasync).
>>>>
>>>> I reran the 60 client test with explicit DISTRIBUTE BY HASH and
>>>>MODULO, confirming the obj_MODULO and obj_HASH in the tables:
>>>>     "table_11_obj_MODULO_idx" btree ("obj_MODULO")
>>>> It didn't make an appreciable difference (~10 seconds on a 5m50s test)
>>>>which leads me to believe that the 5m50s number is slightly in error.
>>>>
>>>> Chris...
>>>>
>>>> ________________________________________
>>>> From: Jim Mlodgenski [ji...@gm...]
>>>> Sent: March 19, 2013 7:26 AM
>>>> To: Chris Brown
>>>> Cc: Koichi Suzuki; pos...@li...
>>>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>>>
>>>> Can you send along some more details of the test? I find it odd that
>>>> in every scenario, the load times are consistent despite the change in
>>>> the number of threads. Are you committing your transaction after each
>>>> insert? If so, your limiting factor is probably the fsync to disk
>>>> which would account for why PostgreSQL 9.2 is about the same speed as
>>>> XC 1.0.
>>>>
>>>> On Tue, Mar 19, 2013 at 10:06 AM, Chris Brown <cb...@in...>
>>>>wrote:
>>>>> Here's a sample table:
>>>>>
>>>>> CREATE TABLE table_11 (
>>>>>   "part" int not null,
>>>>>   "key1" bigint not null,
>>>>>   "key2" bigint not null,
>>>>>   "col_1" varchar(23),
>>>>>   "col_2" varchar(5),
>>>>>   "col_3" varchar(32),
>>>>>   "col_4" varchar(5),
>>>>>   PRIMARY KEY(part,key1,key2)
>>>>> );
>>>>>
>>>>> I guess I'm not specifying a distribution key.  I assumed, perhaps
>>>>>incorrectly, that hash on the first element of the primary key would
>>>>>be the default.  I will try a run with an explicit DISTRIBUTE BY
>>>>>HASH(part).
>>>>>
>>>>> Chris...
>>>>> ________________________________________
>>>>> From: Koichi Suzuki [koi...@gm...]
>>>>> Sent: March 18, 2013 10:47 PM
>>>>> To: Chris Brown
>>>>> Cc: pos...@li...
>>>>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>>>>
>>>>> What kind of distribution did you use in your case?   If all the
>>>>> tables are replicated, all the inserts are associated with implicit
>>>>> 2PCs.   Frequently-updated tables should be distributed (hash will be
>>>>> better if the table has a suitable column).
>>>>>
>>>>> Regards;
>>>>> ----------
>>>>> Koichi Suzuki
>>>>>
>>>>>
>>>>> 2013/3/19 Chris Brown <cb...@in...>:
>>>>>> Hi all,
>>>>>>
>>>>>> I've been doing some performance testing of PostgreSQL-XC.  My
>>>>>>workload is
>>>>>> all insert in this case.  I was expecting PG-XC to perform worse
>>>>>>than the
>>>>>> alternatives but I was shocked by how much worse it was than
>>>>>>PostgreSQL.
>>>>>> I'm well aware that NDB has durability and features issues so I
>>>>>>expected
>>>>>> it to perform better - I don't consider the 0.8 to be particularly
>>>>>> interesting.  I'd like anyone's thoughts on this.
>>>>>>
>>>>>> Chris...
>>>>>>
>>>>>>
>>>>>> Results:
>>>>>>
>>>>>> All results are very approximate!
>>>>>> CPU is an estimate on the number of cores used by server processes
>>>>>> (ndbmtd, ndb_mgmd, postgresql, gtm, gtm_proxy).
>>>>>> TC  is a comparison of (Time*CPU) using PostgreSQL as the baseline
>>>>>>and is
>>>>>> intended to represent a comparative CPU cost for a single insert
>>>>>> transaction.
>>>>>>
>>>>>> Client   PostgreSQL               PostgreSQL-XC             MySQL
>>>>>>Cluster
>>>>>>           PostgreSQL-XC-2
>>>>>> Threads  Time   CPU        TC     Time   CPU         TC     Time
>>>>>>CPU
>>>>>>    TC     Time   CPU         TC
>>>>>> 16       5m08s  4.3 cores  1.0
>>>>>> 24       4m56s  4.6 cores  1.0    6m10s  10.7 cores  2.9
>>>>>> 32       4m48s  4.7 cores  1.0    5m26s  11.9 cores  2.9    2m27s
>>>>>>7.8
>>>>>> cores  0.8
>>>>>> 48       4m49s  4.7 cores  1.0    5m01s  12.6 cores  2.8    2m16s
>>>>>>8.0
>>>>>> cores  0.8    6m06s  10.2 cores  2.7
>>>>>> 64                                5m01s  13.1 cores  2.9    2m11s
>>>>>>8.3
>>>>>> cores  0.8    5m50s  10.5 cores  2.7
>>>>>> 96                                                          2m13s
>>>>>>8.4
>>>>>> cores  0.8
>>>>>>
>>>>>>
>>>>>> Configurations:
>>>>>>
>>>>>> PostgreSQL:
>>>>>> Datanodes        q1
>>>>>> Version: 9.2.x (redhat)
>>>>>>
>>>>>> PostgreSQL-XC:
>>>>>> Datanodes        q1-q2 (striped)
>>>>>> Coordinators     q1-q2
>>>>>> GTM      q3
>>>>>> Version 9.1.7 (tar)
>>>>>>
>>>>>> PostgreSQL-XC-2:
>>>>>> Datanodes        q1-q2 (striped)
>>>>>> Coordinators     q1-q2
>>>>>> GTM Proxy        q1-q2
>>>>>> GTM      q3
>>>>>> Version 9.1.7 (tar)
>>>>>>
>>>>>> MySQL Cluster:
>>>>>> ndbmtd   q1-q2 (striped)
>>>>>> mysqld   q1-q2
>>>>>> ndb_mgmd         q1
>>>>>> Version 7.1.17 (oracle rhel 5)
>>>>>>
>>>>>> q1, q2, q3 are identical boxes each with:
>>>>>>   24g ram, 1 cpu (6 cores), middle-tier raid with battery backed ram
>>>>>>and 3
>>>>>> platters in raid 5.
>>>>>> They are lan local with a vlan-aware switch (I don't know which one
>>>>>>and I
>>>>>> didn't check to see if it was saturated).
>>>>>>
>>>>>>
>>>>>> Tables & Data:
>>>>>>
>>>>>> 161 tables with primary key ("part" int not null, "key1" bigint not
>>>>>>null,
>>>>>> "key2" bigint not null).  Each table has a selection of varchar
>>>>>>columns
>>>>>> typically less than varchar(80).  All non-primary-key columns have
>>>>>> non-unique indexes.  5.8 million rows were imported with a different
>>>>>> number of rows to each table.
>>>>>>
>>>>>>
>>>>>>
>>>>>>----------------------------------------------------------------------
>>>>>>--------
>>>>>> Everyone hates slow websites. So do we.
>>>>>> Make your web apps faster with AppDynamics
>>>>>> Download AppDynamics Lite for free today:
>>>>>> http://p.sf.net/sfu/appdyn_d2d_mar
>>>>>> _______________________________________________
>>>>>> Postgres-xc-general mailing list
>>>>>> Pos...@li...
>>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>>>
>>>>>
>>>>>-----------------------------------------------------------------------
>>>>>-------
>>>>> Everyone hates slow websites. So do we.
>>>>> Make your web apps faster with AppDynamics
>>>>> Download AppDynamics Lite for free today:
>>>>> http://p.sf.net/sfu/appdyn_d2d_mar
>>>>> _______________________________________________
>>>>> Postgres-xc-general mailing list
>>>>> Pos...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>>
>>>>
>>>>------------------------------------------------------------------------
>>>>------
>>>> Everyone hates slow websites. So do we.
>>>> Make your web apps faster with AppDynamics
>>>> Download AppDynamics Lite for free today:
>>>> http://p.sf.net/sfu/appdyn_d2d_mar
>>>> _______________________________________________
>>>> Postgres-xc-general mailing list
>>>> Pos...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>

For PG (no-XC) it looks like I'm pegged on disk (still getting better
instrumentation).  I would expect that my disk usage would be nearly
halved when striping data - still gathering info though.

On 13-03-19 8:40 AM, "Jim Mlodgenski" <ji...@gm...> wrote:

>It may be easier to just concentrate on the PostgreSQL test to find
>the bottleneck. The fact that the XC test (without GTM Proxy) and the
>PostgreSQL test is so similar, is leading me to believe you're hitting
>the same bottleneck.
>
>On Tue, Mar 19, 2013 at 11:35 AM, Chris Brown <cb...@in...> wrote:
>> I assume I'm I/O bound.  The CPU on the datanode/coordinator/gtm_proxy
>>boxes gets to around 85%.  I'll need some more instrumentation to be
>>sure - I'll get back to you with details.
>>
>> Chris...
>>
>> ________________________________________
>> From: Jim Mlodgenski [ji...@gm...]
>> Sent: March 19, 2013 8:32 AM
>> To: Chris Brown
>> Cc: pos...@li...
>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>
>> When running the test against straight PostgreSQL, where is the
>> bottleneck? Are you CPU bound or IO bound?
>>
>> On Tue, Mar 19, 2013 at 10:59 AM, Chris Brown <cb...@in...>
>>wrote:
>>> I also just discovered that my assumption about postgres (non xc)
>>>being 9.2 was incorrect:
>>>   postgresql-server-9.1.7-1.fc16.x86_64
>>> Oops, sorry about the misinformation.
>>>
>>> Chris...
>>> ________________________________________
>>> From: Chris Brown [cb...@in...]
>>> Sent: March 19, 2013 7:45 AM
>>> To: Jim Mlodgenski
>>> Cc: pos...@li...
>>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>>
>>> Hi Jim,
>>>
>>> I believe I'm doing an fsync (fdatasync) for each commit.  I'm doing
>>>the inserts as actual inserts performed by psql.  They are not batched
>>>- there is one insert per line and each file contains about 10,000
>>>inserts.  This was a conscious choice since I was testing expected
>>>transaction rate as opposed to import rate.
>>>
>>> I haven't done much tuning to postgres.  Datanodes received this:
>>>   checkpoint_segements=32
>>>   effective_cache_size=4GB
>>>   shared_buffers=4GB
>>>   random_page_cost=3.0
>>> For the coordinators, no tuning was done.
>>> For xc, I also added:
>>>   max_prepared_transactions = 10
>>>   gtm_host = 'localhost'  # or the remote host, as appropriate on
>>>coordinators
>>>   gtm_port = 6666
>>> Otherwise the postgresql.conf file is the default one provided by
>>>initdb.  If I understand this, then in linux (3.1.9-1.fc16.x86_64) I am
>>>using the defaults of wal_level(minimum), fsync(on),
>>>wal_sync_method(fdatasync).
>>>
>>> I reran the 60 client test with explicit DISTRIBUTE BY HASH and
>>>MODULO, confirming the obj_MODULO and obj_HASH in the tables:
>>>     "table_11_obj_MODULO_idx" btree ("obj_MODULO")
>>> It didn't make an appreciable difference (~10 seconds on a 5m50s test)
>>>which leads me to believe that the 5m50s number is slightly in error.
>>>
>>> Chris...
>>>
>>> ________________________________________
>>> From: Jim Mlodgenski [ji...@gm...]
>>> Sent: March 19, 2013 7:26 AM
>>> To: Chris Brown
>>> Cc: Koichi Suzuki; pos...@li...
>>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>>
>>> Can you send along some more details of the test? I find it odd that
>>> in every scenario, the load times are consistent despite the change in
>>> the number of threads. Are you committing your transaction after each
>>> insert? If so, your limiting factor is probably the fsync to disk
>>> which would account for why PostgreSQL 9.2 is about the same speed as
>>> XC 1.0.
>>>
>>> On Tue, Mar 19, 2013 at 10:06 AM, Chris Brown <cb...@in...>
>>>wrote:
>>>> Here's a sample table:
>>>>
>>>> CREATE TABLE table_11 (
>>>>   "part" int not null,
>>>>   "key1" bigint not null,
>>>>   "key2" bigint not null,
>>>>   "col_1" varchar(23),
>>>>   "col_2" varchar(5),
>>>>   "col_3" varchar(32),
>>>>   "col_4" varchar(5),
>>>>   PRIMARY KEY(part,key1,key2)
>>>> );
>>>>
>>>> I guess I'm not specifying a distribution key.  I assumed, perhaps
>>>>incorrectly, that hash on the first element of the primary key would
>>>>be the default.  I will try a run with an explicit DISTRIBUTE BY
>>>>HASH(part).
>>>>
>>>> Chris...
>>>> ________________________________________
>>>> From: Koichi Suzuki [koi...@gm...]
>>>> Sent: March 18, 2013 10:47 PM
>>>> To: Chris Brown
>>>> Cc: pos...@li...
>>>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>>>
>>>> What kind of distribution did you use in your case?   If all the
>>>> tables are replicated, all the inserts are associated with implicit
>>>> 2PCs.   Frequently-updated tables should be distributed (hash will be
>>>> better if the table has a suitable column).
>>>>
>>>> Regards;
>>>> ----------
>>>> Koichi Suzuki
>>>>
>>>>
>>>> 2013/3/19 Chris Brown <cb...@in...>:
>>>>> Hi all,
>>>>>
>>>>> I've been doing some performance testing of PostgreSQL-XC.  My
>>>>>workload is
>>>>> all insert in this case.  I was expecting PG-XC to perform worse
>>>>>than the
>>>>> alternatives but I was shocked by how much worse it was than
>>>>>PostgreSQL.
>>>>> I'm well aware that NDB has durability and features issues so I
>>>>>expected
>>>>> it to perform better - I don't consider the 0.8 to be particularly
>>>>> interesting.  I'd like anyone's thoughts on this.
>>>>>
>>>>> Chris...
>>>>>
>>>>>
>>>>> Results:
>>>>>
>>>>> All results are very approximate!
>>>>> CPU is an estimate on the number of cores used by server processes
>>>>> (ndbmtd, ndb_mgmd, postgresql, gtm, gtm_proxy).
>>>>> TC  is a comparison of (Time*CPU) using PostgreSQL as the baseline
>>>>>and is
>>>>> intended to represent a comparative CPU cost for a single insert
>>>>> transaction.
>>>>>
>>>>> Client   PostgreSQL               PostgreSQL-XC             MySQL
>>>>>Cluster
>>>>>           PostgreSQL-XC-2
>>>>> Threads  Time   CPU        TC     Time   CPU         TC     Time
>>>>>CPU
>>>>>    TC     Time   CPU         TC
>>>>> 16       5m08s  4.3 cores  1.0
>>>>> 24       4m56s  4.6 cores  1.0    6m10s  10.7 cores  2.9
>>>>> 32       4m48s  4.7 cores  1.0    5m26s  11.9 cores  2.9    2m27s
>>>>>7.8
>>>>> cores  0.8
>>>>> 48       4m49s  4.7 cores  1.0    5m01s  12.6 cores  2.8    2m16s
>>>>>8.0
>>>>> cores  0.8    6m06s  10.2 cores  2.7
>>>>> 64                                5m01s  13.1 cores  2.9    2m11s
>>>>>8.3
>>>>> cores  0.8    5m50s  10.5 cores  2.7
>>>>> 96                                                          2m13s
>>>>>8.4
>>>>> cores  0.8
>>>>>
>>>>>
>>>>> Configurations:
>>>>>
>>>>> PostgreSQL:
>>>>> Datanodes        q1
>>>>> Version: 9.2.x (redhat)
>>>>>
>>>>> PostgreSQL-XC:
>>>>> Datanodes        q1-q2 (striped)
>>>>> Coordinators     q1-q2
>>>>> GTM      q3
>>>>> Version 9.1.7 (tar)
>>>>>
>>>>> PostgreSQL-XC-2:
>>>>> Datanodes        q1-q2 (striped)
>>>>> Coordinators     q1-q2
>>>>> GTM Proxy        q1-q2
>>>>> GTM      q3
>>>>> Version 9.1.7 (tar)
>>>>>
>>>>> MySQL Cluster:
>>>>> ndbmtd   q1-q2 (striped)
>>>>> mysqld   q1-q2
>>>>> ndb_mgmd         q1
>>>>> Version 7.1.17 (oracle rhel 5)
>>>>>
>>>>> q1, q2, q3 are identical boxes each with:
>>>>>   24g ram, 1 cpu (6 cores), middle-tier raid with battery backed ram
>>>>>and 3
>>>>> platters in raid 5.
>>>>> They are lan local with a vlan-aware switch (I don't know which one
>>>>>and I
>>>>> didn't check to see if it was saturated).
>>>>>
>>>>>
>>>>> Tables & Data:
>>>>>
>>>>> 161 tables with primary key ("part" int not null, "key1" bigint not
>>>>>null,
>>>>> "key2" bigint not null).  Each table has a selection of varchar
>>>>>columns
>>>>> typically less than varchar(80).  All non-primary-key columns have
>>>>> non-unique indexes.  5.8 million rows were imported with a different
>>>>> number of rows to each table.
>>>>>
>>>>>
>>>>> 
>>>>>----------------------------------------------------------------------
>>>>>--------
>>>>> Everyone hates slow websites. So do we.
>>>>> Make your web apps faster with AppDynamics
>>>>> Download AppDynamics Lite for free today:
>>>>> http://p.sf.net/sfu/appdyn_d2d_mar
>>>>> _______________________________________________
>>>>> Postgres-xc-general mailing list
>>>>> Pos...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>>
>>>> 
>>>>-----------------------------------------------------------------------
>>>>-------
>>>> Everyone hates slow websites. So do we.
>>>> Make your web apps faster with AppDynamics
>>>> Download AppDynamics Lite for free today:
>>>> http://p.sf.net/sfu/appdyn_d2d_mar
>>>> _______________________________________________
>>>> Postgres-xc-general mailing list
>>>> Pos...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>
>>> 
>>>------------------------------------------------------------------------
>>>------
>>> Everyone hates slow websites. So do we.
>>> Make your web apps faster with AppDynamics
>>> Download AppDynamics Lite for free today:
>>> http://p.sf.net/sfu/appdyn_d2d_mar
>>> _______________________________________________
>>> Postgres-xc-general mailing list
>>> Pos...@li...
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

It may be easier to just concentrate on the PostgreSQL test to find
the bottleneck. The fact that the XC test (without GTM Proxy) and the
PostgreSQL test is so similar, is leading me to believe you're hitting
the same bottleneck.

On Tue, Mar 19, 2013 at 11:35 AM, Chris Brown <cb...@in...> wrote:
> I assume I'm I/O bound.  The CPU on the datanode/coordinator/gtm_proxy boxes gets to around 85%.  I'll need some more instrumentation to be sure - I'll get back to you with details.
>
> Chris...
>
> ________________________________________
> From: Jim Mlodgenski [ji...@gm...]
> Sent: March 19, 2013 8:32 AM
> To: Chris Brown
> Cc: pos...@li...
> Subject: Re: [Postgres-xc-general] PG-XC Performance
>
> When running the test against straight PostgreSQL, where is the
> bottleneck? Are you CPU bound or IO bound?
>
> On Tue, Mar 19, 2013 at 10:59 AM, Chris Brown <cb...@in...> wrote:
>> I also just discovered that my assumption about postgres (non xc) being 9.2 was incorrect:
>>   postgresql-server-9.1.7-1.fc16.x86_64
>> Oops, sorry about the misinformation.
>>
>> Chris...
>> ________________________________________
>> From: Chris Brown [cb...@in...]
>> Sent: March 19, 2013 7:45 AM
>> To: Jim Mlodgenski
>> Cc: pos...@li...
>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>
>> Hi Jim,
>>
>> I believe I'm doing an fsync (fdatasync) for each commit.  I'm doing the inserts as actual inserts performed by psql.  They are not batched - there is one insert per line and each file contains about 10,000 inserts.  This was a conscious choice since I was testing expected transaction rate as opposed to import rate.
>>
>> I haven't done much tuning to postgres.  Datanodes received this:
>>   checkpoint_segements=32
>>   effective_cache_size=4GB
>>   shared_buffers=4GB
>>   random_page_cost=3.0
>> For the coordinators, no tuning was done.
>> For xc, I also added:
>>   max_prepared_transactions = 10
>>   gtm_host = 'localhost'  # or the remote host, as appropriate on coordinators
>>   gtm_port = 6666
>> Otherwise the postgresql.conf file is the default one provided by initdb.  If I understand this, then in linux (3.1.9-1.fc16.x86_64) I am using the defaults of wal_level(minimum), fsync(on), wal_sync_method(fdatasync).
>>
>> I reran the 60 client test with explicit DISTRIBUTE BY HASH and MODULO, confirming the obj_MODULO and obj_HASH in the tables:
>>     "table_11_obj_MODULO_idx" btree ("obj_MODULO")
>> It didn't make an appreciable difference (~10 seconds on a 5m50s test) which leads me to believe that the 5m50s number is slightly in error.
>>
>> Chris...
>>
>> ________________________________________
>> From: Jim Mlodgenski [ji...@gm...]
>> Sent: March 19, 2013 7:26 AM
>> To: Chris Brown
>> Cc: Koichi Suzuki; pos...@li...
>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>
>> Can you send along some more details of the test? I find it odd that
>> in every scenario, the load times are consistent despite the change in
>> the number of threads. Are you committing your transaction after each
>> insert? If so, your limiting factor is probably the fsync to disk
>> which would account for why PostgreSQL 9.2 is about the same speed as
>> XC 1.0.
>>
>> On Tue, Mar 19, 2013 at 10:06 AM, Chris Brown <cb...@in...> wrote:
>>> Here's a sample table:
>>>
>>> CREATE TABLE table_11 (
>>>   "part" int not null,
>>>   "key1" bigint not null,
>>>   "key2" bigint not null,
>>>   "col_1" varchar(23),
>>>   "col_2" varchar(5),
>>>   "col_3" varchar(32),
>>>   "col_4" varchar(5),
>>>   PRIMARY KEY(part,key1,key2)
>>> );
>>>
>>> I guess I'm not specifying a distribution key.  I assumed, perhaps incorrectly, that hash on the first element of the primary key would be the default.  I will try a run with an explicit DISTRIBUTE BY HASH(part).
>>>
>>> Chris...
>>> ________________________________________
>>> From: Koichi Suzuki [koi...@gm...]
>>> Sent: March 18, 2013 10:47 PM
>>> To: Chris Brown
>>> Cc: pos...@li...
>>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>>
>>> What kind of distribution did you use in your case?   If all the
>>> tables are replicated, all the inserts are associated with implicit
>>> 2PCs.   Frequently-updated tables should be distributed (hash will be
>>> better if the table has a suitable column).
>>>
>>> Regards;
>>> ----------
>>> Koichi Suzuki
>>>
>>>
>>> 2013/3/19 Chris Brown <cb...@in...>:
>>>> Hi all,
>>>>
>>>> I've been doing some performance testing of PostgreSQL-XC.  My workload is
>>>> all insert in this case.  I was expecting PG-XC to perform worse than the
>>>> alternatives but I was shocked by how much worse it was than PostgreSQL.
>>>> I'm well aware that NDB has durability and features issues so I expected
>>>> it to perform better - I don't consider the 0.8 to be particularly
>>>> interesting.  I'd like anyone's thoughts on this.
>>>>
>>>> Chris...
>>>>
>>>>
>>>> Results:
>>>>
>>>> All results are very approximate!
>>>> CPU is an estimate on the number of cores used by server processes
>>>> (ndbmtd, ndb_mgmd, postgresql, gtm, gtm_proxy).
>>>> TC  is a comparison of (Time*CPU) using PostgreSQL as the baseline and is
>>>> intended to represent a comparative CPU cost for a single insert
>>>> transaction.
>>>>
>>>> Client   PostgreSQL               PostgreSQL-XC             MySQL Cluster
>>>>           PostgreSQL-XC-2
>>>> Threads  Time   CPU        TC     Time   CPU         TC     Time   CPU
>>>>    TC     Time   CPU         TC
>>>> 16       5m08s  4.3 cores  1.0
>>>> 24       4m56s  4.6 cores  1.0    6m10s  10.7 cores  2.9
>>>> 32       4m48s  4.7 cores  1.0    5m26s  11.9 cores  2.9    2m27s  7.8
>>>> cores  0.8
>>>> 48       4m49s  4.7 cores  1.0    5m01s  12.6 cores  2.8    2m16s  8.0
>>>> cores  0.8    6m06s  10.2 cores  2.7
>>>> 64                                5m01s  13.1 cores  2.9    2m11s  8.3
>>>> cores  0.8    5m50s  10.5 cores  2.7
>>>> 96                                                          2m13s  8.4
>>>> cores  0.8
>>>>
>>>>
>>>> Configurations:
>>>>
>>>> PostgreSQL:
>>>> Datanodes        q1
>>>> Version: 9.2.x (redhat)
>>>>
>>>> PostgreSQL-XC:
>>>> Datanodes        q1-q2 (striped)
>>>> Coordinators     q1-q2
>>>> GTM      q3
>>>> Version 9.1.7 (tar)
>>>>
>>>> PostgreSQL-XC-2:
>>>> Datanodes        q1-q2 (striped)
>>>> Coordinators     q1-q2
>>>> GTM Proxy        q1-q2
>>>> GTM      q3
>>>> Version 9.1.7 (tar)
>>>>
>>>> MySQL Cluster:
>>>> ndbmtd   q1-q2 (striped)
>>>> mysqld   q1-q2
>>>> ndb_mgmd         q1
>>>> Version 7.1.17 (oracle rhel 5)
>>>>
>>>> q1, q2, q3 are identical boxes each with:
>>>>   24g ram, 1 cpu (6 cores), middle-tier raid with battery backed ram and 3
>>>> platters in raid 5.
>>>> They are lan local with a vlan-aware switch (I don't know which one and I
>>>> didn't check to see if it was saturated).
>>>>
>>>>
>>>> Tables & Data:
>>>>
>>>> 161 tables with primary key ("part" int not null, "key1" bigint not null,
>>>> "key2" bigint not null).  Each table has a selection of varchar columns
>>>> typically less than varchar(80).  All non-primary-key columns have
>>>> non-unique indexes.  5.8 million rows were imported with a different
>>>> number of rows to each table.
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Everyone hates slow websites. So do we.
>>>> Make your web apps faster with AppDynamics
>>>> Download AppDynamics Lite for free today:
>>>> http://p.sf.net/sfu/appdyn_d2d_mar
>>>> _______________________________________________
>>>> Postgres-xc-general mailing list
>>>> Pos...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>
>>> ------------------------------------------------------------------------------
>>> Everyone hates slow websites. So do we.
>>> Make your web apps faster with AppDynamics
>>> Download AppDynamics Lite for free today:
>>> http://p.sf.net/sfu/appdyn_d2d_mar
>>> _______________________________________________
>>> Postgres-xc-general mailing list
>>> Pos...@li...
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>
>> ------------------------------------------------------------------------------
>> Everyone hates slow websites. So do we.
>> Make your web apps faster with AppDynamics
>> Download AppDynamics Lite for free today:
>> http://p.sf.net/sfu/appdyn_d2d_mar
>> _______________________________________________
>> Postgres-xc-general mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

I assume I'm I/O bound.  The CPU on the datanode/coordinator/gtm_proxy boxes gets to around 85%.  I'll need some more instrumentation to be sure - I'll get back to you with details.

Chris...

________________________________________
From: Jim Mlodgenski [ji...@gm...]
Sent: March 19, 2013 8:32 AM
To: Chris Brown
Cc: pos...@li...
Subject: Re: [Postgres-xc-general] PG-XC Performance

When running the test against straight PostgreSQL, where is the
bottleneck? Are you CPU bound or IO bound?

On Tue, Mar 19, 2013 at 10:59 AM, Chris Brown <cb...@in...> wrote:
> I also just discovered that my assumption about postgres (non xc) being 9.2 was incorrect:
>   postgresql-server-9.1.7-1.fc16.x86_64
> Oops, sorry about the misinformation.
>
> Chris...
> ________________________________________
> From: Chris Brown [cb...@in...]
> Sent: March 19, 2013 7:45 AM
> To: Jim Mlodgenski
> Cc: pos...@li...
> Subject: Re: [Postgres-xc-general] PG-XC Performance
>
> Hi Jim,
>
> I believe I'm doing an fsync (fdatasync) for each commit.  I'm doing the inserts as actual inserts performed by psql.  They are not batched - there is one insert per line and each file contains about 10,000 inserts.  This was a conscious choice since I was testing expected transaction rate as opposed to import rate.
>
> I haven't done much tuning to postgres.  Datanodes received this:
>   checkpoint_segements=32
>   effective_cache_size=4GB
>   shared_buffers=4GB
>   random_page_cost=3.0
> For the coordinators, no tuning was done.
> For xc, I also added:
>   max_prepared_transactions = 10
>   gtm_host = 'localhost'  # or the remote host, as appropriate on coordinators
>   gtm_port = 6666
> Otherwise the postgresql.conf file is the default one provided by initdb.  If I understand this, then in linux (3.1.9-1.fc16.x86_64) I am using the defaults of wal_level(minimum), fsync(on), wal_sync_method(fdatasync).
>
> I reran the 60 client test with explicit DISTRIBUTE BY HASH and MODULO, confirming the obj_MODULO and obj_HASH in the tables:
>     "table_11_obj_MODULO_idx" btree ("obj_MODULO")
> It didn't make an appreciable difference (~10 seconds on a 5m50s test) which leads me to believe that the 5m50s number is slightly in error.
>
> Chris...
>
> ________________________________________
> From: Jim Mlodgenski [ji...@gm...]
> Sent: March 19, 2013 7:26 AM
> To: Chris Brown
> Cc: Koichi Suzuki; pos...@li...
> Subject: Re: [Postgres-xc-general] PG-XC Performance
>
> Can you send along some more details of the test? I find it odd that
> in every scenario, the load times are consistent despite the change in
> the number of threads. Are you committing your transaction after each
> insert? If so, your limiting factor is probably the fsync to disk
> which would account for why PostgreSQL 9.2 is about the same speed as
> XC 1.0.
>
> On Tue, Mar 19, 2013 at 10:06 AM, Chris Brown <cb...@in...> wrote:
>> Here's a sample table:
>>
>> CREATE TABLE table_11 (
>>   "part" int not null,
>>   "key1" bigint not null,
>>   "key2" bigint not null,
>>   "col_1" varchar(23),
>>   "col_2" varchar(5),
>>   "col_3" varchar(32),
>>   "col_4" varchar(5),
>>   PRIMARY KEY(part,key1,key2)
>> );
>>
>> I guess I'm not specifying a distribution key.  I assumed, perhaps incorrectly, that hash on the first element of the primary key would be the default.  I will try a run with an explicit DISTRIBUTE BY HASH(part).
>>
>> Chris...
>> ________________________________________
>> From: Koichi Suzuki [koi...@gm...]
>> Sent: March 18, 2013 10:47 PM
>> To: Chris Brown
>> Cc: pos...@li...
>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>
>> What kind of distribution did you use in your case?   If all the
>> tables are replicated, all the inserts are associated with implicit
>> 2PCs.   Frequently-updated tables should be distributed (hash will be
>> better if the table has a suitable column).
>>
>> Regards;
>> ----------
>> Koichi Suzuki
>>
>>
>> 2013/3/19 Chris Brown <cb...@in...>:
>>> Hi all,
>>>
>>> I've been doing some performance testing of PostgreSQL-XC.  My workload is
>>> all insert in this case.  I was expecting PG-XC to perform worse than the
>>> alternatives but I was shocked by how much worse it was than PostgreSQL.
>>> I'm well aware that NDB has durability and features issues so I expected
>>> it to perform better - I don't consider the 0.8 to be particularly
>>> interesting.  I'd like anyone's thoughts on this.
>>>
>>> Chris...
>>>
>>>
>>> Results:
>>>
>>> All results are very approximate!
>>> CPU is an estimate on the number of cores used by server processes
>>> (ndbmtd, ndb_mgmd, postgresql, gtm, gtm_proxy).
>>> TC  is a comparison of (Time*CPU) using PostgreSQL as the baseline and is
>>> intended to represent a comparative CPU cost for a single insert
>>> transaction.
>>>
>>> Client   PostgreSQL               PostgreSQL-XC             MySQL Cluster
>>>           PostgreSQL-XC-2
>>> Threads  Time   CPU        TC     Time   CPU         TC     Time   CPU
>>>    TC     Time   CPU         TC
>>> 16       5m08s  4.3 cores  1.0
>>> 24       4m56s  4.6 cores  1.0    6m10s  10.7 cores  2.9
>>> 32       4m48s  4.7 cores  1.0    5m26s  11.9 cores  2.9    2m27s  7.8
>>> cores  0.8
>>> 48       4m49s  4.7 cores  1.0    5m01s  12.6 cores  2.8    2m16s  8.0
>>> cores  0.8    6m06s  10.2 cores  2.7
>>> 64                                5m01s  13.1 cores  2.9    2m11s  8.3
>>> cores  0.8    5m50s  10.5 cores  2.7
>>> 96                                                          2m13s  8.4
>>> cores  0.8
>>>
>>>
>>> Configurations:
>>>
>>> PostgreSQL:
>>> Datanodes        q1
>>> Version: 9.2.x (redhat)
>>>
>>> PostgreSQL-XC:
>>> Datanodes        q1-q2 (striped)
>>> Coordinators     q1-q2
>>> GTM      q3
>>> Version 9.1.7 (tar)
>>>
>>> PostgreSQL-XC-2:
>>> Datanodes        q1-q2 (striped)
>>> Coordinators     q1-q2
>>> GTM Proxy        q1-q2
>>> GTM      q3
>>> Version 9.1.7 (tar)
>>>
>>> MySQL Cluster:
>>> ndbmtd   q1-q2 (striped)
>>> mysqld   q1-q2
>>> ndb_mgmd         q1
>>> Version 7.1.17 (oracle rhel 5)
>>>
>>> q1, q2, q3 are identical boxes each with:
>>>   24g ram, 1 cpu (6 cores), middle-tier raid with battery backed ram and 3
>>> platters in raid 5.
>>> They are lan local with a vlan-aware switch (I don't know which one and I
>>> didn't check to see if it was saturated).
>>>
>>>
>>> Tables & Data:
>>>
>>> 161 tables with primary key ("part" int not null, "key1" bigint not null,
>>> "key2" bigint not null).  Each table has a selection of varchar columns
>>> typically less than varchar(80).  All non-primary-key columns have
>>> non-unique indexes.  5.8 million rows were imported with a different
>>> number of rows to each table.
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Everyone hates slow websites. So do we.
>>> Make your web apps faster with AppDynamics
>>> Download AppDynamics Lite for free today:
>>> http://p.sf.net/sfu/appdyn_d2d_mar
>>> _______________________________________________
>>> Postgres-xc-general mailing list
>>> Pos...@li...
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>
>> ------------------------------------------------------------------------------
>> Everyone hates slow websites. So do we.
>> Make your web apps faster with AppDynamics
>> Download AppDynamics Lite for free today:
>> http://p.sf.net/sfu/appdyn_d2d_mar
>> _______________________________________________
>> Postgres-xc-general mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_mar
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

When running the test against straight PostgreSQL, where is the
bottleneck? Are you CPU bound or IO bound?

On Tue, Mar 19, 2013 at 10:59 AM, Chris Brown <cb...@in...> wrote:
> I also just discovered that my assumption about postgres (non xc) being 9.2 was incorrect:
>   postgresql-server-9.1.7-1.fc16.x86_64
> Oops, sorry about the misinformation.
>
> Chris...
> ________________________________________
> From: Chris Brown [cb...@in...]
> Sent: March 19, 2013 7:45 AM
> To: Jim Mlodgenski
> Cc: pos...@li...
> Subject: Re: [Postgres-xc-general] PG-XC Performance
>
> Hi Jim,
>
> I believe I'm doing an fsync (fdatasync) for each commit.  I'm doing the inserts as actual inserts performed by psql.  They are not batched - there is one insert per line and each file contains about 10,000 inserts.  This was a conscious choice since I was testing expected transaction rate as opposed to import rate.
>
> I haven't done much tuning to postgres.  Datanodes received this:
>   checkpoint_segements=32
>   effective_cache_size=4GB
>   shared_buffers=4GB
>   random_page_cost=3.0
> For the coordinators, no tuning was done.
> For xc, I also added:
>   max_prepared_transactions = 10
>   gtm_host = 'localhost'  # or the remote host, as appropriate on coordinators
>   gtm_port = 6666
> Otherwise the postgresql.conf file is the default one provided by initdb.  If I understand this, then in linux (3.1.9-1.fc16.x86_64) I am using the defaults of wal_level(minimum), fsync(on), wal_sync_method(fdatasync).
>
> I reran the 60 client test with explicit DISTRIBUTE BY HASH and MODULO, confirming the obj_MODULO and obj_HASH in the tables:
>     "table_11_obj_MODULO_idx" btree ("obj_MODULO")
> It didn't make an appreciable difference (~10 seconds on a 5m50s test) which leads me to believe that the 5m50s number is slightly in error.
>
> Chris...
>
> ________________________________________
> From: Jim Mlodgenski [ji...@gm...]
> Sent: March 19, 2013 7:26 AM
> To: Chris Brown
> Cc: Koichi Suzuki; pos...@li...
> Subject: Re: [Postgres-xc-general] PG-XC Performance
>
> Can you send along some more details of the test? I find it odd that
> in every scenario, the load times are consistent despite the change in
> the number of threads. Are you committing your transaction after each
> insert? If so, your limiting factor is probably the fsync to disk
> which would account for why PostgreSQL 9.2 is about the same speed as
> XC 1.0.
>
> On Tue, Mar 19, 2013 at 10:06 AM, Chris Brown <cb...@in...> wrote:
>> Here's a sample table:
>>
>> CREATE TABLE table_11 (
>>   "part" int not null,
>>   "key1" bigint not null,
>>   "key2" bigint not null,
>>   "col_1" varchar(23),
>>   "col_2" varchar(5),
>>   "col_3" varchar(32),
>>   "col_4" varchar(5),
>>   PRIMARY KEY(part,key1,key2)
>> );
>>
>> I guess I'm not specifying a distribution key.  I assumed, perhaps incorrectly, that hash on the first element of the primary key would be the default.  I will try a run with an explicit DISTRIBUTE BY HASH(part).
>>
>> Chris...
>> ________________________________________
>> From: Koichi Suzuki [koi...@gm...]
>> Sent: March 18, 2013 10:47 PM
>> To: Chris Brown
>> Cc: pos...@li...
>> Subject: Re: [Postgres-xc-general] PG-XC Performance
>>
>> What kind of distribution did you use in your case?   If all the
>> tables are replicated, all the inserts are associated with implicit
>> 2PCs.   Frequently-updated tables should be distributed (hash will be
>> better if the table has a suitable column).
>>
>> Regards;
>> ----------
>> Koichi Suzuki
>>
>>
>> 2013/3/19 Chris Brown <cb...@in...>:
>>> Hi all,
>>>
>>> I've been doing some performance testing of PostgreSQL-XC.  My workload is
>>> all insert in this case.  I was expecting PG-XC to perform worse than the
>>> alternatives but I was shocked by how much worse it was than PostgreSQL.
>>> I'm well aware that NDB has durability and features issues so I expected
>>> it to perform better - I don't consider the 0.8 to be particularly
>>> interesting.  I'd like anyone's thoughts on this.
>>>
>>> Chris...
>>>
>>>
>>> Results:
>>>
>>> All results are very approximate!
>>> CPU is an estimate on the number of cores used by server processes
>>> (ndbmtd, ndb_mgmd, postgresql, gtm, gtm_proxy).
>>> TC  is a comparison of (Time*CPU) using PostgreSQL as the baseline and is
>>> intended to represent a comparative CPU cost for a single insert
>>> transaction.
>>>
>>> Client   PostgreSQL               PostgreSQL-XC             MySQL Cluster
>>>           PostgreSQL-XC-2
>>> Threads  Time   CPU        TC     Time   CPU         TC     Time   CPU
>>>    TC     Time   CPU         TC
>>> 16       5m08s  4.3 cores  1.0
>>> 24       4m56s  4.6 cores  1.0    6m10s  10.7 cores  2.9
>>> 32       4m48s  4.7 cores  1.0    5m26s  11.9 cores  2.9    2m27s  7.8
>>> cores  0.8
>>> 48       4m49s  4.7 cores  1.0    5m01s  12.6 cores  2.8    2m16s  8.0
>>> cores  0.8    6m06s  10.2 cores  2.7
>>> 64                                5m01s  13.1 cores  2.9    2m11s  8.3
>>> cores  0.8    5m50s  10.5 cores  2.7
>>> 96                                                          2m13s  8.4
>>> cores  0.8
>>>
>>>
>>> Configurations:
>>>
>>> PostgreSQL:
>>> Datanodes        q1
>>> Version: 9.2.x (redhat)
>>>
>>> PostgreSQL-XC:
>>> Datanodes        q1-q2 (striped)
>>> Coordinators     q1-q2
>>> GTM      q3
>>> Version 9.1.7 (tar)
>>>
>>> PostgreSQL-XC-2:
>>> Datanodes        q1-q2 (striped)
>>> Coordinators     q1-q2
>>> GTM Proxy        q1-q2
>>> GTM      q3
>>> Version 9.1.7 (tar)
>>>
>>> MySQL Cluster:
>>> ndbmtd   q1-q2 (striped)
>>> mysqld   q1-q2
>>> ndb_mgmd         q1
>>> Version 7.1.17 (oracle rhel 5)
>>>
>>> q1, q2, q3 are identical boxes each with:
>>>   24g ram, 1 cpu (6 cores), middle-tier raid with battery backed ram and 3
>>> platters in raid 5.
>>> They are lan local with a vlan-aware switch (I don't know which one and I
>>> didn't check to see if it was saturated).
>>>
>>>
>>> Tables & Data:
>>>
>>> 161 tables with primary key ("part" int not null, "key1" bigint not null,
>>> "key2" bigint not null).  Each table has a selection of varchar columns
>>> typically less than varchar(80).  All non-primary-key columns have
>>> non-unique indexes.  5.8 million rows were imported with a different
>>> number of rows to each table.
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Everyone hates slow websites. So do we.
>>> Make your web apps faster with AppDynamics
>>> Download AppDynamics Lite for free today:
>>> http://p.sf.net/sfu/appdyn_d2d_mar
>>> _______________________________________________
>>> Postgres-xc-general mailing list
>>> Pos...@li...
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>
>> ------------------------------------------------------------------------------
>> Everyone hates slow websites. So do we.
>> Make your web apps faster with AppDynamics
>> Download AppDynamics Lite for free today:
>> http://p.sf.net/sfu/appdyn_d2d_mar
>> _______________________________________________
>> Postgres-xc-general mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_mar
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

I also just discovered that my assumption about postgres (non xc) being 9.2 was incorrect:
  postgresql-server-9.1.7-1.fc16.x86_64
Oops, sorry about the misinformation.

Chris...
________________________________________
From: Chris Brown [cb...@in...]
Sent: March 19, 2013 7:45 AM
To: Jim Mlodgenski
Cc: pos...@li...
Subject: Re: [Postgres-xc-general] PG-XC Performance

Hi Jim,

I believe I'm doing an fsync (fdatasync) for each commit.  I'm doing the inserts as actual inserts performed by psql.  They are not batched - there is one insert per line and each file contains about 10,000 inserts.  This was a conscious choice since I was testing expected transaction rate as opposed to import rate.

I haven't done much tuning to postgres.  Datanodes received this:
  checkpoint_segements=32
  effective_cache_size=4GB
  shared_buffers=4GB
  random_page_cost=3.0
For the coordinators, no tuning was done.
For xc, I also added:
  max_prepared_transactions = 10
  gtm_host = 'localhost'  # or the remote host, as appropriate on coordinators
  gtm_port = 6666
Otherwise the postgresql.conf file is the default one provided by initdb.  If I understand this, then in linux (3.1.9-1.fc16.x86_64) I am using the defaults of wal_level(minimum), fsync(on), wal_sync_method(fdatasync).

I reran the 60 client test with explicit DISTRIBUTE BY HASH and MODULO, confirming the obj_MODULO and obj_HASH in the tables:
    "table_11_obj_MODULO_idx" btree ("obj_MODULO")
It didn't make an appreciable difference (~10 seconds on a 5m50s test) which leads me to believe that the 5m50s number is slightly in error.

Chris...

________________________________________
From: Jim Mlodgenski [ji...@gm...]
Sent: March 19, 2013 7:26 AM
To: Chris Brown
Cc: Koichi Suzuki; pos...@li...
Subject: Re: [Postgres-xc-general] PG-XC Performance

Can you send along some more details of the test? I find it odd that
in every scenario, the load times are consistent despite the change in
the number of threads. Are you committing your transaction after each
insert? If so, your limiting factor is probably the fsync to disk
which would account for why PostgreSQL 9.2 is about the same speed as
XC 1.0.

On Tue, Mar 19, 2013 at 10:06 AM, Chris Brown <cb...@in...> wrote:
> Here's a sample table:
>
> CREATE TABLE table_11 (
>   "part" int not null,
>   "key1" bigint not null,
>   "key2" bigint not null,
>   "col_1" varchar(23),
>   "col_2" varchar(5),
>   "col_3" varchar(32),
>   "col_4" varchar(5),
>   PRIMARY KEY(part,key1,key2)
> );
>
> I guess I'm not specifying a distribution key.  I assumed, perhaps incorrectly, that hash on the first element of the primary key would be the default.  I will try a run with an explicit DISTRIBUTE BY HASH(part).
>
> Chris...
> ________________________________________
> From: Koichi Suzuki [koi...@gm...]
> Sent: March 18, 2013 10:47 PM
> To: Chris Brown
> Cc: pos...@li...
> Subject: Re: [Postgres-xc-general] PG-XC Performance
>
> What kind of distribution did you use in your case?   If all the
> tables are replicated, all the inserts are associated with implicit
> 2PCs.   Frequently-updated tables should be distributed (hash will be
> better if the table has a suitable column).
>
> Regards;
> ----------
> Koichi Suzuki
>
>
> 2013/3/19 Chris Brown <cb...@in...>:
>> Hi all,
>>
>> I've been doing some performance testing of PostgreSQL-XC.  My workload is
>> all insert in this case.  I was expecting PG-XC to perform worse than the
>> alternatives but I was shocked by how much worse it was than PostgreSQL.
>> I'm well aware that NDB has durability and features issues so I expected
>> it to perform better - I don't consider the 0.8 to be particularly
>> interesting.  I'd like anyone's thoughts on this.
>>
>> Chris...
>>
>>
>> Results:
>>
>> All results are very approximate!
>> CPU is an estimate on the number of cores used by server processes
>> (ndbmtd, ndb_mgmd, postgresql, gtm, gtm_proxy).
>> TC  is a comparison of (Time*CPU) using PostgreSQL as the baseline and is
>> intended to represent a comparative CPU cost for a single insert
>> transaction.
>>
>> Client   PostgreSQL               PostgreSQL-XC             MySQL Cluster
>>           PostgreSQL-XC-2
>> Threads  Time   CPU        TC     Time   CPU         TC     Time   CPU
>>    TC     Time   CPU         TC
>> 16       5m08s  4.3 cores  1.0
>> 24       4m56s  4.6 cores  1.0    6m10s  10.7 cores  2.9
>> 32       4m48s  4.7 cores  1.0    5m26s  11.9 cores  2.9    2m27s  7.8
>> cores  0.8
>> 48       4m49s  4.7 cores  1.0    5m01s  12.6 cores  2.8    2m16s  8.0
>> cores  0.8    6m06s  10.2 cores  2.7
>> 64                                5m01s  13.1 cores  2.9    2m11s  8.3
>> cores  0.8    5m50s  10.5 cores  2.7
>> 96                                                          2m13s  8.4
>> cores  0.8
>>
>>
>> Configurations:
>>
>> PostgreSQL:
>> Datanodes        q1
>> Version: 9.2.x (redhat)
>>
>> PostgreSQL-XC:
>> Datanodes        q1-q2 (striped)
>> Coordinators     q1-q2
>> GTM      q3
>> Version 9.1.7 (tar)
>>
>> PostgreSQL-XC-2:
>> Datanodes        q1-q2 (striped)
>> Coordinators     q1-q2
>> GTM Proxy        q1-q2
>> GTM      q3
>> Version 9.1.7 (tar)
>>
>> MySQL Cluster:
>> ndbmtd   q1-q2 (striped)
>> mysqld   q1-q2
>> ndb_mgmd         q1
>> Version 7.1.17 (oracle rhel 5)
>>
>> q1, q2, q3 are identical boxes each with:
>>   24g ram, 1 cpu (6 cores), middle-tier raid with battery backed ram and 3
>> platters in raid 5.
>> They are lan local with a vlan-aware switch (I don't know which one and I
>> didn't check to see if it was saturated).
>>
>>
>> Tables & Data:
>>
>> 161 tables with primary key ("part" int not null, "key1" bigint not null,
>> "key2" bigint not null).  Each table has a selection of varchar columns
>> typically less than varchar(80).  All non-primary-key columns have
>> non-unique indexes.  5.8 million rows were imported with a different
>> number of rows to each table.
>>
>>
>> ------------------------------------------------------------------------------
>> Everyone hates slow websites. So do we.
>> Make your web apps faster with AppDynamics
>> Download AppDynamics Lite for free today:
>> http://p.sf.net/sfu/appdyn_d2d_mar
>> _______________________________________________
>> Postgres-xc-general mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_mar
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_mar
_______________________________________________
Postgres-xc-general mailing list
Pos...@li...
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

Hi Jim,

I believe I'm doing an fsync (fdatasync) for each commit.  I'm doing the inserts as actual inserts performed by psql.  They are not batched - there is one insert per line and each file contains about 10,000 inserts.  This was a conscious choice since I was testing expected transaction rate as opposed to import rate.

I haven't done much tuning to postgres.  Datanodes received this:
  checkpoint_segements=32
  effective_cache_size=4GB
  shared_buffers=4GB
  random_page_cost=3.0
For the coordinators, no tuning was done.
For xc, I also added:
  max_prepared_transactions = 10
  gtm_host = 'localhost'  # or the remote host, as appropriate on coordinators
  gtm_port = 6666
Otherwise the postgresql.conf file is the default one provided by initdb.  If I understand this, then in linux (3.1.9-1.fc16.x86_64) I am using the defaults of wal_level(minimum), fsync(on), wal_sync_method(fdatasync).

I reran the 60 client test with explicit DISTRIBUTE BY HASH and MODULO, confirming the obj_MODULO and obj_HASH in the tables:
    "table_11_obj_MODULO_idx" btree ("obj_MODULO")
It didn't make an appreciable difference (~10 seconds on a 5m50s test) which leads me to believe that the 5m50s number is slightly in error.

Chris...

________________________________________
From: Jim Mlodgenski [ji...@gm...]
Sent: March 19, 2013 7:26 AM
To: Chris Brown
Cc: Koichi Suzuki; pos...@li...
Subject: Re: [Postgres-xc-general] PG-XC Performance

Can you send along some more details of the test? I find it odd that
in every scenario, the load times are consistent despite the change in
the number of threads. Are you committing your transaction after each
insert? If so, your limiting factor is probably the fsync to disk
which would account for why PostgreSQL 9.2 is about the same speed as
XC 1.0.

On Tue, Mar 19, 2013 at 10:06 AM, Chris Brown <cb...@in...> wrote:
> Here's a sample table:
>
> CREATE TABLE table_11 (
>   "part" int not null,
>   "key1" bigint not null,
>   "key2" bigint not null,
>   "col_1" varchar(23),
>   "col_2" varchar(5),
>   "col_3" varchar(32),
>   "col_4" varchar(5),
>   PRIMARY KEY(part,key1,key2)
> );
>
> I guess I'm not specifying a distribution key.  I assumed, perhaps incorrectly, that hash on the first element of the primary key would be the default.  I will try a run with an explicit DISTRIBUTE BY HASH(part).
>
> Chris...
> ________________________________________
> From: Koichi Suzuki [koi...@gm...]
> Sent: March 18, 2013 10:47 PM
> To: Chris Brown
> Cc: pos...@li...
> Subject: Re: [Postgres-xc-general] PG-XC Performance
>
> What kind of distribution did you use in your case?   If all the
> tables are replicated, all the inserts are associated with implicit
> 2PCs.   Frequently-updated tables should be distributed (hash will be
> better if the table has a suitable column).
>
> Regards;
> ----------
> Koichi Suzuki
>
>
> 2013/3/19 Chris Brown <cb...@in...>:
>> Hi all,
>>
>> I've been doing some performance testing of PostgreSQL-XC.  My workload is
>> all insert in this case.  I was expecting PG-XC to perform worse than the
>> alternatives but I was shocked by how much worse it was than PostgreSQL.
>> I'm well aware that NDB has durability and features issues so I expected
>> it to perform better - I don't consider the 0.8 to be particularly
>> interesting.  I'd like anyone's thoughts on this.
>>
>> Chris...
>>
>>
>> Results:
>>
>> All results are very approximate!
>> CPU is an estimate on the number of cores used by server processes
>> (ndbmtd, ndb_mgmd, postgresql, gtm, gtm_proxy).
>> TC  is a comparison of (Time*CPU) using PostgreSQL as the baseline and is
>> intended to represent a comparative CPU cost for a single insert
>> transaction.
>>
>> Client   PostgreSQL               PostgreSQL-XC             MySQL Cluster
>>           PostgreSQL-XC-2
>> Threads  Time   CPU        TC     Time   CPU         TC     Time   CPU
>>    TC     Time   CPU         TC
>> 16       5m08s  4.3 cores  1.0
>> 24       4m56s  4.6 cores  1.0    6m10s  10.7 cores  2.9
>> 32       4m48s  4.7 cores  1.0    5m26s  11.9 cores  2.9    2m27s  7.8
>> cores  0.8
>> 48       4m49s  4.7 cores  1.0    5m01s  12.6 cores  2.8    2m16s  8.0
>> cores  0.8    6m06s  10.2 cores  2.7
>> 64                                5m01s  13.1 cores  2.9    2m11s  8.3
>> cores  0.8    5m50s  10.5 cores  2.7
>> 96                                                          2m13s  8.4
>> cores  0.8
>>
>>
>> Configurations:
>>
>> PostgreSQL:
>> Datanodes        q1
>> Version: 9.2.x (redhat)
>>
>> PostgreSQL-XC:
>> Datanodes        q1-q2 (striped)
>> Coordinators     q1-q2
>> GTM      q3
>> Version 9.1.7 (tar)
>>
>> PostgreSQL-XC-2:
>> Datanodes        q1-q2 (striped)
>> Coordinators     q1-q2
>> GTM Proxy        q1-q2
>> GTM      q3
>> Version 9.1.7 (tar)
>>
>> MySQL Cluster:
>> ndbmtd   q1-q2 (striped)
>> mysqld   q1-q2
>> ndb_mgmd         q1
>> Version 7.1.17 (oracle rhel 5)
>>
>> q1, q2, q3 are identical boxes each with:
>>   24g ram, 1 cpu (6 cores), middle-tier raid with battery backed ram and 3
>> platters in raid 5.
>> They are lan local with a vlan-aware switch (I don't know which one and I
>> didn't check to see if it was saturated).
>>
>>
>> Tables & Data:
>>
>> 161 tables with primary key ("part" int not null, "key1" bigint not null,
>> "key2" bigint not null).  Each table has a selection of varchar columns
>> typically less than varchar(80).  All non-primary-key columns have
>> non-unique indexes.  5.8 million rows were imported with a different
>> number of rows to each table.
>>
>>
>> ------------------------------------------------------------------------------
>> Everyone hates slow websites. So do we.
>> Make your web apps faster with AppDynamics
>> Download AppDynamics Lite for free today:
>> http://p.sf.net/sfu/appdyn_d2d_mar
>> _______________________________________________
>> Postgres-xc-general mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_mar
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

Can you send along some more details of the test? I find it odd that
in every scenario, the load times are consistent despite the change in
the number of threads. Are you committing your transaction after each
insert? If so, your limiting factor is probably the fsync to disk
which would account for why PostgreSQL 9.2 is about the same speed as
XC 1.0.

On Tue, Mar 19, 2013 at 10:06 AM, Chris Brown <cb...@in...> wrote:
> Here's a sample table:
>
> CREATE TABLE table_11 (
>   "part" int not null,
>   "key1" bigint not null,
>   "key2" bigint not null,
>   "col_1" varchar(23),
>   "col_2" varchar(5),
>   "col_3" varchar(32),
>   "col_4" varchar(5),
>   PRIMARY KEY(part,key1,key2)
> );
>
> I guess I'm not specifying a distribution key.  I assumed, perhaps incorrectly, that hash on the first element of the primary key would be the default.  I will try a run with an explicit DISTRIBUTE BY HASH(part).
>
> Chris...
> ________________________________________
> From: Koichi Suzuki [koi...@gm...]
> Sent: March 18, 2013 10:47 PM
> To: Chris Brown
> Cc: pos...@li...
> Subject: Re: [Postgres-xc-general] PG-XC Performance
>
> What kind of distribution did you use in your case?   If all the
> tables are replicated, all the inserts are associated with implicit
> 2PCs.   Frequently-updated tables should be distributed (hash will be
> better if the table has a suitable column).
>
> Regards;
> ----------
> Koichi Suzuki
>
>
> 2013/3/19 Chris Brown <cb...@in...>:
>> Hi all,
>>
>> I've been doing some performance testing of PostgreSQL-XC.  My workload is
>> all insert in this case.  I was expecting PG-XC to perform worse than the
>> alternatives but I was shocked by how much worse it was than PostgreSQL.
>> I'm well aware that NDB has durability and features issues so I expected
>> it to perform better - I don't consider the 0.8 to be particularly
>> interesting.  I'd like anyone's thoughts on this.
>>
>> Chris...
>>
>>
>> Results:
>>
>> All results are very approximate!
>> CPU is an estimate on the number of cores used by server processes
>> (ndbmtd, ndb_mgmd, postgresql, gtm, gtm_proxy).
>> TC  is a comparison of (Time*CPU) using PostgreSQL as the baseline and is
>> intended to represent a comparative CPU cost for a single insert
>> transaction.
>>
>> Client   PostgreSQL               PostgreSQL-XC             MySQL Cluster
>>           PostgreSQL-XC-2
>> Threads  Time   CPU        TC     Time   CPU         TC     Time   CPU
>>    TC     Time   CPU         TC
>> 16       5m08s  4.3 cores  1.0
>> 24       4m56s  4.6 cores  1.0    6m10s  10.7 cores  2.9
>> 32       4m48s  4.7 cores  1.0    5m26s  11.9 cores  2.9    2m27s  7.8
>> cores  0.8
>> 48       4m49s  4.7 cores  1.0    5m01s  12.6 cores  2.8    2m16s  8.0
>> cores  0.8    6m06s  10.2 cores  2.7
>> 64                                5m01s  13.1 cores  2.9    2m11s  8.3
>> cores  0.8    5m50s  10.5 cores  2.7
>> 96                                                          2m13s  8.4
>> cores  0.8
>>
>>
>> Configurations:
>>
>> PostgreSQL:
>> Datanodes        q1
>> Version: 9.2.x (redhat)
>>
>> PostgreSQL-XC:
>> Datanodes        q1-q2 (striped)
>> Coordinators     q1-q2
>> GTM      q3
>> Version 9.1.7 (tar)
>>
>> PostgreSQL-XC-2:
>> Datanodes        q1-q2 (striped)
>> Coordinators     q1-q2
>> GTM Proxy        q1-q2
>> GTM      q3
>> Version 9.1.7 (tar)
>>
>> MySQL Cluster:
>> ndbmtd   q1-q2 (striped)
>> mysqld   q1-q2
>> ndb_mgmd         q1
>> Version 7.1.17 (oracle rhel 5)
>>
>> q1, q2, q3 are identical boxes each with:
>>   24g ram, 1 cpu (6 cores), middle-tier raid with battery backed ram and 3
>> platters in raid 5.
>> They are lan local with a vlan-aware switch (I don't know which one and I
>> didn't check to see if it was saturated).
>>
>>
>> Tables & Data:
>>
>> 161 tables with primary key ("part" int not null, "key1" bigint not null,
>> "key2" bigint not null).  Each table has a selection of varchar columns
>> typically less than varchar(80).  All non-primary-key columns have
>> non-unique indexes.  5.8 million rows were imported with a different
>> number of rows to each table.
>>
>>
>> ------------------------------------------------------------------------------
>> Everyone hates slow websites. So do we.
>> Make your web apps faster with AppDynamics
>> Download AppDynamics Lite for free today:
>> http://p.sf.net/sfu/appdyn_d2d_mar
>> _______________________________________________
>> Postgres-xc-general mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_mar
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

Here's a sample table:

CREATE TABLE table_11 (
  "part" int not null,
  "key1" bigint not null,
  "key2" bigint not null,
  "col_1" varchar(23),
  "col_2" varchar(5),
  "col_3" varchar(32),
  "col_4" varchar(5),
  PRIMARY KEY(part,key1,key2)
);

I guess I'm not specifying a distribution key.  I assumed, perhaps incorrectly, that hash on the first element of the primary key would be the default.  I will try a run with an explicit DISTRIBUTE BY HASH(part).

Chris...
________________________________________
From: Koichi Suzuki [koi...@gm...]
Sent: March 18, 2013 10:47 PM
To: Chris Brown
Cc: pos...@li...
Subject: Re: [Postgres-xc-general] PG-XC Performance

What kind of distribution did you use in your case?   If all the
tables are replicated, all the inserts are associated with implicit
2PCs.   Frequently-updated tables should be distributed (hash will be
better if the table has a suitable column).

Regards;
----------
Koichi Suzuki


2013/3/19 Chris Brown <cb...@in...>:
> Hi all,
>
> I've been doing some performance testing of PostgreSQL-XC.  My workload is
> all insert in this case.  I was expecting PG-XC to perform worse than the
> alternatives but I was shocked by how much worse it was than PostgreSQL.
> I'm well aware that NDB has durability and features issues so I expected
> it to perform better - I don't consider the 0.8 to be particularly
> interesting.  I'd like anyone's thoughts on this.
>
> Chris...
>
>
> Results:
>
> All results are very approximate!
> CPU is an estimate on the number of cores used by server processes
> (ndbmtd, ndb_mgmd, postgresql, gtm, gtm_proxy).
> TC  is a comparison of (Time*CPU) using PostgreSQL as the baseline and is
> intended to represent a comparative CPU cost for a single insert
> transaction.
>
> Client   PostgreSQL               PostgreSQL-XC             MySQL Cluster
>           PostgreSQL-XC-2
> Threads  Time   CPU        TC     Time   CPU         TC     Time   CPU
>    TC     Time   CPU         TC
> 16       5m08s  4.3 cores  1.0
> 24       4m56s  4.6 cores  1.0    6m10s  10.7 cores  2.9
> 32       4m48s  4.7 cores  1.0    5m26s  11.9 cores  2.9    2m27s  7.8
> cores  0.8
> 48       4m49s  4.7 cores  1.0    5m01s  12.6 cores  2.8    2m16s  8.0
> cores  0.8    6m06s  10.2 cores  2.7
> 64                                5m01s  13.1 cores  2.9    2m11s  8.3
> cores  0.8    5m50s  10.5 cores  2.7
> 96                                                          2m13s  8.4
> cores  0.8
>
>
> Configurations:
>
> PostgreSQL:
> Datanodes        q1
> Version: 9.2.x (redhat)
>
> PostgreSQL-XC:
> Datanodes        q1-q2 (striped)
> Coordinators     q1-q2
> GTM      q3
> Version 9.1.7 (tar)
>
> PostgreSQL-XC-2:
> Datanodes        q1-q2 (striped)
> Coordinators     q1-q2
> GTM Proxy        q1-q2
> GTM      q3
> Version 9.1.7 (tar)
>
> MySQL Cluster:
> ndbmtd   q1-q2 (striped)
> mysqld   q1-q2
> ndb_mgmd         q1
> Version 7.1.17 (oracle rhel 5)
>
> q1, q2, q3 are identical boxes each with:
>   24g ram, 1 cpu (6 cores), middle-tier raid with battery backed ram and 3
> platters in raid 5.
> They are lan local with a vlan-aware switch (I don't know which one and I
> didn't check to see if it was saturated).
>
>
> Tables & Data:
>
> 161 tables with primary key ("part" int not null, "key1" bigint not null,
> "key2" bigint not null).  Each table has a selection of varchar columns
> typically less than varchar(80).  All non-primary-key columns have
> non-unique indexes.  5.8 million rows were imported with a different
> number of rows to each table.
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_mar
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

What kind of distribution did you use in your case?   If all the
tables are replicated, all the inserts are associated with implicit
2PCs.   Frequently-updated tables should be distributed (hash will be
better if the table has a suitable column).

Regards;
----------
Koichi Suzuki


2013/3/19 Chris Brown <cb...@in...>:
> Hi all,
>
> I've been doing some performance testing of PostgreSQL-XC.  My workload is
> all insert in this case.  I was expecting PG-XC to perform worse than the
> alternatives but I was shocked by how much worse it was than PostgreSQL.
> I'm well aware that NDB has durability and features issues so I expected
> it to perform better - I don't consider the 0.8 to be particularly
> interesting.  I'd like anyone's thoughts on this.
>
> Chris...
>
>
> Results:
>
> All results are very approximate!
> CPU is an estimate on the number of cores used by server processes
> (ndbmtd, ndb_mgmd, postgresql, gtm, gtm_proxy).
> TC  is a comparison of (Time*CPU) using PostgreSQL as the baseline and is
> intended to represent a comparative CPU cost for a single insert
> transaction.
>
> Client   PostgreSQL               PostgreSQL-XC             MySQL Cluster
>           PostgreSQL-XC-2
> Threads  Time   CPU        TC     Time   CPU         TC     Time   CPU
>    TC     Time   CPU         TC
> 16       5m08s  4.3 cores  1.0
> 24       4m56s  4.6 cores  1.0    6m10s  10.7 cores  2.9
> 32       4m48s  4.7 cores  1.0    5m26s  11.9 cores  2.9    2m27s  7.8
> cores  0.8
> 48       4m49s  4.7 cores  1.0    5m01s  12.6 cores  2.8    2m16s  8.0
> cores  0.8    6m06s  10.2 cores  2.7
> 64                                5m01s  13.1 cores  2.9    2m11s  8.3
> cores  0.8    5m50s  10.5 cores  2.7
> 96                                                          2m13s  8.4
> cores  0.8
>
>
> Configurations:
>
> PostgreSQL:
> Datanodes        q1
> Version: 9.2.x (redhat)
>
> PostgreSQL-XC:
> Datanodes        q1-q2 (striped)
> Coordinators     q1-q2
> GTM      q3
> Version 9.1.7 (tar)
>
> PostgreSQL-XC-2:
> Datanodes        q1-q2 (striped)
> Coordinators     q1-q2
> GTM Proxy        q1-q2
> GTM      q3
> Version 9.1.7 (tar)
>
> MySQL Cluster:
> ndbmtd   q1-q2 (striped)
> mysqld   q1-q2
> ndb_mgmd         q1
> Version 7.1.17 (oracle rhel 5)
>
> q1, q2, q3 are identical boxes each with:
>   24g ram, 1 cpu (6 cores), middle-tier raid with battery backed ram and 3
> platters in raid 5.
> They are lan local with a vlan-aware switch (I don't know which one and I
> didn't check to see if it was saturated).
>
>
> Tables & Data:
>
> 161 tables with primary key ("part" int not null, "key1" bigint not null,
> "key2" bigint not null).  Each table has a selection of varchar columns
> typically less than varchar(80).  All non-primary-key columns have
> non-unique indexes.  5.8 million rows were imported with a different
> number of rows to each table.
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_mar
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

In this case, even though each node can access to GTM, update
transaction fails. If a transaction is involved by an unreachable
node, such local update fails, and then global update fails.  This
maintains cluster-wide database consistency.   Naturally, if a
transaction is involved only by reachable nodes, such transaction
succeeds and maintains cluster-wide consistency too.

Serious split bran case is: if you run two different GTM at the same
time and try to connect different coordinators to different GTMs.   As
in the case of almost other case, as Nikhil pointed out, this should
be resolved by some external tools.
----------
Koichi Suzuki


2013/3/19 Nikhil Sontakke <ni...@st...>:
> Hi Chris,
>
> Split brain is not solved internally by Postgres-XC. It's the job of
> an external layer to ensure that such scenarios do not occur.
>
> For example, one can use Corosync/Pacemaker to monitor XC nodes and
> then use proper mechanisms to ensure failover and STONITH steps. CS/PM
> solve the tricky problem of node membership and consistent cluster
> view pretty well.
>
> Regards,
> Nikhils
>
> On Tue, Mar 19, 2013 at 2:16 AM, Chris Brown <cb...@in...> wrote:
>> Hi all,
>>
>> I just read through:
>>
>> wiki.postgresql.org/images/4/44/Pgxc_HA_20121024.pdf
>>
>> and I wanted to know how PG-XC solves the problem of split brain.
>>
>> For example, let's say there is a pair:
>>   Box 1: datanode1-master, datanode2-slave, coordinator master, gtm master
>>   Box 2: datanode1-slave, datanode2-master, coordinator slave, gtm slave
>> and communication is severed between the two boxes.
>>
>> Another example:
>>   Box 1: datanode1-master, datanode2-slave, coordinator master, gtm proxy
>>   Box 2: datanode1-slave, datanode2-master, coordinator slave, gtm proxy
>>   Box 3: gtm (no slave exists for simplicity)
>> and communication is severed between box 1 and 2, but both can still
>> communicate with box 3.  (Or another network outage scenario - there are
>> obviously several.)
>>
>> Chris...
>>
>> ------------------------------------------------------------------------------
>> Everyone hates slow websites. So do we.
>> Make your web apps faster with AppDynamics
>> Download AppDynamics Lite for free today:
>> http://p.sf.net/sfu/appdyn_d2d_mar
>> _______________________________________________
>> Postgres-xc-general mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>
>
>
>
> --
> StormDB - http://www.stormdb.com
> The Database Cloud
> Postgres-XC Support and Service
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_mar
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

Hi Chris,

Split brain is not solved internally by Postgres-XC. It's the job of
an external layer to ensure that such scenarios do not occur.

For example, one can use Corosync/Pacemaker to monitor XC nodes and
then use proper mechanisms to ensure failover and STONITH steps. CS/PM
solve the tricky problem of node membership and consistent cluster
view pretty well.

Regards,
Nikhils

On Tue, Mar 19, 2013 at 2:16 AM, Chris Brown <cb...@in...> wrote:
> Hi all,
>
> I just read through:
>
> wiki.postgresql.org/images/4/44/Pgxc_HA_20121024.pdf
>
> and I wanted to know how PG-XC solves the problem of split brain.
>
> For example, let's say there is a pair:
>   Box 1: datanode1-master, datanode2-slave, coordinator master, gtm master
>   Box 2: datanode1-slave, datanode2-master, coordinator slave, gtm slave
> and communication is severed between the two boxes.
>
> Another example:
>   Box 1: datanode1-master, datanode2-slave, coordinator master, gtm proxy
>   Box 2: datanode1-slave, datanode2-master, coordinator slave, gtm proxy
>   Box 3: gtm (no slave exists for simplicity)
> and communication is severed between box 1 and 2, but both can still
> communicate with box 3.  (Or another network outage scenario - there are
> obviously several.)
>
> Chris...
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_mar
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>



-- 
StormDB - http://www.stormdb.com
The Database Cloud
Postgres-XC Support and Service

Hi all,

I've been doing some performance testing of PostgreSQL-XC.  My workload is
all insert in this case.  I was expecting PG-XC to perform worse than the
alternatives but I was shocked by how much worse it was than PostgreSQL.
I'm well aware that NDB has durability and features issues so I expected
it to perform better - I don't consider the 0.8 to be particularly
interesting.  I'd like anyone's thoughts on this.

Chris...


Results:

All results are very approximate!
CPU is an estimate on the number of cores used by server processes
(ndbmtd, ndb_mgmd, postgresql, gtm, gtm_proxy).
TC  is a comparison of (Time*CPU) using PostgreSQL as the baseline and is
intended to represent a comparative CPU cost for a single insert
transaction.

Client   PostgreSQL               PostgreSQL-XC             MySQL Cluster
          PostgreSQL-XC-2
Threads  Time   CPU        TC     Time   CPU         TC     Time   CPU
   TC     Time   CPU         TC
16       5m08s  4.3 cores  1.0
24       4m56s  4.6 cores  1.0    6m10s  10.7 cores  2.9
32       4m48s  4.7 cores  1.0    5m26s  11.9 cores  2.9    2m27s  7.8
cores  0.8
48       4m49s  4.7 cores  1.0    5m01s  12.6 cores  2.8    2m16s  8.0
cores  0.8    6m06s  10.2 cores  2.7
64                                5m01s  13.1 cores  2.9    2m11s  8.3
cores  0.8    5m50s  10.5 cores  2.7
96                                                          2m13s  8.4
cores  0.8


Configurations:

PostgreSQL:
Datanodes	 q1
Version: 9.2.x (redhat)

PostgreSQL-XC:
Datanodes	 q1-q2 (striped)
Coordinators	 q1-q2
GTM	 q3
Version 9.1.7 (tar)

PostgreSQL-XC-2:
Datanodes	 q1-q2 (striped)
Coordinators	 q1-q2
GTM Proxy	 q1-q2
GTM	 q3
Version 9.1.7 (tar)

MySQL Cluster:
ndbmtd	 q1-q2 (striped)
mysqld	 q1-q2
ndb_mgmd	 q1
Version 7.1.17 (oracle rhel 5)

q1, q2, q3 are identical boxes each with:
  24g ram, 1 cpu (6 cores), middle-tier raid with battery backed ram and 3
platters in raid 5.
They are lan local with a vlan-aware switch (I don't know which one and I
didn't check to see if it was saturated).


Tables & Data:

161 tables with primary key ("part" int not null, "key1" bigint not null,
"key2" bigint not null).  Each table has a selection of varchar columns
typically less than varchar(80).  All non-primary-key columns have
non-unique indexes.  5.8 million rows were imported with a different
number of rows to each table.