Menu
▾
▴
postgres-xc-general — General info and messages
You can subscribe to this list here.
| 2010 |
Jan
|
Feb
|
Mar
|
Apr
|
May
(2) |
Jun
|
Jul
|
Aug
(6) |
Sep
|
Oct
(19) |
Nov
(1) |
Dec
|
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2011 |
Jan
(12) |
Feb
(1) |
Mar
(4) |
Apr
(4) |
May
(32) |
Jun
(12) |
Jul
(11) |
Aug
(1) |
Sep
(6) |
Oct
(3) |
Nov
|
Dec
(10) |
| 2012 |
Jan
(11) |
Feb
(1) |
Mar
(3) |
Apr
(25) |
May
(53) |
Jun
(38) |
Jul
(103) |
Aug
(54) |
Sep
(31) |
Oct
(66) |
Nov
(77) |
Dec
(20) |
| 2013 |
Jan
(91) |
Feb
(86) |
Mar
(103) |
Apr
(107) |
May
(25) |
Jun
(37) |
Jul
(17) |
Aug
(59) |
Sep
(38) |
Oct
(78) |
Nov
(29) |
Dec
(15) |
| 2014 |
Jan
(23) |
Feb
(82) |
Mar
(118) |
Apr
(101) |
May
(103) |
Jun
(45) |
Jul
(6) |
Aug
(10) |
Sep
|
Oct
(32) |
Nov
|
Dec
(9) |
| 2015 |
Jan
(3) |
Feb
(5) |
Mar
|
Apr
(1) |
May
|
Jun
|
Jul
(9) |
Aug
(4) |
Sep
(3) |
Oct
|
Nov
|
Dec
|
| 2016 |
Jan
(3) |
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2017 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
(3) |
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2018 |
Jan
|
Feb
|
Mar
|
Apr
|
May
(4) |
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| S | M | T | W | T | F | S |
|---|---|---|---|---|---|---|
|
|
1
(2) |
2
(2) |
3
(4) |
4
(5) |
5
(17) |
6
(4) |
|
7
(7) |
8
(12) |
9
(1) |
10
(1) |
11
(6) |
12
(7) |
13
|
|
14
(1) |
15
(3) |
16
(2) |
17
(1) |
18
(2) |
19
(8) |
20
|
|
21
(4) |
22
(5) |
23
(3) |
24
|
25
(1) |
26
(3) |
27
(2) |
|
28
|
29
(1) |
30
(3) |
|
|
|
|
|
From: Rene R. B. <ren...@gm...> - 2013-04-19 04:22:31
|
Thanks for your kind replies. The benchmark was the standard pgbench benchmark (select, insert, delete, update). I'll look into the table distribution issue. I'm not complaining at all, I really like your work, I just wan't to understand what's going on, an someday maybe help you make tests. Good night. 2013/4/18 Ashutosh Bapat <ash...@en...> > > > > On Fri, Apr 19, 2013 at 9:38 AM, 鈴木 幸市 <ko...@in...> wrote: > >> Unfortunately, read-only benchmark does not scale well. Write-intensive >> benchmark will scale better. >> > > Not necessarily Suzuki-san. It all boils down to how the tables are > distributed. For read-only benchmarks, one of the options is to have all > the data replicated and use multiple coordinators with different preferred > node. > > >> --- >> Koichi Suzuki >> >> >> >> On 2013/04/19, at 12:49, Ashutosh Bapat <ash...@en...> >> wrote: >> >> Hi Rene, >> Please make sure that the distributions of the tables involved in the >> benchmark are as per guidelines. You will find some material on this topic >> at http://wiki.postgresql.org/images/f/f6/PGXC_Scalability_PGOpen2012.pdf<http://wiki.postgresql.org/images/f/f6/PGXC_Scalability_PGOpen2012..pdf> >> . >> >> >> On Fri, Apr 19, 2013 at 4:37 AM, Rene Romero Benavides < >> ren...@gm...> wrote: >> >>> Hello. using version 1.0.2 >>> I followed the short version of the installation process and ran the >>> following benchmark: >>> >>> pgbench -T 60 -c 4 -j 4 benchdb >>> starting vacuum...end. >>> transaction type: TPC-B (sort of) >>> scaling factor: 4 >>> query mode: simple >>> number of clients: 4 >>> number of threads: 4 >>> duration: 60 s >>> number of transactions actually processed: 1631 >>> tps = 27.108444 (including connections establishing) >>> tps = 27.111498 (excluding connections establishing) >>> >>> This is the related sar and iotop info: >>> http://img32.imageshack.us/img32/3990/highio.png >>> >>> Same test on a single server (no postgres-xc) with the same >>> configuration: >>> >>> pgbench -T 60 -c 4 -j 4 -p 5433 pgbench >>> starting vacuum...end. >>> transaction type: TPC-B (sort of) >>> scaling factor: 4 >>> query mode: simple >>> number of clients: 4 >>> number of threads: 4 >>> duration: 60 s >>> number of transactions actually processed: 6179 >>> tps = 102.775810 (including connections establishing) >>> tps = 102.788523 (excluding connections establishing) >>> >>> sar and iotop output: >>> >>> http://img823.imageshack.us/img823/8775/highio11.png >>> >>> Is this high IO activity expected? I know that prepare transaction / >>> commit prepared are expensive operations but this is too much IMO. >>> >>> Both cases using default server configuration. >>> >>> Is there any tuning that'd mitigate this IO bottleneck? >>> >>> -- >>> El genio es 1% inspiración y 99% transpiración. >>> Thomas Alva Edison >>> http://pglearn.blogspot.mx/ >>> >>> >>> >>> ------------------------------------------------------------------------------ >>> Precog is a next-generation analytics platform capable of advanced >>> analytics on semi-structured data. The platform includes APIs for >>> building >>> apps and a phenomenal toolset for data science. Developers can use >>> our toolset for easy data analysis & visualization. Get a free account! >>> http://www2.precog.com/precogplatform/slashdotnewsletter >>> _______________________________________________ >>> Postgres-xc-general mailing list >>> Pos...@li... >>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general >>> >>> >> >> >> -- >> Best Wishes, >> Ashutosh Bapat >> EntepriseDB Corporation >> The Enterprise Postgres Company >> ------------------------------------------------------------------------------ >> Precog is a next-generation analytics platform capable of advanced >> analytics on semi-structured data. The platform includes APIs for building >> apps and a phenomenal toolset for data science. Developers can use >> our toolset for easy data analysis & visualization. Get a free account! >> >> http://www2.precog.com/precogplatform/slashdotnewsletter_______________________________________________ >> Postgres-xc-general mailing list >> Pos...@li... >> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general >> >> >> > > > -- > Best Wishes, > Ashutosh Bapat > EntepriseDB Corporation > The Enterprise Postgres Company > -- El genio es 1% inspiración y 99% transpiración. Thomas Alva Edison http://pglearn.blogspot.mx/ |
|
From: Ashutosh B. <ash...@en...> - 2013-04-19 04:12:47
|
On Fri, Apr 19, 2013 at 9:38 AM, 鈴木 幸市 <ko...@in...> wrote: > Unfortunately, read-only benchmark does not scale well. Write-intensive > benchmark will scale better. > Not necessarily Suzuki-san. It all boils down to how the tables are distributed. For read-only benchmarks, one of the options is to have all the data replicated and use multiple coordinators with different preferred node. > --- > Koichi Suzuki > > > > On 2013/04/19, at 12:49, Ashutosh Bapat <ash...@en...> > wrote: > > Hi Rene, > Please make sure that the distributions of the tables involved in the > benchmark are as per guidelines. You will find some material on this topic > at http://wiki.postgresql.org/images/f/f6/PGXC_Scalability_PGOpen2012.pdf<http://wiki.postgresql.org/images/f/f6/PGXC_Scalability_PGOpen2012..pdf> > . > > > On Fri, Apr 19, 2013 at 4:37 AM, Rene Romero Benavides < > ren...@gm...> wrote: > >> Hello. using version 1.0.2 >> I followed the short version of the installation process and ran the >> following benchmark: >> >> pgbench -T 60 -c 4 -j 4 benchdb >> starting vacuum...end. >> transaction type: TPC-B (sort of) >> scaling factor: 4 >> query mode: simple >> number of clients: 4 >> number of threads: 4 >> duration: 60 s >> number of transactions actually processed: 1631 >> tps = 27.108444 (including connections establishing) >> tps = 27.111498 (excluding connections establishing) >> >> This is the related sar and iotop info: >> http://img32.imageshack.us/img32/3990/highio.png >> >> Same test on a single server (no postgres-xc) with the same configuration: >> >> pgbench -T 60 -c 4 -j 4 -p 5433 pgbench >> starting vacuum...end. >> transaction type: TPC-B (sort of) >> scaling factor: 4 >> query mode: simple >> number of clients: 4 >> number of threads: 4 >> duration: 60 s >> number of transactions actually processed: 6179 >> tps = 102.775810 (including connections establishing) >> tps = 102.788523 (excluding connections establishing) >> >> sar and iotop output: >> >> http://img823.imageshack.us/img823/8775/highio11.png >> >> Is this high IO activity expected? I know that prepare transaction / >> commit prepared are expensive operations but this is too much IMO. >> >> Both cases using default server configuration. >> >> Is there any tuning that'd mitigate this IO bottleneck? >> >> -- >> El genio es 1% inspiración y 99% transpiración. >> Thomas Alva Edison >> http://pglearn.blogspot.mx/ >> >> >> >> ------------------------------------------------------------------------------ >> Precog is a next-generation analytics platform capable of advanced >> analytics on semi-structured data. The platform includes APIs for building >> apps and a phenomenal toolset for data science. Developers can use >> our toolset for easy data analysis & visualization. Get a free account! >> http://www2.precog.com/precogplatform/slashdotnewsletter >> _______________________________________________ >> Postgres-xc-general mailing list >> Pos...@li... >> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general >> >> > > > -- > Best Wishes, > Ashutosh Bapat > EntepriseDB Corporation > The Enterprise Postgres Company > ------------------------------------------------------------------------------ > Precog is a next-generation analytics platform capable of advanced > analytics on semi-structured data. The platform includes APIs for building > apps and a phenomenal toolset for data science. Developers can use > our toolset for easy data analysis & visualization. Get a free account! > > http://www2.precog.com/precogplatform/slashdotnewsletter_______________________________________________ > Postgres-xc-general mailing list > Pos...@li... > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general > > > -- Best Wishes, Ashutosh Bapat EntepriseDB Corporation The Enterprise Postgres Company |
|
From: 鈴木 幸市 <ko...@in...> - 2013-04-19 04:08:23
|
Unfortunately, read-only benchmark does not scale well. Write-intensive benchmark will scale better. --- Koichi Suzuki On 2013/04/19, at 12:49, Ashutosh Bapat <ash...@en...> wrote: > Hi Rene, > Please make sure that the distributions of the tables involved in the benchmark are as per guidelines. You will find some material on this topic at http://wiki.postgresql.org/images/f/f6/PGXC_Scalability_PGOpen2012.pdf. > > > On Fri, Apr 19, 2013 at 4:37 AM, Rene Romero Benavides <ren...@gm...> wrote: > Hello. using version 1.0.2 > I followed the short version of the installation process and ran the following benchmark: > > pgbench -T 60 -c 4 -j 4 benchdb > starting vacuum...end. > transaction type: TPC-B (sort of) > scaling factor: 4 > query mode: simple > number of clients: 4 > number of threads: 4 > duration: 60 s > number of transactions actually processed: 1631 > tps = 27.108444 (including connections establishing) > tps = 27.111498 (excluding connections establishing) > > This is the related sar and iotop info: > http://img32.imageshack.us/img32/3990/highio.png > > Same test on a single server (no postgres-xc) with the same configuration: > > pgbench -T 60 -c 4 -j 4 -p 5433 pgbench > starting vacuum...end. > transaction type: TPC-B (sort of) > scaling factor: 4 > query mode: simple > number of clients: 4 > number of threads: 4 > duration: 60 s > number of transactions actually processed: 6179 > tps = 102.775810 (including connections establishing) > tps = 102.788523 (excluding connections establishing) > > sar and iotop output: > > http://img823.imageshack.us/img823/8775/highio11.png > > Is this high IO activity expected? I know that prepare transaction / commit prepared are expensive operations but this is too much IMO. > > Both cases using default server configuration. > > Is there any tuning that'd mitigate this IO bottleneck? > > -- > El genio es 1% inspiración y 99% transpiración. > Thomas Alva Edison > http://pglearn.blogspot.mx/ > > > ------------------------------------------------------------------------------ > Precog is a next-generation analytics platform capable of advanced > analytics on semi-structured data. The platform includes APIs for building > apps and a phenomenal toolset for data science. Developers can use > our toolset for easy data analysis & visualization. Get a free account! > http://www2.precog.com/precogplatform/slashdotnewsletter > _______________________________________________ > Postgres-xc-general mailing list > Pos...@li... > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general > > > > > -- > Best Wishes, > Ashutosh Bapat > EntepriseDB Corporation > The Enterprise Postgres Company > ------------------------------------------------------------------------------ > Precog is a next-generation analytics platform capable of advanced > analytics on semi-structured data. The platform includes APIs for building > apps and a phenomenal toolset for data science. Developers can use > our toolset for easy data analysis & visualization. Get a free account! > http://www2.precog.com/precogplatform/slashdotnewsletter_______________________________________________ > Postgres-xc-general mailing list > Pos...@li... > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general |
|
From: Ashutosh B. <ash...@en...> - 2013-04-19 03:50:05
|
Hi Rene, Please make sure that the distributions of the tables involved in the benchmark are as per guidelines. You will find some material on this topic at http://wiki.postgresql.org/images/f/f6/PGXC_Scalability_PGOpen2012.pdf. On Fri, Apr 19, 2013 at 4:37 AM, Rene Romero Benavides < ren...@gm...> wrote: > Hello. using version 1.0.2 > I followed the short version of the installation process and ran the > following benchmark: > > pgbench -T 60 -c 4 -j 4 benchdb > starting vacuum...end. > transaction type: TPC-B (sort of) > scaling factor: 4 > query mode: simple > number of clients: 4 > number of threads: 4 > duration: 60 s > number of transactions actually processed: 1631 > tps = 27.108444 (including connections establishing) > tps = 27.111498 (excluding connections establishing) > > This is the related sar and iotop info: > http://img32.imageshack.us/img32/3990/highio.png > > Same test on a single server (no postgres-xc) with the same configuration: > > pgbench -T 60 -c 4 -j 4 -p 5433 pgbench > starting vacuum...end. > transaction type: TPC-B (sort of) > scaling factor: 4 > query mode: simple > number of clients: 4 > number of threads: 4 > duration: 60 s > number of transactions actually processed: 6179 > tps = 102.775810 (including connections establishing) > tps = 102.788523 (excluding connections establishing) > > sar and iotop output: > > http://img823.imageshack.us/img823/8775/highio11.png > > Is this high IO activity expected? I know that prepare transaction / > commit prepared are expensive operations but this is too much IMO. > > Both cases using default server configuration. > > Is there any tuning that'd mitigate this IO bottleneck? > > -- > El genio es 1% inspiración y 99% transpiración. > Thomas Alva Edison > http://pglearn.blogspot.mx/ > > > > ------------------------------------------------------------------------------ > Precog is a next-generation analytics platform capable of advanced > analytics on semi-structured data. The platform includes APIs for building > apps and a phenomenal toolset for data science. Developers can use > our toolset for easy data analysis & visualization. Get a free account! > http://www2.precog.com/precogplatform/slashdotnewsletter > _______________________________________________ > Postgres-xc-general mailing list > Pos...@li... > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general > > -- Best Wishes, Ashutosh Bapat EntepriseDB Corporation The Enterprise Postgres Company |
|
From: Ashutosh B. <ash...@en...> - 2013-04-19 03:46:54
|
Hi Cronje, In the current release and upcoming release, we haven't address utilising replicated tables as stand-bys for each other. Theoretically, and also architecture-wise, yes, if someone has replicated the tables, the corresponding nodes should be able to provide the required data in case, one of them fails. But that's not implemented yet. On Thu, Apr 18, 2013 at 2:29 PM, Cronje Fourie <cr...@ab...> wrote: > Morning all…**** > > ** ** > > I’ve been reading a fair bit about pgsql-xc and its shared nothing > architecture as well as some of its HA setups.**** > > ** ** > > For my part I’m in the process of researching an RDBMS that we can use for > a HA system. Based on my initial reading of the docs etc it looked like > pgsql-xc is a perfect fit for our situation since we can configure 2 or > more systems and have data replicated across all the nodes.**** > > ** ** > > My understanding is that this should provide us with a shared nothing > solution in which any node can go down at any time without affecting the > solution.**** > > ** ** > > But the docs I’ve seen always refer to a HA setups using master slave / > synching with HA tools used to fail over between the master and slave etc… > **** > > ** ** > > Are these configurations required only in the case of distributed tables? > Or does this remain a requirement for replicated tables?**** > > ** ** > > In short my desired configuration would be**** > > ** ** > > SERVERA – GTM, COORDA, DATANODEA**** > > SERVERB – GTM-STANDBY, COORDB, DATANODEB**** > > ** ** > > LB1 – Load balancing between SERVERA, SERVERB**** > > ** ** > > All tables are replicated across all nodes.**** > > In theory we should then be able to add**** > > SERVERC – GTM-STANDBY, COORDC, DATANODEC**** > > ** ** > > Data is replicated from SERVERA or SERVERB to SERVERC and once ready we > can activate on LB1.**** > > ** ** > > Also if SERVERA or SERVERB goes down, the GTM-STANDBY config will handle > that failover while the LB1 then just needs to feed all requests to the > remaining COORD.**** > > ** ** > > This should provide us with a theoretical 100% uptime without the need for > master / slave hot standby replication.**** > > ** ** > > Yet I don’t see this in any examples? Why? Is it so simple to do that no > one has bothered setting up such an example? Or is it in fact not possible > with pgsql-xc?**** > > ** ** > > Thanks in advance > Cronje**** > > > ------------------------------------------------------------------------------ > Precog is a next-generation analytics platform capable of advanced > analytics on semi-structured data. The platform includes APIs for building > apps and a phenomenal toolset for data science. Developers can use > our toolset for easy data analysis & visualization. Get a free account! > http://www2.precog.com/precogplatform/slashdotnewsletter > _______________________________________________ > Postgres-xc-general mailing list > Pos...@li... > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general > > -- Best Wishes, Ashutosh Bapat EntepriseDB Corporation The Enterprise Postgres Company |
|
From: Rene R. B. <ren...@gm...> - 2013-04-18 23:07:42
|
Hello. using version 1.0.2 I followed the short version of the installation process and ran the following benchmark: pgbench -T 60 -c 4 -j 4 benchdb starting vacuum...end. transaction type: TPC-B (sort of) scaling factor: 4 query mode: simple number of clients: 4 number of threads: 4 duration: 60 s number of transactions actually processed: 1631 tps = 27.108444 (including connections establishing) tps = 27.111498 (excluding connections establishing) This is the related sar and iotop info: http://img32.imageshack.us/img32/3990/highio.png Same test on a single server (no postgres-xc) with the same configuration: pgbench -T 60 -c 4 -j 4 -p 5433 pgbench starting vacuum...end. transaction type: TPC-B (sort of) scaling factor: 4 query mode: simple number of clients: 4 number of threads: 4 duration: 60 s number of transactions actually processed: 6179 tps = 102.775810 (including connections establishing) tps = 102.788523 (excluding connections establishing) sar and iotop output: http://img823.imageshack.us/img823/8775/highio11.png Is this high IO activity expected? I know that prepare transaction / commit prepared are expensive operations but this is too much IMO. Both cases using default server configuration. Is there any tuning that'd mitigate this IO bottleneck? -- El genio es 1% inspiración y 99% transpiración. Thomas Alva Edison http://pglearn.blogspot.mx/ |
|
From: Cronje F. <cr...@ab...> - 2013-04-18 09:28:35
|
Morning all. I've been reading a fair bit about pgsql-xc and its shared nothing architecture as well as some of its HA setups. For my part I'm in the process of researching an RDBMS that we can use for a HA system. Based on my initial reading of the docs etc it looked like pgsql-xc is a perfect fit for our situation since we can configure 2 or more systems and have data replicated across all the nodes. My understanding is that this should provide us with a shared nothing solution in which any node can go down at any time without affecting the solution. But the docs I've seen always refer to a HA setups using master slave / synching with HA tools used to fail over between the master and slave etc. Are these configurations required only in the case of distributed tables? Or does this remain a requirement for replicated tables? In short my desired configuration would be SERVERA - GTM, COORDA, DATANODEA SERVERB - GTM-STANDBY, COORDB, DATANODEB LB1 - Load balancing between SERVERA, SERVERB All tables are replicated across all nodes. In theory we should then be able to add SERVERC - GTM-STANDBY, COORDC, DATANODEC Data is replicated from SERVERA or SERVERB to SERVERC and once ready we can activate on LB1. Also if SERVERA or SERVERB goes down, the GTM-STANDBY config will handle that failover while the LB1 then just needs to feed all requests to the remaining COORD. This should provide us with a theoretical 100% uptime without the need for master / slave hot standby replication. Yet I don't see this in any examples? Why? Is it so simple to do that no one has bothered setting up such an example? Or is it in fact not possible with pgsql-xc? Thanks in advance Cronje |
|
From: 鈴木 幸市 <ko...@in...> - 2013-04-17 01:40:29
|
Thanks a lot. I've committed the fix. Best; --- Koichi Suzuki On 2013/04/16, at 17:32, 鈴木 幸市 <ko...@in...> wrote: > Ouch! This bash (not easy to debug) will be replaced with C version in 1.1 or later. > > Anyway, I will commit this change to github. > > Thanks. > --- > Koichi Suzuki > > > > On 2013/04/15, at 22:48, Theodotos Andreou <th...@ub...> wrote: > >> I found another one on line 4566: >> >> if [ "$1" == "gtm" ] || [ "$1" == "$gtmName"]; then >> >> t needs a space before ']' >> >> if [ "$1" =="gtm" ] || ["$1" =="$gtmName" ]; then >> >> >> >> On 04/15/2013 04:09 AM, 鈴木 幸市 wrote: >>> Oh, it's a bug. log_eco should be log_echo, which is pgxc_ctl internal function. >>> >>> Verry sorry for the mistake. I committed the fix to git repository. >>> >>> Please checkin the latest one. >>> >>> Thank you very much; >>> --- >>> Koichi Suzuki >>> >>> >>> On 2013/04/13, at 8:06, Theodotos Andreou <th...@ub...> wrote: >>> >>>> Hello to all, >>>> >>>> When I run the following command I get >>>> $ pgxc_ctl start all >>>> /var/lib/postgres-xc/bin/pgxc_ctl: line 4003: log_eco: command not found >>>> >>>> Is this an actual command or a function inside pgxc_ctl? >>>> >>>> ------------------------------------------------------------------------------ >>>> Precog is a next-generation analytics platform capable of advanced >>>> analytics on semi-structured data. The platform includes APIs for building >>>> apps and a phenomenal toolset for data science. Developers can use >>>> our toolset for easy data analysis & visualization. Get a free account! >>>> http://www2.precog.com/precogplatform/slashdotnewsletter >>>> _______________________________________________ >>>> Postgres-xc-general mailing list >>>> Pos...@li... >>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general >>>> >> >> > > > ------------------------------------------------------------------------------ > Precog is a next-generation analytics platform capable of advanced > analytics on semi-structured data. The platform includes APIs for building > apps and a phenomenal toolset for data science. Developers can use > our toolset for easy data analysis & visualization. Get a free account! > http://www2.precog.com/precogplatform/slashdotnewsletter > _______________________________________________ > Postgres-xc-general mailing list > Pos...@li... > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general > |
|
From: 鈴木 幸市 <ko...@in...> - 2013-04-16 08:32:48
|
Ouch! This bash (not easy to debug) will be replaced with C version in 1.1 or later. Anyway, I will commit this change to github. Thanks. --- Koichi Suzuki On 2013/04/15, at 22:48, Theodotos Andreou <th...@ub...> wrote: > I found another one on line 4566: > > if [ "$1" == "gtm" ] || [ "$1" == "$gtmName"]; then > > t needs a space before ']' > > if [ "$1" =="gtm" ] || ["$1" =="$gtmName" ]; then > > > > On 04/15/2013 04:09 AM, 鈴木 幸市 wrote: >> Oh, it's a bug. log_eco should be log_echo, which is pgxc_ctl internal function. >> >> Verry sorry for the mistake. I committed the fix to git repository. >> >> Please checkin the latest one. >> >> Thank you very much; >> --- >> Koichi Suzuki >> >> >> On 2013/04/13, at 8:06, Theodotos Andreou <th...@ub...> wrote: >> >>> Hello to all, >>> >>> When I run the following command I get >>> $ pgxc_ctl start all >>> /var/lib/postgres-xc/bin/pgxc_ctl: line 4003: log_eco: command not found >>> >>> Is this an actual command or a function inside pgxc_ctl? >>> >>> ------------------------------------------------------------------------------ >>> Precog is a next-generation analytics platform capable of advanced >>> analytics on semi-structured data. The platform includes APIs for building >>> apps and a phenomenal toolset for data science. Developers can use >>> our toolset for easy data analysis & visualization. Get a free account! >>> http://www2.precog.com/precogplatform/slashdotnewsletter >>> _______________________________________________ >>> Postgres-xc-general mailing list >>> Pos...@li... >>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general >>> > > |
|
From: 鈴木 幸市 <ko...@in...> - 2013-04-16 08:31:50
|
If you don't have a PRIMARY NODE, be careful about updating replicated tables. You have to control the order of each update to each node exactly the same to make things consistent. Best; --- Koichi Suzuki On 2013/04/16, at 0:57, Paul Jones <pb...@cm...> wrote: > I will send this when we are able to reconfigure our cluster. RIght now, we don't have PRIMARY enabled. > > PJ > > > >> ________________________________ >> From: Koichi Suzuki <ko...@in...> >> To: Paul Jones <pb...@cm...> >> Cc: "pos...@li..." <pos...@li...> >> Sent: Thursday, April 11, 2013 10:31 PM >> Subject: Re: [Postgres-xc-general] Questions about PRIMARY and a problem >> >> >> I did the same thing with 4CO and 4DN, one of them as primary node, and found all the datanode contains identical set or rows. >> >> Could I have pgxc_node in each coordinator? >> >> Regards; >> --- >> Koichi Suzuki >> >> On Mon, 8 Apr 2013 12:58:52 -0700 (PDT) >> Paul Jones <pb...@cm...> wrote: >> >>> Thanks for everyone's explanation of PRIMARY. It is much clearer now. >>> >>> I believe, then, that we may have uncovered a bug in PRIMARY. >>> >>> We created a new cluster (8 nodes, 3 coordinators), but with only one PRIMARY datanode. The PRIMARY was >>> declared the same on all 3 coordinators. >>> >>> A table that was declared DISTRIBUTE BY REPLICATION and loaded by \copy did not have any rows >>> present on the PRIMARY! Further, other tables with FK's referring to this table had RI failures when they were loaded, >>> even though there were complete copies of the table in all the other non-PRIMARY datanodes. >>> >>> When we remade the cluster without any PRIMARY, this table loaded into all datanodes and there were no RI failures. >>> >>> Is this a bug? Unfortunately I won't be able to experiment with this until we finish executing our test plan, perhaps >>> a few days. >>> >>> PJ >>> >>> >>> >>> >>>> ________________________________ >>>> From: Koichi Suzuki <koi...@gm...> >>>> To: Andrei Martsinchyk <and...@gm...> >>>> Cc: "pos...@li..." <pos...@li...> >>>> Sent: Sunday, April 7, 2013 9:23 AM >>>> Subject: Re: [Postgres-xc-general] Questions about PRIMARY and a problem >>>> >>>> >>>> Primary node is useful to maintain replicated table in a consistent status in all the datanode. All the writes to a replicated table goes first to the primary node so all the conflicts are resolved here and prevents conflict writes in other datanodes. In this sense, this may prvent some deadlocks but it does not remove the chance of deadlocks in general sense. >>>> >>>> On the othe hand, preferred node (datanode again) saves inter-server communication to read replciated table. It does not work to maintain replicated table consistensy but helps to gain some performance. >>>> >>>> Regards; >>>> --- >>>> Koichi Suzuki >>>> >>>> >>>> >>>> ---------- >>>> Koichi Suzuki >>>> >>>> >>>> 2013/4/7 Andrei Martsinchyk <and...@gm...> >>>> >>>> >>>>> >>>>> >>>>> >>>>> >>>>> 2013/4/7 Jov <am...@am...> >>>>> >>>>> datanode use primary node to solve replication table write,it is good,but how coordinator solve the dead lock problem? the coordinator nodes replication all globle catalog tables across coords,they are some kind replication table. >>>>>> >>>>>> >>>>>> eg. >>>>>> >>>>>> client 1 run alter table tb on coord node A,it will lock local catalog data on A,and wait other coord node B. >>>>>> client 2 run alter table tb on coord node B,it will lock local catalog data on B,and wait other coord node A. >>>>>> >>>>>> >>>>>> >>>>>> so how XC handle this dead lock? >>>>>> >>>>>> >>>>> >>>>> >>>>> XC does not handle this, it will be deadlocked. >>>>> Fortunately, chance of concurrent DDL much less then chance of concurrent replicated update. >>>>> >>>>> >>>>> >>>>> >>>>>> >>>>>> 2013/4/6 Andrei Martsinchyk <and...@gm...> >>>>>> >>>>>> PRIMARY was introduced to avoid distributed deadlocks when updating replicated tables. >>>>>>> To better understand the problem, imagine two transactions A and B are updating the same tuple in replicated concurrently. >>>>>>> Normally coordinator sends the same commands to all datanodes at the same time, and if on some node A updates the tuple first, B will be waiting for the end of transaction A. If on other node B wins the race, both transactions will be waiting for each other. It is hard to detect such deadlock, the information about locks is not sent across network. >>>>>>> But it is possible to avoid. The idea is to set one datanode as a primary, and execute distributed update on primary node first, and go on with other nodes only if operation succeeds on primary. >>>>>>> With this approach row lock on primary would stop concurrent transactions from taking row locks on other nodes that could prevent command completion. >>>>>>> So, to have this stuff working properly you should >>>>>>> 1) set only one datanode as a primary; >>>>>>> 2) if you have multiple coordinators, the same datanode should be set as a primary on each of them. >>>>>>> Obvious drawback of the approach is double execution time of replicated updates. >>>>>>> Note: "update" means any write access. >>>>>>> Hope this answers 1)-3) >>>>>>> Regarding 4), the query >>>>>>> >>>>>>> >>>>>>> select nodeoids from pg_class, pgxc_class where pg_class.oid = pcrelid and relname = '<your table name>'; >>>>>>> >>>>>>> >>>>>>> >>>>>>> returns the list of nodes, where the specified table is distributed on. I guess there are 7 of them. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> 2013/4/5 Paul Jones <pb...@cm...> >>>>>>> >>>>>>> >>>>>>>> We are experimenting with an 8-datanode, 3-coordinator cluster and we >>>>>>>> have some questions about the use of PRIMARY and a problem. >>>>>>>> >>>>>>>> The manual explains what PRIMARY means but does not provide much detail >>>>>>>> about when you would use it or not use it. >>>>>>>> >>>>>>>> 1) Can PRIMARY apply to coordinators and if so, when would you >>>>>>>> want it or not? >>>>>>>> >>>>>>>> 2) When would you use PRIMARY for datanodes or not, and would you >>>>>>>> ever want more than one datanode to be a primary? >>>>>>>> >>>>>>>> 3) Does a pgxc_node datanode entry on its own server have to be >>>>>>>> the FQDN server name or can it be 'localhost'? >>>>>>>> >>>>>>>> 4) We have a table that is defined as DISTRIBUTE BY REPLICATION. >>>>>>>> It only loads on the first 7 nodes. It will just not load on >>>>>>>> node 8. There are a lot of FK references from other tables to it, >>>>>>>> but it itself only has a simple CHAR(11) PK, one constraint, >>>>>>>> and 3 indices. >>>>>>>> >>>>>>>> Has anyone seen anything like this before? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Paul Jones >>>>>>>> >>>>>>>> ------------------------------------------------------------------------------ >>>>>>>> Minimize network downtime and maximize team effectiveness. >>>>>>>> Reduce network management and security costs.Learn how to hire >>>>>>>> the most talented Cisco Certified professionals. Visit the >>>>>>>> Employer Resources Portal >>>>>>>> http://www.cisco.com/web/learning/employer_resources/index.html >>>>>>>> _______________________________________________ >>>>>>>> Postgres-xc-general mailing list >>>>>>>> Pos...@li... >>>>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Andrei Martsinchyk >>>>>>> >>>>>>> StormDB - http://www.stormdb.com >>>>>>> The Database Cloud >>>>>>> >>>>>>> >>>>>>> >>>>>>> ------------------------------------------------------------------------------ >>>>>>> Minimize network downtime and maximize team effectiveness. >>>>>>> Reduce network management and security costs.Learn how to hire >>>>>>> the most talented Cisco Certified professionals. Visit the >>>>>>> Employer Resources Portal >>>>>>> http://www.cisco.com/web/learning/employer_resources/index.html >>>>>>> _______________________________________________ >>>>>>> Postgres-xc-general mailing list >>>>>>> Pos...@li... >>>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Jov >>>>>> >>>>>> blog: http:amutu.com/blog >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Andrei Martsinchyk >>>>> >>>>> StormDB - http://www.stormdb.com >>>>> The Database Cloud >>>>> >>>>> >>>>> >>>>> ------------------------------------------------------------------------------ >>>>> Minimize network downtime and maximize team effectiveness. >>>>> Reduce network management and security costs.Learn how to hire >>>>> the most talented Cisco Certified professionals. Visit the >>>>> Employer Resources Portal >>>>> http://www.cisco.com/web/learning/employer_resources/index.html >>>>> _______________________________________________ >>>>> Postgres-xc-general mailing list >>>>> Pos...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general >>>>> >>>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> Minimize network downtime and maximize team effectiveness. >>>> Reduce network management and security costs.Learn how to hire >>>> the most talented Cisco Certified professionals. Visit the >>>> Employer Resources Portal >>>> http://www.cisco.com/web/learning/employer_resources/index.html >>>> _______________________________________________ >>>> Postgres-xc-general mailing list >>>> Pos...@li... >>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general >>>> >>>> >>>> >>> >>> ------------------------------------------------------------------------------ >>> Minimize network downtime and maximize team effectiveness. >>> Reduce network management and security costs.Learn how to hire >>> the most talented Cisco Certified professionals. Visit the >>> Employer Resources Portal >>> http://www.cisco.com/web/learning/employer_resources/index.html >>> _______________________________________________ >>> Postgres-xc-general mailing list >>> Pos...@li... >>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general >>> >> >> >> > |
|
From: Paul J. <pb...@cm...> - 2013-04-15 15:57:23
|
I will send this when we are able to reconfigure our cluster. RIght now, we don't have PRIMARY enabled. PJ >________________________________ > From: Koichi Suzuki <ko...@in...> >To: Paul Jones <pb...@cm...> >Cc: "pos...@li..." <pos...@li...> >Sent: Thursday, April 11, 2013 10:31 PM >Subject: Re: [Postgres-xc-general] Questions about PRIMARY and a problem > > >I did the same thing with 4CO and 4DN, one of them as primary node, and found all the datanode contains identical set or rows. > >Could I have pgxc_node in each coordinator? > >Regards; >--- >Koichi Suzuki > >On Mon, 8 Apr 2013 12:58:52 -0700 (PDT) >Paul Jones <pb...@cm...> wrote: > >> Thanks for everyone's explanation of PRIMARY. It is much clearer now. >> >> I believe, then, that we may have uncovered a bug in PRIMARY. >> >> We created a new cluster (8 nodes, 3 coordinators), but with only one PRIMARY datanode. The PRIMARY was >> declared the same on all 3 coordinators. >> >> A table that was declared DISTRIBUTE BY REPLICATION and loaded by \copy did not have any rows >> present on the PRIMARY! Further, other tables with FK's referring to this table had RI failures when they were loaded, >> even though there were complete copies of the table in all the other non-PRIMARY datanodes. >> >> When we remade the cluster without any PRIMARY, this table loaded into all datanodes and there were no RI failures. >> >> Is this a bug? Unfortunately I won't be able to experiment with this until we finish executing our test plan, perhaps >> a few days. >> >> PJ >> >> >> >> >> >________________________________ >> > From: Koichi Suzuki <koi...@gm...> >> >To: Andrei Martsinchyk <and...@gm...> >> >Cc: "pos...@li..." <pos...@li...> >> >Sent: Sunday, April 7, 2013 9:23 AM >> >Subject: Re: [Postgres-xc-general] Questions about PRIMARY and a problem >> > >> > >> >Primary node is useful to maintain replicated table in a consistent status in all the datanode. All the writes to a replicated table goes first to the primary node so all the conflicts are resolved here and prevents conflict writes in other datanodes. In this sense, this may prvent some deadlocks but it does not remove the chance of deadlocks in general sense. >> > >> >On the othe hand, preferred node (datanode again) saves inter-server communication to read replciated table. It does not work to maintain replicated table consistensy but helps to gain some performance. >> > >> >Regards; >> >--- >> >Koichi Suzuki >> > >> > >> > >> >---------- >> >Koichi Suzuki >> > >> > >> >2013/4/7 Andrei Martsinchyk <and...@gm...> >> > >> > >> >> >> >> >> >> >> >> >> >>2013/4/7 Jov <am...@am...> >> >> >> >>datanode use primary node to solve replication table write,it is good,but how coordinator solve the dead lock problem? the coordinator nodes replication all globle catalog tables across coords,they are some kind replication table. >> >>> >> >>> >> >>>eg. >> >>> >> >>>client 1 run alter table tb on coord node A,it will lock local catalog data on A,and wait other coord node B. >> >>>client 2 run alter table tb on coord node B,it will lock local catalog data on B,and wait other coord node A. >> >>> >> >>> >> >>> >> >>>so how XC handle this dead lock? >> >>> >> >>> >> >> >> >> >> >>XC does not handle this, it will be deadlocked. >> >>Fortunately, chance of concurrent DDL much less then chance of concurrent replicated update. >> >> >> >> >> >> >> >> >> >>> >> >>>2013/4/6 Andrei Martsinchyk <and...@gm...> >> >>> >> >>>PRIMARY was introduced to avoid distributed deadlocks when updating replicated tables. >> >>>>To better understand the problem, imagine two transactions A and B are updating the same tuple in replicated concurrently. >> >>>>Normally coordinator sends the same commands to all datanodes at the same time, and if on some node A updates the tuple first, B will be waiting for the end of transaction A. If on other node B wins the race, both transactions will be waiting for each other. It is hard to detect such deadlock, the information about locks is not sent across network. >> >>>>But it is possible to avoid. The idea is to set one datanode as a primary, and execute distributed update on primary node first, and go on with other nodes only if operation succeeds on primary. >> >>>>With this approach row lock on primary would stop concurrent transactions from taking row locks on other nodes that could prevent command completion. >> >>>>So, to have this stuff working properly you should >> >>>>1) set only one datanode as a primary; >> >>>>2) if you have multiple coordinators, the same datanode should be set as a primary on each of them. >> >>>>Obvious drawback of the approach is double execution time of replicated updates. >> >>>>Note: "update" means any write access. >> >>>>Hope this answers 1)-3) >> >>>>Regarding 4), the query >> >>>> >> >>>> >> >>>>select nodeoids from pg_class, pgxc_class where pg_class.oid = pcrelid and relname = '<your table name>'; >> >>>> >> >>>> >> >>>> >> >>>>returns the list of nodes, where the specified table is distributed on. I guess there are 7 of them. >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>>2013/4/5 Paul Jones <pb...@cm...> >> >>>> >> >>>> >> >>>>>We are experimenting with an 8-datanode, 3-coordinator cluster and we >> >>>>>have some questions about the use of PRIMARY and a problem. >> >>>>> >> >>>>>The manual explains what PRIMARY means but does not provide much detail >> >>>>>about when you would use it or not use it. >> >>>>> >> >>>>>1) Can PRIMARY apply to coordinators and if so, when would you >> >>>>> want it or not? >> >>>>> >> >>>>>2) When would you use PRIMARY for datanodes or not, and would you >> >>>>> ever want more than one datanode to be a primary? >> >>>>> >> >>>>>3) Does a pgxc_node datanode entry on its own server have to be >> >>>>> the FQDN server name or can it be 'localhost'? >> >>>>> >> >>>>>4) We have a table that is defined as DISTRIBUTE BY REPLICATION. >> >>>>> It only loads on the first 7 nodes. It will just not load on >> >>>>> node 8. There are a lot of FK references from other tables to it, >> >>>>> but it itself only has a simple CHAR(11) PK, one constraint, >> >>>>> and 3 indices. >> >>>>> >> >>>>> Has anyone seen anything like this before? >> >>>>> >> >>>>>Thanks, >> >>>>>Paul Jones >> >>>>> >> >>>>>------------------------------------------------------------------------------ >> >>>>>Minimize network downtime and maximize team effectiveness. >> >>>>>Reduce network management and security costs.Learn how to hire >> >>>>>the most talented Cisco Certified professionals. Visit the >> >>>>>Employer Resources Portal >> >>>>>http://www.cisco.com/web/learning/employer_resources/index.html >> >>>>>_______________________________________________ >> >>>>>Postgres-xc-general mailing list >> >>>>>Pos...@li... >> >>>>>https://lists.sourceforge.net/lists/listinfo/postgres-xc-general >> >>>>> >> >>>> >> >>>> >> >>>> >> >>>>-- >> >>>>Andrei Martsinchyk >> >>>> >> >>>>StormDB - http://www.stormdb.com >> >>>>The Database Cloud >> >>>> >> >>>> >> >>>> >> >>>>------------------------------------------------------------------------------ >> >>>>Minimize network downtime and maximize team effectiveness. >> >>>>Reduce network management and security costs.Learn how to hire >> >>>>the most talented Cisco Certified professionals. Visit the >> >>>>Employer Resources Portal >> >>>>http://www.cisco.com/web/learning/employer_resources/index.html >> >>>>_______________________________________________ >> >>>>Postgres-xc-general mailing list >> >>>>Pos...@li... >> >>>>https://lists.sourceforge.net/lists/listinfo/postgres-xc-general >> >>>> >> >>>> >> >>> >> >>> >> >>> >> >>>-- >> >>>Jov >> >>> >> >>>blog: http:amutu.com/blog >> >> >> >> >> >> >> >> >> >>-- >> >>Andrei Martsinchyk >> >> >> >>StormDB - http://www.stormdb.com >> >>The Database Cloud >> >> >> >> >> >> >> >>------------------------------------------------------------------------------ >> >>Minimize network downtime and maximize team effectiveness. >> >>Reduce network management and security costs.Learn how to hire >> >>the most talented Cisco Certified professionals. Visit the >> >>Employer Resources Portal >> >>http://www.cisco.com/web/learning/employer_resources/index.html >> >>_______________________________________________ >> >>Postgres-xc-general mailing list >> >>Pos...@li... >> >>https://lists.sourceforge.net/lists/listinfo/postgres-xc-general >> >> >> >> >> > >> >------------------------------------------------------------------------------ >> >Minimize network downtime and maximize team effectiveness. >> >Reduce network management and security costs.Learn how to hire >> >the most talented Cisco Certified professionals. Visit the >> >Employer Resources Portal >> >http://www.cisco.com/web/learning/employer_resources/index.html >> >_______________________________________________ >> >Postgres-xc-general mailing list >> >Pos...@li... >> >https://lists.sourceforge.net/lists/listinfo/postgres-xc-general >> > >> > >> > >> >> ------------------------------------------------------------------------------ >> Minimize network downtime and maximize team effectiveness. >> Reduce network management and security costs.Learn how to hire >> the most talented Cisco Certified professionals. Visit the >> Employer Resources Portal >> http://www.cisco.com/web/learning/employer_resources/index.html >> _______________________________________________ >> Postgres-xc-general mailing list >> Pos...@li... >> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general >> > > > |
|
From: Theodotos A. <th...@ub...> - 2013-04-15 13:49:14
|
I found another one on line 4566: if [ "$1" == "gtm" ] || [ "$1" == "$gtmName"]; then t needs a space before ']' if [ "$1" =="gtm" ] || ["$1" =="$gtmName" ]; then On 04/15/2013 04:09 AM, 鈴木 幸市 wrote: > Oh, it's a bug. log_eco should be log_echo, which is pgxc_ctl internal function. > > Verry sorry for the mistake. I committed the fix to git repository. > > Please checkin the latest one. > > Thank you very much; > --- > Koichi Suzuki > > > On 2013/04/13, at 8:06, Theodotos Andreou <th...@ub...> wrote: > >> Hello to all, >> >> When I run the following command I get >> $ pgxc_ctl start all >> /var/lib/postgres-xc/bin/pgxc_ctl: line 4003: log_eco: command not found >> >> Is this an actual command or a function inside pgxc_ctl? >> >> ------------------------------------------------------------------------------ >> Precog is a next-generation analytics platform capable of advanced >> analytics on semi-structured data. The platform includes APIs for building >> apps and a phenomenal toolset for data science. Developers can use >> our toolset for easy data analysis & visualization. Get a free account! >> http://www2.precog.com/precogplatform/slashdotnewsletter >> _______________________________________________ >> Postgres-xc-general mailing list >> Pos...@li... >> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general >> |
|
From: 鈴木 幸市 <ko...@in...> - 2013-04-15 01:10:09
|
Oh, it's a bug. log_eco should be log_echo, which is pgxc_ctl internal function. Verry sorry for the mistake. I committed the fix to git repository. Please checkin the latest one. Thank you very much; --- Koichi Suzuki On 2013/04/13, at 8:06, Theodotos Andreou <th...@ub...> wrote: > Hello to all, > > When I run the following command I get > $ pgxc_ctl start all > /var/lib/postgres-xc/bin/pgxc_ctl: line 4003: log_eco: command not found > > Is this an actual command or a function inside pgxc_ctl? > > ------------------------------------------------------------------------------ > Precog is a next-generation analytics platform capable of advanced > analytics on semi-structured data. The platform includes APIs for building > apps and a phenomenal toolset for data science. Developers can use > our toolset for easy data analysis & visualization. Get a free account! > http://www2.precog.com/precogplatform/slashdotnewsletter > _______________________________________________ > Postgres-xc-general mailing list > Pos...@li... > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general > |
|
From: Mason S. <mas...@gm...> - 2013-04-14 00:02:50
|
Hi Ashutosh, I will post a link to the updated slide deck next week. Regards, Mason On Fri, Apr 12, 2013 at 4:19 AM, Ashutosh Bapat < ash...@en...> wrote: > This is interesting. > I have been pondering over this. Can you please share some more details? > > > On Thu, Apr 11, 2013 at 12:25 PM, Mason Sharp <mas...@gm...>wrote: > >> I will be presenting on Monday April 15th in Philadelphia at the Philly >> PUG about using Postgres-XC as a key-value store. >> >> I will discuss hstore, XML and JSON, and include performance comparisons >> to MongoDB. >> >> More information can be found here: >> >> http://www.phlpug.org/events/112128082/?eventId=112128082&action=detail >> >> Thanks, >> >> Mason Sharp >> >> >> ------------------------------------------------------------------------------ >> Precog is a next-generation analytics platform capable of advanced >> analytics on semi-structured data. The platform includes APIs for building >> apps and a phenomenal toolset for data science. Developers can use >> our toolset for easy data analysis & visualization. Get a free account! >> http://www2.precog.com/precogplatform/slashdotnewsletter >> _______________________________________________ >> postgres-xc-announce mailing list >> pos...@li... >> https://lists.sourceforge.net/lists/listinfo/postgres-xc-announce >> >> > > > -- > Best Wishes, > Ashutosh Bapat > EntepriseDB Corporation > The Enterprise Postgres Company > |
|
From: Theodotos A. <th...@ub...> - 2013-04-12 23:06:19
|
Hello to all, When I run the following command I get $ pgxc_ctl start all /var/lib/postgres-xc/bin/pgxc_ctl: line 4003: log_eco: command not found Is this an actual command or a function inside pgxc_ctl? |
|
From: Ashutosh B. <ash...@en...> - 2013-04-12 03:34:08
|
Thanks Suzuki-san. On Fri, Apr 12, 2013 at 6:35 AM, Koichi Suzuki <ko...@in...>wrote: > This is published in January and Marcos Rodriguez Gomez posted a link to > LinkedIn. > > On Fri, 12 Apr 2013 10:02:38 +0900 > Koichi Suzuki <ko...@in...> wrote: > > > Hello; > > > > Ashutosh's article is now on linux for you site. > > > http://www.linuxforu.com/2012/01/postgres-xc-database-clustering-solution/ > > > > Thanks Ashutosh for the work. > > > > Enjoy everybody. > > --- > > Koichi Suzuki > > > > > ------------------------------------------------------------------------------ > > Precog is a next-generation analytics platform capable of advanced > > analytics on semi-structured data. The platform includes APIs for > building > > apps and a phenomenal toolset for data science. Developers can use > > our toolset for easy data analysis & visualization. Get a free account! > > http://www2.precog.com/precogplatform/slashdotnewsletter > > _______________________________________________ > > Postgres-xc-general mailing list > > Pos...@li... > > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general > > > > > ------------------------------------------------------------------------------ > Precog is a next-generation analytics platform capable of advanced > analytics on semi-structured data. The platform includes APIs for building > apps and a phenomenal toolset for data science. Developers can use > our toolset for easy data analysis & visualization. Get a free account! > http://www2.precog.com/precogplatform/slashdotnewsletter > _______________________________________________ > Postgres-xc-core mailing list > Pos...@li... > https://lists.sourceforge.net/lists/listinfo/postgres-xc-core > -- Best Wishes, Ashutosh Bapat EntepriseDB Corporation The Enterprise Postgres Company |
|
From: Koichi S. <ko...@in...> - 2013-04-12 03:29:52
|
I did the same thing with 4CO and 4DN, one of them as primary node, and found all the datanode contains identical set or rows. Could I have pgxc_node in each coordinator? Regards; --- Koichi Suzuki On Mon, 8 Apr 2013 12:58:52 -0700 (PDT) Paul Jones <pb...@cm...> wrote: > Thanks for everyone's explanation of PRIMARY. It is much clearer now. > > I believe, then, that we may have uncovered a bug in PRIMARY. > > We created a new cluster (8 nodes, 3 coordinators), but with only one PRIMARY datanode. The PRIMARY was > declared the same on all 3 coordinators. > > A table that was declared DISTRIBUTE BY REPLICATION and loaded by \copy did not have any rows > present on the PRIMARY! Further, other tables with FK's referring to this table had RI failures when they were loaded, > even though there were complete copies of the table in all the other non-PRIMARY datanodes. > > When we remade the cluster without any PRIMARY, this table loaded into all datanodes and there were no RI failures. > > Is this a bug? Unfortunately I won't be able to experiment with this until we finish executing our test plan, perhaps > a few days. > > PJ > > > > > >________________________________ > > From: Koichi Suzuki <koi...@gm...> > >To: Andrei Martsinchyk <and...@gm...> > >Cc: "pos...@li..." <pos...@li...> > >Sent: Sunday, April 7, 2013 9:23 AM > >Subject: Re: [Postgres-xc-general] Questions about PRIMARY and a problem > > > > > >Primary node is useful to maintain replicated table in a consistent status in all the datanode. All the writes to a replicated table goes first to the primary node so all the conflicts are resolved here and prevents conflict writes in other datanodes. In this sense, this may prvent some deadlocks but it does not remove the chance of deadlocks in general sense. > > > >On the othe hand, preferred node (datanode again) saves inter-server communication to read replciated table. It does not work to maintain replicated table consistensy but helps to gain some performance. > > > >Regards; > >--- > >Koichi Suzuki > > > > > > > >---------- > >Koichi Suzuki > > > > > >2013/4/7 Andrei Martsinchyk <and...@gm...> > > > > > >> > >> > >> > >> > >>2013/4/7 Jov <am...@am...> > >> > >>datanode use primary node to solve replication table write,it is good,but how coordinator solve the dead lock problem? the coordinator nodes replication all globle catalog tables across coords,they are some kind replication table. > >>> > >>> > >>>eg. > >>> > >>>client 1 run alter table tb on coord node A,it will lock local catalog data on A,and wait other coord node B. > >>>client 2 run alter table tb on coord node B,it will lock local catalog data on B,and wait other coord node A. > >>> > >>> > >>> > >>>so how XC handle this dead lock? > >>> > >>> > >> > >> > >>XC does not handle this, it will be deadlocked. > >>Fortunately, chance of concurrent DDL much less then chance of concurrent replicated update. > >> > >> > >> > >> > >>> > >>>2013/4/6 Andrei Martsinchyk <and...@gm...> > >>> > >>>PRIMARY was introduced to avoid distributed deadlocks when updating replicated tables. > >>>>To better understand the problem, imagine two transactions A and B are updating the same tuple in replicated concurrently. > >>>>Normally coordinator sends the same commands to all datanodes at the same time, and if on some node A updates the tuple first, B will be waiting for the end of transaction A. If on other node B wins the race, both transactions will be waiting for each other. It is hard to detect such deadlock, the information about locks is not sent across network. > >>>>But it is possible to avoid. The idea is to set one datanode as a primary, and execute distributed update on primary node first, and go on with other nodes only if operation succeeds on primary. > >>>>With this approach row lock on primary would stop concurrent transactions from taking row locks on other nodes that could prevent command completion. > >>>>So, to have this stuff working properly you should > >>>>1) set only one datanode as a primary; > >>>>2) if you have multiple coordinators, the same datanode should be set as a primary on each of them. > >>>>Obvious drawback of the approach is double execution time of replicated updates. > >>>>Note: "update" means any write access. > >>>>Hope this answers 1)-3) > >>>>Regarding 4), the query > >>>> > >>>> > >>>>select nodeoids from pg_class, pgxc_class where pg_class.oid = pcrelid and relname = '<your table name>'; > >>>> > >>>> > >>>> > >>>>returns the list of nodes, where the specified table is distributed on. I guess there are 7 of them. > >>>> > >>>> > >>>> > >>>> > >>>> > >>>>2013/4/5 Paul Jones <pb...@cm...> > >>>> > >>>> > >>>>>We are experimenting with an 8-datanode, 3-coordinator cluster and we > >>>>>have some questions about the use of PRIMARY and a problem. > >>>>> > >>>>>The manual explains what PRIMARY means but does not provide much detail > >>>>>about when you would use it or not use it. > >>>>> > >>>>>1) Can PRIMARY apply to coordinators and if so, when would you > >>>>> want it or not? > >>>>> > >>>>>2) When would you use PRIMARY for datanodes or not, and would you > >>>>> ever want more than one datanode to be a primary? > >>>>> > >>>>>3) Does a pgxc_node datanode entry on its own server have to be > >>>>> the FQDN server name or can it be 'localhost'? > >>>>> > >>>>>4) We have a table that is defined as DISTRIBUTE BY REPLICATION. > >>>>> It only loads on the first 7 nodes. It will just not load on > >>>>> node 8. There are a lot of FK references from other tables to it, > >>>>> but it itself only has a simple CHAR(11) PK, one constraint, > >>>>> and 3 indices. > >>>>> > >>>>> Has anyone seen anything like this before? > >>>>> > >>>>>Thanks, > >>>>>Paul Jones > >>>>> > >>>>>------------------------------------------------------------------------------ > >>>>>Minimize network downtime and maximize team effectiveness. > >>>>>Reduce network management and security costs.Learn how to hire > >>>>>the most talented Cisco Certified professionals. Visit the > >>>>>Employer Resources Portal > >>>>>http://www.cisco.com/web/learning/employer_resources/index.html > >>>>>_______________________________________________ > >>>>>Postgres-xc-general mailing list > >>>>>Pos...@li... > >>>>>https://lists.sourceforge.net/lists/listinfo/postgres-xc-general > >>>>> > >>>> > >>>> > >>>> > >>>>-- > >>>>Andrei Martsinchyk > >>>> > >>>>StormDB - http://www.stormdb.com > >>>>The Database Cloud > >>>> > >>>> > >>>> > >>>>------------------------------------------------------------------------------ > >>>>Minimize network downtime and maximize team effectiveness. > >>>>Reduce network management and security costs.Learn how to hire > >>>>the most talented Cisco Certified professionals. Visit the > >>>>Employer Resources Portal > >>>>http://www.cisco.com/web/learning/employer_resources/index.html > >>>>_______________________________________________ > >>>>Postgres-xc-general mailing list > >>>>Pos...@li... > >>>>https://lists.sourceforge.net/lists/listinfo/postgres-xc-general > >>>> > >>>> > >>> > >>> > >>> > >>>-- > >>>Jov > >>> > >>>blog: http:amutu.com/blog > >> > >> > >> > >> > >>-- > >>Andrei Martsinchyk > >> > >>StormDB - http://www.stormdb.com > >>The Database Cloud > >> > >> > >> > >>------------------------------------------------------------------------------ > >>Minimize network downtime and maximize team effectiveness. > >>Reduce network management and security costs.Learn how to hire > >>the most talented Cisco Certified professionals. Visit the > >>Employer Resources Portal > >>http://www.cisco.com/web/learning/employer_resources/index.html > >>_______________________________________________ > >>Postgres-xc-general mailing list > >>Pos...@li... > >>https://lists.sourceforge.net/lists/listinfo/postgres-xc-general > >> > >> > > > >------------------------------------------------------------------------------ > >Minimize network downtime and maximize team effectiveness. > >Reduce network management and security costs.Learn how to hire > >the most talented Cisco Certified professionals. Visit the > >Employer Resources Portal > >http://www.cisco.com/web/learning/employer_resources/index.html > >_______________________________________________ > >Postgres-xc-general mailing list > >Pos...@li... > >https://lists.sourceforge.net/lists/listinfo/postgres-xc-general > > > > > > > > ------------------------------------------------------------------------------ > Minimize network downtime and maximize team effectiveness. > Reduce network management and security costs.Learn how to hire > the most talented Cisco Certified professionals. Visit the > Employer Resources Portal > http://www.cisco.com/web/learning/employer_resources/index.html > _______________________________________________ > Postgres-xc-general mailing list > Pos...@li... > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general > |
|
From: Ashutosh B. <ash...@en...> - 2013-04-12 03:20:02
|
This is interesting. I have been pondering over this. Can you please share some more details? On Thu, Apr 11, 2013 at 12:25 PM, Mason Sharp <mas...@gm...> wrote: > I will be presenting on Monday April 15th in Philadelphia at the Philly > PUG about using Postgres-XC as a key-value store. > > I will discuss hstore, XML and JSON, and include performance comparisons > to MongoDB. > > More information can be found here: > > http://www.phlpug.org/events/112128082/?eventId=112128082&action=detail > > Thanks, > > Mason Sharp > > > ------------------------------------------------------------------------------ > Precog is a next-generation analytics platform capable of advanced > analytics on semi-structured data. The platform includes APIs for building > apps and a phenomenal toolset for data science. Developers can use > our toolset for easy data analysis & visualization. Get a free account! > http://www2.precog.com/precogplatform/slashdotnewsletter > _______________________________________________ > postgres-xc-announce mailing list > pos...@li... > https://lists.sourceforge.net/lists/listinfo/postgres-xc-announce > > -- Best Wishes, Ashutosh Bapat EntepriseDB Corporation The Enterprise Postgres Company |
|
From: Koichi S. <ko...@in...> - 2013-04-12 01:03:39
|
This is published in January and Marcos Rodriguez Gomez posted a link to LinkedIn. On Fri, 12 Apr 2013 10:02:38 +0900 Koichi Suzuki <ko...@in...> wrote: > Hello; > > Ashutosh's article is now on linux for you site. > http://www.linuxforu.com/2012/01/postgres-xc-database-clustering-solution/ > > Thanks Ashutosh for the work. > > Enjoy everybody. > --- > Koichi Suzuki > > ------------------------------------------------------------------------------ > Precog is a next-generation analytics platform capable of advanced > analytics on semi-structured data. The platform includes APIs for building > apps and a phenomenal toolset for data science. Developers can use > our toolset for easy data analysis & visualization. Get a free account! > http://www2.precog.com/precogplatform/slashdotnewsletter > _______________________________________________ > Postgres-xc-general mailing list > Pos...@li... > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general > |
|
From: Koichi S. <ko...@in...> - 2013-04-12 01:01:11
|
Hello; Ashutosh's article is now on linux for you site. http://www.linuxforu.com/2012/01/postgres-xc-database-clustering-solution/ Thanks Ashutosh for the work. Enjoy everybody. --- Koichi Suzuki |
|
From: Koichi S. <ko...@in...> - 2013-04-12 00:52:29
|
On Thu, 11 Apr 2013 18:07:13 +0900 Michael Paquier <mic...@gm...> wrote: > On Thu, Apr 11, 2013 at 4:22 PM, Theodotos Andreou <th...@ub...>wrote: > > > On 04/11/2013 04:05 AM, Hitoshi HEMMI wrote: > > > Hi Theodotos, > > > > > >> This is great! Are these RA open/freely available? Download Links? > > >> Maybe some guide/documentation? > > > > > > We use following RA. > > > > > https://github.com/t-matsuo/resource-agents/blob/7bf4daf29ec6d2e0aed4cf4b39a53118c2986a5a/heartbeat/pgsql > > > > > > This is for PostgreSQL but works fine for PGXC coordinator and > > > datanode without modification. > > > This is not for gtm/gtm-proxy. > > > > > > Hitoshi > > > > > > > Thanks Hitoshi! I will use this for coordinators and datanodes and > > pgxc_ctl for gtm. AFAIK there is no need for gtm_proxy failover, right? > > > Does pgxc_ctl provide failover and monitoring of GTM? > I thought it was only used for cluster set up... Yes it does. https://github.com/koichi-szk/PGXC-Tools/blob/master/pgxc_ctl/manual.txt gives bash-version manual, most of the features are in C-version as well. C-version will come with additional features as: 1) Add slave on the fly, 2) Add master on the fly (depends upon Abbas's node addition/removal), 3) Add gtm proxy on the fly. Please understand that pgxc_ctl is not "automatic" failover system. This feature will be available when it is integrated with HA middleware. I believe NTT people are developing such feature as general RA for pacemaker/heartbeat and I expect this will run with corosync as well. A couple of bash-version features depend upon direct support by bash and are not imported to C-version, although it has some alternatives. Regards; --- Koichi > -- > Michael |
|
From: Michael P. <mic...@gm...> - 2013-04-11 09:07:24
|
On Thu, Apr 11, 2013 at 4:22 PM, Theodotos Andreou <th...@ub...>wrote: > On 04/11/2013 04:05 AM, Hitoshi HEMMI wrote: > > Hi Theodotos, > > > >> This is great! Are these RA open/freely available? Download Links? > >> Maybe some guide/documentation? > > > > We use following RA. > > > https://github.com/t-matsuo/resource-agents/blob/7bf4daf29ec6d2e0aed4cf4b39a53118c2986a5a/heartbeat/pgsql > > > > This is for PostgreSQL but works fine for PGXC coordinator and > > datanode without modification. > > This is not for gtm/gtm-proxy. > > > > Hitoshi > > > > Thanks Hitoshi! I will use this for coordinators and datanodes and > pgxc_ctl for gtm. AFAIK there is no need for gtm_proxy failover, right? > Does pgxc_ctl provide failover and monitoring of GTM? I thought it was only used for cluster set up... -- Michael |
|
From: Koichi S. <koi...@gm...> - 2013-04-11 07:26:26
|
Wonderful! It is another are for XC. As I've just announced, we're going to have XC-Day at Ottawa, May 21st Tuesday, just after the cluster summit. I'd like to share your presentation and feedback from the audience there. ---------- Koichi Suzuki 2013/4/11 Mason Sharp <mas...@gm...> > I will be presenting on Monday April 15th in Philadelphia at the Philly > PUG about using Postgres-XC as a key-value store. > > I will discuss hstore, XML and JSON, and include performance comparisons > to MongoDB. > > More information can be found here: > > http://www.phlpug.org/events/112128082/?eventId=112128082&action=detail > > Thanks, > > Mason Sharp > > > ------------------------------------------------------------------------------ > Precog is a next-generation analytics platform capable of advanced > analytics on semi-structured data. The platform includes APIs for building > apps and a phenomenal toolset for data science. Developers can use > our toolset for easy data analysis & visualization. Get a free account! > http://www2.precog.com/precogplatform/slashdotnewsletter > _______________________________________________ > postgres-xc-announce mailing list > pos...@li... > https://lists.sourceforge.net/lists/listinfo/postgres-xc-announce > > |
|
From: Theodotos A. <th...@ub...> - 2013-04-11 07:22:56
|
On 04/11/2013 04:05 AM, Hitoshi HEMMI wrote: > Hi Theodotos, > >> This is great! Are these RA open/freely available? Download Links? >> Maybe some guide/documentation? > > We use following RA. > https://github.com/t-matsuo/resource-agents/blob/7bf4daf29ec6d2e0aed4cf4b39a53118c2986a5a/heartbeat/pgsql > > This is for PostgreSQL but works fine for PGXC coordinator and > datanode without modification. > This is not for gtm/gtm-proxy. > > Hitoshi > Thanks Hitoshi! I will use this for coordinators and datanodes and pgxc_ctl for gtm. AFAIK there is no need for gtm_proxy failover, right? > > Theodotos Andreou さんは書きました: >> On 04/08/2013 06:52 AM, Nikhil Sontakke wrote: >>> Hi Theodotos, >>> >>>> Automatic monitoring and failover has not been provided yet. As >>>> Michael >>>> suggested, yes it's up to you at present. Sakata-san at NTT >>>> announced that >>>> they're developing XC RA for pacemaker/heartbeat (I hope it runs with >>>> Corosync too). >>>> >>> Yes, resource agents for pacemaker/corosync indeed work quite well for >>> XC HA. And yeah using heartbeat or corosync should be equivalent >>> mostly. >>> >>> We have indeed written custom agents for datanode and GTM monitoring >>> and HA using Corosync/Pacemaker and they work pretty well. >>> >>> Regards, >>> Nikhils >> This is great! Are these RA open/freely available? Download Links? >> Maybe some guide/documentation? >> >> ------------------------------------------------------------------------------ >> >> Minimize network downtime and maximize team effectiveness. >> Reduce network management and security costs.Learn how to hire the >> most talented Cisco Certified professionals. Visit the Employer >> Resources Portal >> http://www.cisco.com/web/learning/employer_resources/index.html >> _______________________________________________ >> Postgres-xc-general mailing list >> Pos...@li... >> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general >> > > |
|
From: Mason S. <mas...@gm...> - 2013-04-11 06:55:27
|
I will be presenting on Monday April 15th in Philadelphia at the Philly PUG about using Postgres-XC as a key-value store. I will discuss hstore, XML and JSON, and include performance comparisons to MongoDB. More information can be found here: http://www.phlpug.org/events/112128082/?eventId=112128082&action=detail Thanks, Mason Sharp |
2 messages has been excluded from this view by a project administrator.
