postgres-xc-developers Mailing List for Postgres-XC

On Tue, Feb 19, 2013 at 7:11 PM, Koichi Suzuki <koi...@gm...>wrote:

> Regression test will create two coordinators and datanodes with fixed
> names.   In this case, we don't have to worry about the names.
> Anyway, we have to rewrite affected test cases, which could take time.
>   I'm worrying about this.
>
On the contrary, you have to worry about the names as well, make
installcheck should work with the same transparency as make check.
-- 
Michael

I'm afraid the discussion is going to "all-or-nothing" direction.
Apparently, there's a case where we can determine the target datanode
at planning, as well as the opposite case.    Of course, cache
invalidation is another issue to consider, when we determine the
target datanode at the planning.

So the issue looks to be:
1) How we determine when to determine actual target node,
2) If target node is determined at planning and the cache invalidated,
how to recover from this.

Regards;
----------
Koichi Suzuki


2013/2/15 Ashutosh Bapat <ash...@en...>:
>
>
> On Fri, Feb 15, 2013 at 11:19 AM, Nikhil Sontakke <ni...@st...>
> wrote:
>>
>> >> > There is no way for the second session to know that there are
>> >> > prepared
>> >> > statements out there, so there is no stopping it from changing the
>> >> > preferred
>> >> > nodes. At the same time, I do not see any need to do so. Preferred
>> >> > node
>> >> > should be a dynamic non-blocking setting, very handy to tackle the
>> >> > network
>> >> > dynamics if needed.
>> >> >
>> >>
>> >> In this case the plan cache should be invalidated and re-built afresh.
>> >> The case should be similar to when the involved relations in a plan
>> >> undergo a change AFAICS.
>> >>
>> >
>> > When we can take care of this stuff (and without much change in the
>> > code),
>> > without plan cache invalidation, why to go the route of cache
>> > invalidation?
>> > Plan cache invalidation is a heavy operation in itself and its
>> > consequences
>> > cause performance degradation. Because of plan invalidation all those
>> > plans
>> > which are really not affected by this change, would be hit too.
>> >
>>
>> You make this node ddl stuff sound like a routine activity. It's going
>> to be once in a while, right?
>>
>
> You seem to have forgot the other argument I made about inserts not being
> well-balanced for round-robin tables, which is another argument I have put
> forth for this change.
>
>>
>> Regards,
>> Nikhils
>> --
>> StormDB - http://www.stormdb.com
>> The Database Cloud
>> Postgres-XC Support and Service
>
>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Enterprise Postgres Company
>
> ------------------------------------------------------------------------------
> Free Next-Gen Firewall Hardware Offer
> Buy your Sophos next-gen firewall before the end March 2013
> and get the hardware for free! Learn more.
> http://p.sf.net/sfu/sophos-d2d-feb
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>

Suzuki-san,
We haven't yet standardized any regression environment. The one you are
talking about is what NTT uses. I use a different one and I am sure
everybody else has their own. We took a lot of efforts while writing xc
specific tests to make sure that our tests run and pass on variety of
regression environments so that developers do not have to stick to one or
the other environment. I think we should continue to do that.

On Tue, Feb 19, 2013 at 3:41 PM, Koichi Suzuki <koi...@gm...>wrote:

> Regression test will create two coordinators and datanodes with fixed
> names.   In this case, we don't have to worry about the names.
> Anyway, we have to rewrite affected test cases, which could take time.
>   I'm worrying about this.
>
> Xion, sorry for taking this long.   Maybe it's much better to have
> separate command or to use pre-defined views.
>
> Any more thoughts?
> ----------
> Koichi Suzuki
>
>
> 2013/2/19 Nikhil Sontakke <ni...@st...>:
> > Hi,
> >
> >> According to Ashutosh Bapat, could we just print ALL NODES if the count
> of
> >> nodelist is the same as the count of datanodes, otherwise print the node
> >> list. By this way, the regression test will be resolved.
> >>
> >
> > Umm, I don't particularly like this idea of printing "ALL DATANODES"
> > just to make regressions work...
> >
> > Can we not document that the standard testing environment should have
> > a so and so config with recommended names for coordinators and
> > datanodes?
> >
> > Regards,
> > Nikhils
> >
> >
> >> Thanks & Regards,
> >>
> >> Benny Wang
> >>
> >> 2013/2/15 Ashutosh Bapat <ash...@en...>
> >>>
> >>>
> >>>
> >>> On Fri, Feb 15, 2013 at 8:28 AM, Koichi Suzuki <
> koi...@gm...>
> >>> wrote:
> >>>>
> >>>> I found \d+ is used 41 times in the regression.   If we extend \d+
> >>>> with table distribution, we need to (continue to) modify all of them.
> >>>>  Is it too much?
> >>>>
> >>>
> >>> I don't see regression expected output changes to be of a lot of
> concern.
> >>> We can do them ones and for all. There might be some merge conflicts
> later,
> >>> but that's something, which would part of our ongoing merge. That's
> >>> inevitable for a project like XC, which pulls from PG.
> >>>
> >>> My concern is different. \d+ or \d whatever, would start printing the
> list
> >>> of nodes the table is distributed on, which will be dependent upon the
> >>> individual's enviroment, and names and number of nodes chosen. So,
> tests
> >>> passing for one would not do so for someone else. So, we will need
> some way
> >>> to canonicalize the list of nodes so that it comes out to be same
> >>> independent of the environment.
> >>>
> >>> I suggest this, instead of \d+ printing the individual nodes, it first
> >>> looks at the node groups for a matching node group and prints the name
> of
> >>> group if it finds one matching. Next, if all the datanodes are in the
> list
> >>> of nodes, it would print ALL DATANODES or something like that. Since
> the
> >>> regression tests use default node list, which has all datanodes, for
> almost
> >>> (I am sure all) all \d+ outputs we will get ALL DATANODES, which will
> be
> >>> independent of the environment used.
> >>>
> >>>>
> >>>> Regards;
> >>>> ----------
> >>>> Koichi Suzuki
> >>>>
> >>>>
> >>>> 2013/2/14 Michael Paquier <mic...@gm...>:
> >>>> >
> >>>> >
> >>>> > On Thu, Feb 14, 2013 at 7:44 PM, Nikhil Sontakke <
> ni...@st...>
> >>>> > wrote:
> >>>> >>
> >>>> >> Michael seems to be doing a see-saw here :)
> >>>> >
> >>>> > After doing manually regression merges for 9.1 and 9.2 on more than
> 50
> >>>> > regression tests for each merge, you learn quite a bit... Making it
> >>>> > more
> >>>> > painful will just consume more time than necessary.
> >>>> >
> >>>> >> On some other thread he seemed ok to extend \d+
> >>>> >
> >>>> > Call that experience, I was OK with with ¥d+ before actually having
> a
> >>>> > look
> >>>> > at the number of tests using it.
> >>>> > --
> >>>> > Michael
> >>>>
> >>>>
> >>>>
> ------------------------------------------------------------------------------
> >>>> Free Next-Gen Firewall Hardware Offer
> >>>> Buy your Sophos next-gen firewall before the end March 2013
> >>>> and get the hardware for free! Learn more.
> >>>> http://p.sf.net/sfu/sophos-d2d-feb
> >>>> _______________________________________________
> >>>> Postgres-xc-developers mailing list
> >>>> Pos...@li...
> >>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> Best Wishes,
> >>> Ashutosh Bapat
> >>> EntepriseDB Corporation
> >>> The Enterprise Postgres Company
> >>>
> >>>
> >>>
> ------------------------------------------------------------------------------
> >>> Free Next-Gen Firewall Hardware Offer
> >>> Buy your Sophos next-gen firewall before the end March 2013
> >>> and get the hardware for free! Learn more.
> >>> http://p.sf.net/sfu/sophos-d2d-feb
> >>> _______________________________________________
> >>> Postgres-xc-developers mailing list
> >>> Pos...@li...
> >>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
> >>>
> >>
> >>
> >>
> ------------------------------------------------------------------------------
> >> Everyone hates slow websites. So do we.
> >> Make your web apps faster with AppDynamics
> >> Download AppDynamics Lite for free today:
> >> http://p.sf.net/sfu/appdyn_d2d_feb
> >> _______________________________________________
> >> Postgres-xc-developers mailing list
> >> Pos...@li...
> >> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
> >>
> >
> >
> >
> > --
> > StormDB - http://www.stormdb.com
> > The Database Cloud
> > Postgres-XC Support and Service
> >
> >
> ------------------------------------------------------------------------------
> > Everyone hates slow websites. So do we.
> > Make your web apps faster with AppDynamics
> > Download AppDynamics Lite for free today:
> > http://p.sf.net/sfu/appdyn_d2d_feb
> > _______________________________________________
> > Postgres-xc-developers mailing list
> > Pos...@li...
> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_feb
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>



-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

On Tue, Feb 19, 2013 at 1:32 PM, Nikhil Sontakke <ni...@st...>wrote:

> Hi,
>
> > According to Ashutosh Bapat, could we just print ALL NODES if the count
> of
> > nodelist is the same as the count of datanodes, otherwise print the node
> > list. By this way, the regression test will be resolved.
> >
>
> Umm, I don't particularly like this idea of printing "ALL DATANODES"
> just to make regressions work...
>
>
Making regressions work is one thing, but it will be good for user to know
that the table is distributed/replicated on "ALL" nodes rather than nodes
d1, d2, d3, .., dn. It becomes easy for the user to spot whether a table
intended to be on all nodes is that way or not.


> Can we not document that the standard testing environment should have
> a so and so config with recommended names for coordinators and
> datanodes?
>
> Regards,
> Nikhils
>
>
> > Thanks & Regards,
> >
> > Benny Wang
> >
> > 2013/2/15 Ashutosh Bapat <ash...@en...>
> >>
> >>
> >>
> >> On Fri, Feb 15, 2013 at 8:28 AM, Koichi Suzuki <
> koi...@gm...>
> >> wrote:
> >>>
> >>> I found \d+ is used 41 times in the regression.   If we extend \d+
> >>> with table distribution, we need to (continue to) modify all of them.
> >>>  Is it too much?
> >>>
> >>
> >> I don't see regression expected output changes to be of a lot of
> concern.
> >> We can do them ones and for all. There might be some merge conflicts
> later,
> >> but that's something, which would part of our ongoing merge. That's
> >> inevitable for a project like XC, which pulls from PG.
> >>
> >> My concern is different. \d+ or \d whatever, would start printing the
> list
> >> of nodes the table is distributed on, which will be dependent upon the
> >> individual's enviroment, and names and number of nodes chosen. So, tests
> >> passing for one would not do so for someone else. So, we will need some
> way
> >> to canonicalize the list of nodes so that it comes out to be same
> >> independent of the environment.
> >>
> >> I suggest this, instead of \d+ printing the individual nodes, it first
> >> looks at the node groups for a matching node group and prints the name
> of
> >> group if it finds one matching. Next, if all the datanodes are in the
> list
> >> of nodes, it would print ALL DATANODES or something like that. Since the
> >> regression tests use default node list, which has all datanodes, for
> almost
> >> (I am sure all) all \d+ outputs we will get ALL DATANODES, which will be
> >> independent of the environment used.
> >>
> >>>
> >>> Regards;
> >>> ----------
> >>> Koichi Suzuki
> >>>
> >>>
> >>> 2013/2/14 Michael Paquier <mic...@gm...>:
> >>> >
> >>> >
> >>> > On Thu, Feb 14, 2013 at 7:44 PM, Nikhil Sontakke <
> ni...@st...>
> >>> > wrote:
> >>> >>
> >>> >> Michael seems to be doing a see-saw here :)
> >>> >
> >>> > After doing manually regression merges for 9.1 and 9.2 on more than
> 50
> >>> > regression tests for each merge, you learn quite a bit... Making it
> >>> > more
> >>> > painful will just consume more time than necessary.
> >>> >
> >>> >> On some other thread he seemed ok to extend \d+
> >>> >
> >>> > Call that experience, I was OK with with ¥d+ before actually having a
> >>> > look
> >>> > at the number of tests using it.
> >>> > --
> >>> > Michael
> >>>
> >>>
> >>>
> ------------------------------------------------------------------------------
> >>> Free Next-Gen Firewall Hardware Offer
> >>> Buy your Sophos next-gen firewall before the end March 2013
> >>> and get the hardware for free! Learn more.
> >>> http://p.sf.net/sfu/sophos-d2d-feb
> >>> _______________________________________________
> >>> Postgres-xc-developers mailing list
> >>> Pos...@li...
> >>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
> >>
> >>
> >>
> >>
> >> --
> >> Best Wishes,
> >> Ashutosh Bapat
> >> EntepriseDB Corporation
> >> The Enterprise Postgres Company
> >>
> >>
> >>
> ------------------------------------------------------------------------------
> >> Free Next-Gen Firewall Hardware Offer
> >> Buy your Sophos next-gen firewall before the end March 2013
> >> and get the hardware for free! Learn more.
> >> http://p.sf.net/sfu/sophos-d2d-feb
> >> _______________________________________________
> >> Postgres-xc-developers mailing list
> >> Pos...@li...
> >> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
> >>
> >
> >
> >
> ------------------------------------------------------------------------------
> > Everyone hates slow websites. So do we.
> > Make your web apps faster with AppDynamics
> > Download AppDynamics Lite for free today:
> > http://p.sf.net/sfu/appdyn_d2d_feb
> > _______________________________________________
> > Postgres-xc-developers mailing list
> > Pos...@li...
> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
> >
>
>
>
> --
> StormDB - http://www.stormdb.com
> The Database Cloud
> Postgres-XC Support and Service
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_feb
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>



-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

Regression test will create two coordinators and datanodes with fixed
names.   In this case, we don't have to worry about the names.
Anyway, we have to rewrite affected test cases, which could take time.
  I'm worrying about this.

Xion, sorry for taking this long.   Maybe it's much better to have
separate command or to use pre-defined views.

Any more thoughts?
----------
Koichi Suzuki


2013/2/19 Nikhil Sontakke <ni...@st...>:
> Hi,
>
>> According to Ashutosh Bapat, could we just print ALL NODES if the count of
>> nodelist is the same as the count of datanodes, otherwise print the node
>> list. By this way, the regression test will be resolved.
>>
>
> Umm, I don't particularly like this idea of printing "ALL DATANODES"
> just to make regressions work...
>
> Can we not document that the standard testing environment should have
> a so and so config with recommended names for coordinators and
> datanodes?
>
> Regards,
> Nikhils
>
>
>> Thanks & Regards,
>>
>> Benny Wang
>>
>> 2013/2/15 Ashutosh Bapat <ash...@en...>
>>>
>>>
>>>
>>> On Fri, Feb 15, 2013 at 8:28 AM, Koichi Suzuki <koi...@gm...>
>>> wrote:
>>>>
>>>> I found \d+ is used 41 times in the regression.   If we extend \d+
>>>> with table distribution, we need to (continue to) modify all of them.
>>>>  Is it too much?
>>>>
>>>
>>> I don't see regression expected output changes to be of a lot of concern.
>>> We can do them ones and for all. There might be some merge conflicts later,
>>> but that's something, which would part of our ongoing merge. That's
>>> inevitable for a project like XC, which pulls from PG.
>>>
>>> My concern is different. \d+ or \d whatever, would start printing the list
>>> of nodes the table is distributed on, which will be dependent upon the
>>> individual's enviroment, and names and number of nodes chosen. So, tests
>>> passing for one would not do so for someone else. So, we will need some way
>>> to canonicalize the list of nodes so that it comes out to be same
>>> independent of the environment.
>>>
>>> I suggest this, instead of \d+ printing the individual nodes, it first
>>> looks at the node groups for a matching node group and prints the name of
>>> group if it finds one matching. Next, if all the datanodes are in the list
>>> of nodes, it would print ALL DATANODES or something like that. Since the
>>> regression tests use default node list, which has all datanodes, for almost
>>> (I am sure all) all \d+ outputs we will get ALL DATANODES, which will be
>>> independent of the environment used.
>>>
>>>>
>>>> Regards;
>>>> ----------
>>>> Koichi Suzuki
>>>>
>>>>
>>>> 2013/2/14 Michael Paquier <mic...@gm...>:
>>>> >
>>>> >
>>>> > On Thu, Feb 14, 2013 at 7:44 PM, Nikhil Sontakke <ni...@st...>
>>>> > wrote:
>>>> >>
>>>> >> Michael seems to be doing a see-saw here :)
>>>> >
>>>> > After doing manually regression merges for 9.1 and 9.2 on more than 50
>>>> > regression tests for each merge, you learn quite a bit... Making it
>>>> > more
>>>> > painful will just consume more time than necessary.
>>>> >
>>>> >> On some other thread he seemed ok to extend \d+
>>>> >
>>>> > Call that experience, I was OK with with ¥d+ before actually having a
>>>> > look
>>>> > at the number of tests using it.
>>>> > --
>>>> > Michael
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Free Next-Gen Firewall Hardware Offer
>>>> Buy your Sophos next-gen firewall before the end March 2013
>>>> and get the hardware for free! Learn more.
>>>> http://p.sf.net/sfu/sophos-d2d-feb
>>>> _______________________________________________
>>>> Postgres-xc-developers mailing list
>>>> Pos...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>>
>>>
>>>
>>>
>>> --
>>> Best Wishes,
>>> Ashutosh Bapat
>>> EntepriseDB Corporation
>>> The Enterprise Postgres Company
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Free Next-Gen Firewall Hardware Offer
>>> Buy your Sophos next-gen firewall before the end March 2013
>>> and get the hardware for free! Learn more.
>>> http://p.sf.net/sfu/sophos-d2d-feb
>>> _______________________________________________
>>> Postgres-xc-developers mailing list
>>> Pos...@li...
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Everyone hates slow websites. So do we.
>> Make your web apps faster with AppDynamics
>> Download AppDynamics Lite for free today:
>> http://p.sf.net/sfu/appdyn_d2d_feb
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>
>
>
> --
> StormDB - http://www.stormdb.com
> The Database Cloud
> Postgres-XC Support and Service
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_feb
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers

Hi,

> According to Ashutosh Bapat, could we just print ALL NODES if the count of
> nodelist is the same as the count of datanodes, otherwise print the node
> list. By this way, the regression test will be resolved.
>

Umm, I don't particularly like this idea of printing "ALL DATANODES"
just to make regressions work...

Can we not document that the standard testing environment should have
a so and so config with recommended names for coordinators and
datanodes?

Regards,
Nikhils


> Thanks & Regards,
>
> Benny Wang
>
> 2013/2/15 Ashutosh Bapat <ash...@en...>
>>
>>
>>
>> On Fri, Feb 15, 2013 at 8:28 AM, Koichi Suzuki <koi...@gm...>
>> wrote:
>>>
>>> I found \d+ is used 41 times in the regression.   If we extend \d+
>>> with table distribution, we need to (continue to) modify all of them.
>>>  Is it too much?
>>>
>>
>> I don't see regression expected output changes to be of a lot of concern.
>> We can do them ones and for all. There might be some merge conflicts later,
>> but that's something, which would part of our ongoing merge. That's
>> inevitable for a project like XC, which pulls from PG.
>>
>> My concern is different. \d+ or \d whatever, would start printing the list
>> of nodes the table is distributed on, which will be dependent upon the
>> individual's enviroment, and names and number of nodes chosen. So, tests
>> passing for one would not do so for someone else. So, we will need some way
>> to canonicalize the list of nodes so that it comes out to be same
>> independent of the environment.
>>
>> I suggest this, instead of \d+ printing the individual nodes, it first
>> looks at the node groups for a matching node group and prints the name of
>> group if it finds one matching. Next, if all the datanodes are in the list
>> of nodes, it would print ALL DATANODES or something like that. Since the
>> regression tests use default node list, which has all datanodes, for almost
>> (I am sure all) all \d+ outputs we will get ALL DATANODES, which will be
>> independent of the environment used.
>>
>>>
>>> Regards;
>>> ----------
>>> Koichi Suzuki
>>>
>>>
>>> 2013/2/14 Michael Paquier <mic...@gm...>:
>>> >
>>> >
>>> > On Thu, Feb 14, 2013 at 7:44 PM, Nikhil Sontakke <ni...@st...>
>>> > wrote:
>>> >>
>>> >> Michael seems to be doing a see-saw here :)
>>> >
>>> > After doing manually regression merges for 9.1 and 9.2 on more than 50
>>> > regression tests for each merge, you learn quite a bit... Making it
>>> > more
>>> > painful will just consume more time than necessary.
>>> >
>>> >> On some other thread he seemed ok to extend \d+
>>> >
>>> > Call that experience, I was OK with with ¥d+ before actually having a
>>> > look
>>> > at the number of tests using it.
>>> > --
>>> > Michael
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Free Next-Gen Firewall Hardware Offer
>>> Buy your Sophos next-gen firewall before the end March 2013
>>> and get the hardware for free! Learn more.
>>> http://p.sf.net/sfu/sophos-d2d-feb
>>> _______________________________________________
>>> Postgres-xc-developers mailing list
>>> Pos...@li...
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>>
>>
>>
>> --
>> Best Wishes,
>> Ashutosh Bapat
>> EntepriseDB Corporation
>> The Enterprise Postgres Company
>>
>>
>> ------------------------------------------------------------------------------
>> Free Next-Gen Firewall Hardware Offer
>> Buy your Sophos next-gen firewall before the end March 2013
>> and get the hardware for free! Learn more.
>> http://p.sf.net/sfu/sophos-d2d-feb
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_feb
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>



-- 
StormDB - http://www.stormdb.com
The Database Cloud
Postgres-XC Support and Service

Hi,
Attached please find a patch that locks the cluster so that dump can be
taken to be restored on the new node to be added.

To lock the cluster the patch adds a new GUC parameter called
xc_lock_for_backup, however its status is maintained by the pooler. The
reason is that the default behavior of XC is to release connections as soon
as a command is done and it uses PersistentConnections GUC to control the
behavior. We in this case however need a status that is independent of the
setting of PersistentConnections.

Assume we have two coordinator cluster, the patch provides this behavior:

Case 1: set and show
====================
psql test -p 5432
set xc_lock_for_backup=yes;
show xc_lock_for_backup;
 xc_lock_for_backup
--------------------
 yes
(1 row)

Case 2: set from one client show from other
==================================
psql test -p 5432
set xc_lock_for_backup=yes;
(From another tab)
psql test -p 5432
show xc_lock_for_backup;
 xc_lock_for_backup
--------------------
 yes
(1 row)

Case 3: set from one, quit it, run again and show
======================================
psql test -p 5432
set xc_lock_for_backup=yes;
\q
psql test -p 5432
show xc_lock_for_backup;
 xc_lock_for_backup
--------------------
 yes
(1 row)

Case 4: set on one coordinator, show from other
=====================================
psql test -p 5432
set xc_lock_for_backup=yes;
(From another tab)
psql test -p 5433
show xc_lock_for_backup;
 xc_lock_for_backup
--------------------
 yes
(1 row)

pg_dump and pg_dumpall seem to work fine after locking the cluster for
backup but I would test these utilities in detail next.

Also I have yet to look in detail that standard_ProcessUtility is the only
place that updates the portion of catalog that is dumped. There may be some
other places too that need to be blocked for catalog updates.

The patch adds no extra warnings and regression shows no extra failure.

Comments are welcome.

--
Abbas
Architect
EnterpriseDB Corporation
The Enterprise PostgreSQL Company

Phone: 92-334-5100153

Website: www.enterprisedb.com
EnterpriseDB Blog: http://blogs.enterprisedb.com/
Follow us on Twitter: http://www.twitter.com/enterprisedb

This e-mail message (and any attachment) is intended for the use of
the individual or entity to whom it is addressed. This message
contains information from EnterpriseDB Corporation that may be
privileged, confidential, or exempt from disclosure under applicable
law. If you are not the intended recipient or authorized to receive
this for the intended recipient, any use, dissemination, distribution,
retention, archiving, or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify
the sender immediately by reply e-mail and delete this message.

I'm okay with this idea.

Any other inputs?
----------
Koichi Suzuki


2013/2/19 Xiong Wang <wan...@gm...>:
> Hi All，
>
> According to Ashutosh Bapat, could we just print ALL NODES if the count of
> nodelist is the same as the count of datanodes, otherwise print the node
> list. By this way, the regression test will be resolved.
>
> Thanks & Regards,
>
> Benny Wang
>
> 2013/2/15 Ashutosh Bapat <ash...@en...>
>>
>>
>>
>> On Fri, Feb 15, 2013 at 8:28 AM, Koichi Suzuki <koi...@gm...>
>> wrote:
>>>
>>> I found \d+ is used 41 times in the regression.   If we extend \d+
>>> with table distribution, we need to (continue to) modify all of them.
>>>  Is it too much?
>>>
>>
>> I don't see regression expected output changes to be of a lot of concern.
>> We can do them ones and for all. There might be some merge conflicts later,
>> but that's something, which would part of our ongoing merge. That's
>> inevitable for a project like XC, which pulls from PG.
>>
>> My concern is different. \d+ or \d whatever, would start printing the list
>> of nodes the table is distributed on, which will be dependent upon the
>> individual's enviroment, and names and number of nodes chosen. So, tests
>> passing for one would not do so for someone else. So, we will need some way
>> to canonicalize the list of nodes so that it comes out to be same
>> independent of the environment.
>>
>> I suggest this, instead of \d+ printing the individual nodes, it first
>> looks at the node groups for a matching node group and prints the name of
>> group if it finds one matching. Next, if all the datanodes are in the list
>> of nodes, it would print ALL DATANODES or something like that. Since the
>> regression tests use default node list, which has all datanodes, for almost
>> (I am sure all) all \d+ outputs we will get ALL DATANODES, which will be
>> independent of the environment used.
>>
>>>
>>> Regards;
>>> ----------
>>> Koichi Suzuki
>>>
>>>
>>> 2013/2/14 Michael Paquier <mic...@gm...>:
>>> >
>>> >
>>> > On Thu, Feb 14, 2013 at 7:44 PM, Nikhil Sontakke <ni...@st...>
>>> > wrote:
>>> >>
>>> >> Michael seems to be doing a see-saw here :)
>>> >
>>> > After doing manually regression merges for 9.1 and 9.2 on more than 50
>>> > regression tests for each merge, you learn quite a bit... Making it
>>> > more
>>> > painful will just consume more time than necessary.
>>> >
>>> >> On some other thread he seemed ok to extend \d+
>>> >
>>> > Call that experience, I was OK with with ¥d+ before actually having a
>>> > look
>>> > at the number of tests using it.
>>> > --
>>> > Michael
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Free Next-Gen Firewall Hardware Offer
>>> Buy your Sophos next-gen firewall before the end March 2013
>>> and get the hardware for free! Learn more.
>>> http://p.sf.net/sfu/sophos-d2d-feb
>>> _______________________________________________
>>> Postgres-xc-developers mailing list
>>> Pos...@li...
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>>
>>
>>
>> --
>> Best Wishes,
>> Ashutosh Bapat
>> EntepriseDB Corporation
>> The Enterprise Postgres Company
>>
>>
>> ------------------------------------------------------------------------------
>> Free Next-Gen Firewall Hardware Offer
>> Buy your Sophos next-gen firewall before the end March 2013
>> and get the hardware for free! Learn more.
>> http://p.sf.net/sfu/sophos-d2d-feb
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_feb
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>

>>
>> I think the barrier makes sense when doing a barrier with the rest of the
>> cluster components. Then each element is recoverable to the same point in
>> time.

Oh yes. I did not mean that BARRIER activity should not track GTM
stuff. I only meant that GTM behavior should be done independent of
BARRIER calls as well.

>
> Backing up gtm.control triggerd by BARRIER is not associated with WAL
> at all.   My idea is
> 1) add new message to GTM to report BARRIER is going to be recorded,
> with barrier ID.
> 2) When gtm (and gtm slave) receives this message, gtm/gtm slave
> creates gtm.control backup.
>
> Important thing is to synchronize PITR recovery point with gtm.control.
>

Yeah, +1 for this approach to integrate GTM with BARRIER activity.

>> Most important is XID. With a valid XID, we can get the current sequence
>> value from the individual nodes.
>
> Well, this will save some value consumed by aborted TXs.   This is
> wasted in vanilla PostgreSQL too and it does not violate sequence
> consistency.
>

Yeah, this loose requirement also makes our life easier. We can just
append an appropriate delta to get the next "sane" value for both GXID
and sequences. Unless again if I am mistaken about the semantics or
something.

Regards,
Nikhils


>>
>> We could consider optionally that when a data node connects to GTM that GTM
>> acknowledges and then requests back all of the sequence values from the
>> node. Then GTM can take the maximum from all the nodes to be able to
>> determine what the current value is cluster-wide. No need for an external
>> tool. For the nodes, we can implement it as  a new stored function.
>
> I don't think we need this now.
>
> As Mason wrote, yes we have to consider GXID wraparound.
>
> Regards;
> ---
> Koichi
>
>>
>> Regards,
>>
>> Mason
>>
>>
>>>
>>>
>>> Regards,
>>> Nikhils
>>>
>>>
>>> >
>>> >>
>>> >>
>>> >> >
>>> >> > Thoughts? Any other better suggestions?
>>> >>
>>> >> Regards;
>>> >> ---
>>> >> Koichi Suzuki
>>> >>
>>> >> >
>>> >> > Regards,
>>> >> > Nikhils
>>> >> >
>>> >> >
>>> >> >> 1) If we kill GTM process, GTM will begin to assing GXID values with
>>> >> >> the one of the proveious GTM run.  It is bad and harmful behaviour.
>>> >> >> To
>>> >> >> improve it, GTM should write the next gxid value to gtm.control.
>>> >> >> When gxid
>>> >> >> value reaches this backup, then the backup should be refreshed with
>>> >> >> the new
>>> >> >> one.   That way, we can safely restart GTM even if it was killed.
>>> >> >> When GTM
>>> >> >> stops normally, it should write the next gxid value to gtm.control.
>>> >> >>
>>> >> >> 2) We have BARRIER to synchronize PITR recovery line.   However, GTM
>>> >> >> does not have this capability.  In this case, GTM's sequence
>>> >> >> definition
>>> >> >> could be inconsisitent if ALTER SEQUENCE or DROP SEQUENCE was issued
>>> >> >> after
>>> >> >> the recovery point.   Have WAL in GTM could be a general choice but
>>> >> >> it seems
>>> >> >> to be too much for current GTM feature.  Instead, we can have
>>> >> >> GTM.control
>>> >> >> backups when CREATE BARRIER is issued.   Backup files will be
>>> >> >> qualified by
>>> >> >> BARRIER id so that DBA can choose which backup to use at the end of
>>> >> >> PITR.
>>> >> >>
>>> >> >> Regards;
>>> >> >> ---
>>> >> >> Koichi
>>> >> >>
>>> >> >> On Sun, 17 Feb 2013 10:27:37 +0530
>>> >> >> Nikhil Sontakke <ni...@st...> wrote:
>>> >> >>
>>> >> >>> Hi Suzuki-san,
>>> >> >>>
>>> >> >>> > It's so helpful to have this patch.   Will review and test it
>>> >> >>> > before
>>> >> >>> > committing.
>>> >> >>> >
>>> >> >>>
>>> >> >>> Any comments on this patch? I do not see it in the commit logs.
>>> >> >>>
>>> >> >>> Regards,
>>> >> >>> Nikhils
>>> >> >>>
>>> >> >>> > Best;
>>> >> >>> > ---
>>> >> >>> > Koichi Suzuki
>>> >> >>> >
>>> >> >>> > On Wed, 6 Feb 2013 14:25:48 +0530
>>> >> >>> > Nikhil Sontakke <ni...@st...> wrote:
>>> >> >>> >
>>> >> >>> >> > Also, gtm_ctl -w option
>>> >> >>> >> > seems not work properly.
>>> >> >>> >> >
>>> >> >>> >>
>>> >> >>> >> PFA, a patch to fix "gtm_ctl -w" behavior in GIT HEAD. I also
>>> >> >>> >> tested
>>> >> >>> >> "-w -t nsecs" behavior and it seems to work as well. This patch
>>> >> >>> >> can
>>> >> >>> >> be
>>> >> >>> >> easily be backported to 1.0 if desired. I hope all these
>>> >> >>> >> obnoxious
>>> >> >>> >> gtm
>>> >> >>> >> startup issues are resolved now. Scripting was a bit painful
>>> >> >>> >> because
>>> >> >>> >> of these issues.
>>> >> >>> >>
>>> >> >>> >> Regards,
>>> >> >>> >> Nikhils
>>> >> >>> >>
>>> >> >>> >> > I will take a look at these issue and fix at least in the next
>>> >> >>> >> > major
>>> >> >>> >> > release.    Daemonization should be back-ported to 1.0.x.
>>> >> >>> >> >
>>> >> >>> >> > Kind Regards;
>>> >> >>> >> > ----------
>>> >> >>> >> > Koichi Suzuki
>>> >> >>> >> >
>>> >> >>> >> >
>>> >> >>> >> > 2012/12/27 Michael Meskes <me...@po...>:
>>> >> >>> >> >> I still haven't found the time to dig into this, but could
>>> >> >>> >> >> anyone please
>>> >> >>> >> >> explain to me how gtm is supposed to start up? A simple grep
>>> >> >>> >> >> command doesn't
>>> >> >>> >> >> seem to find any call to setsid() in gtm subdirs. Could you
>>> >> >>> >> >> please point me to
>>> >> >>> >> >> the code where gtm daemonizes? Because if it doesn't
>>> >> >>> >> >> correctly,
>>> >> >>> >> >> there is no
>>> >> >>> >> >> suprise gtm_ctl doesn't come back. Or in other words, this is
>>> >> >>> >> >> a
>>> >> >>> >> >> logical point
>>> >> >>> >> >> to start looking.
>>> >> >>> >> >>
>>> >> >>> >> >> Michael
>>> >> >>> >> >> --
>>> >> >>> >> >> Michael Meskes
>>> >> >>> >> >> Michael at Fam-Meskes dot De, Michael at Meskes dot
>>> >> >>> >> >> (De|Com|Net|Org)
>>> >> >>> >> >> Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql)
>>> >> >>> >> >> dot
>>> >> >>> >> >> Org
>>> >> >>> >> >> Jabber: michael.meskes at gmail dot com
>>> >> >>> >> >> VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux,
>>> >> >>> >> >> PostgreSQL
>>> >> >>> >> >>
>>> >> >>> >> >>
>>> >> >>> >> >>
>>> >> >>> >> >> ------------------------------------------------------------------------------
>>> >> >>> >> >> Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012,
>>> >> >>> >> >> HTML5,
>>> >> >>> >> >> CSS,
>>> >> >>> >> >> MVC, Windows 8 Apps, JavaScript and much more. Keep your
>>> >> >>> >> >> skills
>>> >> >>> >> >> current
>>> >> >>> >> >> with LearnDevNow - 3,200 step-by-step video tutorials by
>>> >> >>> >> >> Microsoft
>>> >> >>> >> >> MVPs and experts. ON SALE this month only -- learn more at:
>>> >> >>> >> >> http://p.sf.net/sfu/learnmore_122712
>>> >> >>> >> >> _______________________________________________
>>> >> >>> >> >> Postgres-xc-developers mailing list
>>> >> >>> >> >> Pos...@li...
>>> >> >>> >> >>
>>> >> >>> >> >>
>>> >> >>> >> >> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>> >> >>> >>
>>> >> >>> >>
>>> >> >>> >>
>>> >> >>> >> --
>>> >> >>> >> StormDB - http://www.stormdb.com
>>> >> >>> >> The Database Cloud
>>> >> >>> >> Postgres-XC Support and Service
>>> >> >>>
>>> >> >>>
>>> >> >>>
>>> >> >>> --
>>> >> >>> StormDB - http://www.stormdb.com
>>> >> >>> The Database Cloud
>>> >> >>> Postgres-XC Support and Service
>>> >> >>>
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> > StormDB - http://www.stormdb.com
>>> >> > The Database Cloud
>>> >> > Postgres-XC Support and Service
>>> >>
>>> >>
>>> >>
>>> >> ------------------------------------------------------------------------------
>>> >> The Go Parallel Website, sponsored by Intel - in partnership with
>>> >> Geeknet,
>>> >> is your hub for all things parallel software development, from weekly
>>> >> thought
>>> >> leadership blogs to news, videos, case studies, tutorials, tech docs,
>>> >> whitepapers, evaluation guides, and opinion stories. Check out the most
>>> >> recent posts - join the conversation now.
>>> >> http://goparallel.sourceforge.net/
>>> >>
>>> >> _______________________________________________
>>> >> Postgres-xc-developers mailing list
>>> >> Pos...@li...
>>> >> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Mason Sharp
>>> >
>>> >
>>> > StormDB - http://www.stormdb.com
>>> > The Database Cloud
>>> > Postgres-XC Support and Services
>>>
>>>
>>>
>>> --
>>> StormDB - http://www.stormdb.com
>>> The Database Cloud
>>> Postgres-XC Support and Service
>>
>>
>>
>>
>> --
>> Mason Sharp
>>
>> StormDB - http://www.stormdb.com
>> The Database Cloud
>> Postgres-XC Support and Services



-- 
StormDB - http://www.stormdb.com
The Database Cloud
Postgres-XC Support and Service

Below are details that cover various points that  need to be
considered for the implemention of BEFORE UPDATE and AFTER UPDATE row
triggers. Because UPDATE covers issues that might occur for both
DELETE as well as INSERT, I have targetted UPDATE first. Attached is
the WIP patch if you need to peek into specifics.

AFTER triggers
--------------

AFTER triggers are executed only at the end of statement/transaction.
The trigger data is saved in a trigger event queue along with the OLD
and NEW ctid. And just before the query/transaction end, the OLD and
NEW tuples corresponding to the OLD and NEW ctid are fetched using
heap_fetch(SnapshotAny), and each of the saved trigger events are
executed. For XC (in the patch), we fetch the tuple remotely from the
datanode. And, in addition to ctid, we store the xc_node_id as well in
the trigger event data. Created a function to do the remote fetch;
this function would be called instead of heap_fetch().

To get NEW row ctid, we add junk attributes ctid and node_id in the
RETURNING target list of UPDATE.
To get OLD row ctid, we already have the ctid in the SELECT subplan.

Currently we fetch all attributes in the SELECT subplans. I have
created another patch to fetch only the required attribtues, but have
not merged that into this patch. Ultimately, when triggers are
present, we want the SELECT to look like this:
SELECT ctid, xc_node_id from tab1

Another way to do the AFTER ROW triggers would be to store all the new
and old rows in the coordinator memory instead of fetching it when
needed. This has a potential of disk writes for large number of update
rows.

When there is a WHEN condition on OLD and NEW row for AFTER triggers,
the WHEN condition is evaluated immediately, and is not deferred until
the query/transaction end, and all the triggers that are evaluated to
false are not added in the event queue. So if WHEN is present, we need
to make it available by fetching the OLD row using SELECT sublan. For
this we add another junk attribute wholerow in the SELECT target list.
So in this case, the SELECT will look like this:
SELECT ctid, xc_node_id, tab1.*::tab1 from tab1



BEFORE triggers
---------------

These are executed just before the row is inserted/updated. There can
be multiple triggers on the same table. The return ROW of a trigger
function is the NEW row of the next trigger, while the OLD row remains
same across all of the triggers. So the NEW row can keep changing
because the trigger function can modify the NEW row before returning.
Ultimately the last NEW row is the one that gets into the table.

We need to add wholerow junk attribute in SELECT for fetching the OLD
row. And the NEW row is created by overwriting the OLD row values by
the modified column values. Once we have OLD and NEW tuples, the rest
of the work is taken care of usign existing trigger PG implementation.

We have been using the subplan slot to generate the data for BINDing
the parameter values for UPDATE. Now for BEFORE ROW triggers, the data
of NEW row is the one that is to be BOUND. And this NEW tuple slot
does not have the ctid and node_id junk attribute values. So we need
to generate the BIND data using the data from NEW row and the junk
attributes from the source data.
Similarly there are changes in the way we generate the parameters
markers for the remote UPDATE statement.


TODOs:
======

1.  Current implementation patch has a very basic working model. There
are various TODOs marked in the patch.

2.  One important TODO for BEFORE trigger is this: Just before
invoking the trigger functions, in PG, the tuple is row-locked
(exclusive) by GetTupleTrigger() and the locked version is fetched
from the table. So it is made sure that while all the triggers for
that table are executed, no one can update that particular row.
In the patch, we haven't locked the row. We need to lock it either by
executing :
   1. SELECT * from tab1 where ctid = <ctid_val> FOR UPDATE, and then
use the returned ROW as the OLD row.
OR
   2. The UPDATE subplan itself should have SELECT for UPDATE so that
the row is already locked, and we don't have to lock it again.
#2 is simple though it might cause some amount of longer waits in general.
Using #1, though the locks would be acquired only when the particular
row is updated, the locks would be released only after transaction
end, so #1 might not be worth implementing.
Also #1 requires another explicit remote fetch for the
lock-and-get-latest-version operation.
I am more inclined towards #2.

3.  The BEFORE trigger function can change the distribution column
itself. We need to add a check at the end of the trigger executions.

4. Fetching OLD row for WHEN clause handling.

5. Testing with mix of Shippable and non-shippable ROW triggers

6. Other types of triggers. INSTEAD triggers are anticipated to work
without significant changes, but they are yet to be tested.
INSERT/DELETE triggers: Most of the infrastructure has been done while
implementing UPDATE triggers. But some changes specific to INSERT and
DELETE are yet to be done.
Deferred triggers to be tested.

7. Regression analysis. There are some new failures. Will post another
fair version of the patch after regression analysis and fixing various
TODOs.

Comments welcome.

Hi All，

According to Ashutosh Bapat, could we just print ALL NODES if the count of
nodelist is the same as the count of datanodes, otherwise print the node
list. By this way, the regression test will be resolved.

Thanks & Regards,

Benny Wang

2013/2/15 Ashutosh Bapat <ash...@en...>

>
>
> On Fri, Feb 15, 2013 at 8:28 AM, Koichi Suzuki <koi...@gm...>wrote:
>
>> I found \d+ is used 41 times in the regression.   If we extend \d+
>> with table distribution, we need to (continue to) modify all of them.
>>  Is it too much?
>>
>>
> I don't see regression expected output changes to be of a lot of concern.
> We can do them ones and for all. There might be some merge conflicts later,
> but that's something, which would part of our ongoing merge. That's
> inevitable for a project like XC, which pulls from PG.
>
> My concern is different. \d+ or \d whatever, would start printing the list
> of nodes the table is distributed on, which will be dependent upon the
> individual's enviroment, and names and number of nodes chosen. So, tests
> passing for one would not do so for someone else. So, we will need some way
> to canonicalize the list of nodes so that it comes out to be same
> independent of the environment.
>
> I suggest this, instead of \d+ printing the individual nodes, it first
> looks at the node groups for a matching node group and prints the name of
> group if it finds one matching. Next, if all the datanodes are in the list
> of nodes, it would print ALL DATANODES or something like that. Since the
> regression tests use default node list, which has all datanodes, for almost
> (I am sure all) all \d+ outputs we will get ALL DATANODES, which will be
> independent of the environment used.
>
>
>> Regards;
>> ----------
>> Koichi Suzuki
>>
>>
>> 2013/2/14 Michael Paquier <mic...@gm...>:
>> >
>> >
>> > On Thu, Feb 14, 2013 at 7:44 PM, Nikhil Sontakke <ni...@st...>
>> > wrote:
>> >>
>> >> Michael seems to be doing a see-saw here :)
>> >
>> > After doing manually regression merges for 9.1 and 9.2 on more than 50
>> > regression tests for each merge, you learn quite a bit... Making it more
>> > painful will just consume more time than necessary.
>> >
>> >> On some other thread he seemed ok to extend \d+
>> >
>> > Call that experience, I was OK with with ¥d+ before actually having a
>> look
>> > at the number of tests using it.
>> > --
>> > Michael
>>
>>
>> ------------------------------------------------------------------------------
>> Free Next-Gen Firewall Hardware Offer
>> Buy your Sophos next-gen firewall before the end March 2013
>> and get the hardware for free! Learn more.
>> http://p.sf.net/sfu/sophos-d2d-feb
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Enterprise Postgres Company
>
>
> ------------------------------------------------------------------------------
> Free Next-Gen Firewall Hardware Offer
> Buy your Sophos next-gen firewall before the end March 2013
> and get the hardware for free! Learn more.
> http://p.sf.net/sfu/sophos-d2d-feb
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>

2013/2/19 Mason Sharp <ma...@st...>:
>
>
> On Mon, Feb 18, 2013 at 11:53 AM, Nikhil Sontakke <ni...@st...>
> wrote:
>>
>> >
>> >
>> > Yes, sequences are trickier. A single transaction can increment multiple
>> > sequences. Barrier seems like one good way to do this periodically, but
>> > we
>> > may be missing info and may have to query the individual data nodes in
>> > order
>> > to get the actual latest sequence values.
>> >
>>
>> I am not in favor of relying on barrier for GTM stuff. Just
>> implementing currval+delta for both XID and sequences seems OK enough
>> to me and it keeps things within the GTM scheme of things.
>
>
> I think the barrier makes sense when doing a barrier with the rest of the
> cluster components. Then each element is recoverable to the same point in
> time.

Backing up gtm.control triggerd by BARRIER is not associated with WAL
at all.   My idea is
1) add new message to GTM to report BARRIER is going to be recorded,
with barrier ID.
2) When gtm (and gtm slave) receives this message, gtm/gtm slave
creates gtm.control backup.

Important thing is to synchronize PITR recovery point with gtm.control.

>
>>
>>
>> > We may instead also consider a WAL-type of mechanism where we write the
>> > latest sequence values that a transaction updated at transaction commit.
>> > With GTM Proxy grouping requests together, too, and by grouping from the
>> > other threads, maybe a single GTM WAL fsync can write out quite a lot of
>> > information.
>> >
>>
>> IMHO, WAL types implementation seems overkill to me. Configurable
>> delta based logic to write to control file can whet everyone's
>> appetite.

Same to me.

>
>
> Most important is XID. With a valid XID, we can get the current sequence
> value from the individual nodes.

Well, this will save some value consumed by aborted TXs.   This is
wasted in vanilla PostgreSQL too and it does not violate sequence
consistency.

>
> We could consider optionally that when a data node connects to GTM that GTM
> acknowledges and then requests back all of the sequence values from the
> node. Then GTM can take the maximum from all the nodes to be able to
> determine what the current value is cluster-wide. No need for an external
> tool. For the nodes, we can implement it as  a new stored function.

I don't think we need this now.

As Mason wrote, yes we have to consider GXID wraparound.

Regards;
---
Koichi

>
> Regards,
>
> Mason
>
>
>>
>>
>> Regards,
>> Nikhils
>>
>>
>> >
>> >>
>> >>
>> >> >
>> >> > Thoughts? Any other better suggestions?
>> >>
>> >> Regards;
>> >> ---
>> >> Koichi Suzuki
>> >>
>> >> >
>> >> > Regards,
>> >> > Nikhils
>> >> >
>> >> >
>> >> >> 1) If we kill GTM process, GTM will begin to assing GXID values with
>> >> >> the one of the proveious GTM run.  It is bad and harmful behaviour.
>> >> >> To
>> >> >> improve it, GTM should write the next gxid value to gtm.control.
>> >> >> When gxid
>> >> >> value reaches this backup, then the backup should be refreshed with
>> >> >> the new
>> >> >> one.   That way, we can safely restart GTM even if it was killed.
>> >> >> When GTM
>> >> >> stops normally, it should write the next gxid value to gtm.control.
>> >> >>
>> >> >> 2) We have BARRIER to synchronize PITR recovery line.   However, GTM
>> >> >> does not have this capability.  In this case, GTM's sequence
>> >> >> definition
>> >> >> could be inconsisitent if ALTER SEQUENCE or DROP SEQUENCE was issued
>> >> >> after
>> >> >> the recovery point.   Have WAL in GTM could be a general choice but
>> >> >> it seems
>> >> >> to be too much for current GTM feature.  Instead, we can have
>> >> >> GTM.control
>> >> >> backups when CREATE BARRIER is issued.   Backup files will be
>> >> >> qualified by
>> >> >> BARRIER id so that DBA can choose which backup to use at the end of
>> >> >> PITR.
>> >> >>
>> >> >> Regards;
>> >> >> ---
>> >> >> Koichi
>> >> >>
>> >> >> On Sun, 17 Feb 2013 10:27:37 +0530
>> >> >> Nikhil Sontakke <ni...@st...> wrote:
>> >> >>
>> >> >>> Hi Suzuki-san,
>> >> >>>
>> >> >>> > It's so helpful to have this patch.   Will review and test it
>> >> >>> > before
>> >> >>> > committing.
>> >> >>> >
>> >> >>>
>> >> >>> Any comments on this patch? I do not see it in the commit logs.
>> >> >>>
>> >> >>> Regards,
>> >> >>> Nikhils
>> >> >>>
>> >> >>> > Best;
>> >> >>> > ---
>> >> >>> > Koichi Suzuki
>> >> >>> >
>> >> >>> > On Wed, 6 Feb 2013 14:25:48 +0530
>> >> >>> > Nikhil Sontakke <ni...@st...> wrote:
>> >> >>> >
>> >> >>> >> > Also, gtm_ctl -w option
>> >> >>> >> > seems not work properly.
>> >> >>> >> >
>> >> >>> >>
>> >> >>> >> PFA, a patch to fix "gtm_ctl -w" behavior in GIT HEAD. I also
>> >> >>> >> tested
>> >> >>> >> "-w -t nsecs" behavior and it seems to work as well. This patch
>> >> >>> >> can
>> >> >>> >> be
>> >> >>> >> easily be backported to 1.0 if desired. I hope all these
>> >> >>> >> obnoxious
>> >> >>> >> gtm
>> >> >>> >> startup issues are resolved now. Scripting was a bit painful
>> >> >>> >> because
>> >> >>> >> of these issues.
>> >> >>> >>
>> >> >>> >> Regards,
>> >> >>> >> Nikhils
>> >> >>> >>
>> >> >>> >> > I will take a look at these issue and fix at least in the next
>> >> >>> >> > major
>> >> >>> >> > release.    Daemonization should be back-ported to 1.0.x.
>> >> >>> >> >
>> >> >>> >> > Kind Regards;
>> >> >>> >> > ----------
>> >> >>> >> > Koichi Suzuki
>> >> >>> >> >
>> >> >>> >> >
>> >> >>> >> > 2012/12/27 Michael Meskes <me...@po...>:
>> >> >>> >> >> I still haven't found the time to dig into this, but could
>> >> >>> >> >> anyone please
>> >> >>> >> >> explain to me how gtm is supposed to start up? A simple grep
>> >> >>> >> >> command doesn't
>> >> >>> >> >> seem to find any call to setsid() in gtm subdirs. Could you
>> >> >>> >> >> please point me to
>> >> >>> >> >> the code where gtm daemonizes? Because if it doesn't
>> >> >>> >> >> correctly,
>> >> >>> >> >> there is no
>> >> >>> >> >> suprise gtm_ctl doesn't come back. Or in other words, this is
>> >> >>> >> >> a
>> >> >>> >> >> logical point
>> >> >>> >> >> to start looking.
>> >> >>> >> >>
>> >> >>> >> >> Michael
>> >> >>> >> >> --
>> >> >>> >> >> Michael Meskes
>> >> >>> >> >> Michael at Fam-Meskes dot De, Michael at Meskes dot
>> >> >>> >> >> (De|Com|Net|Org)
>> >> >>> >> >> Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql)
>> >> >>> >> >> dot
>> >> >>> >> >> Org
>> >> >>> >> >> Jabber: michael.meskes at gmail dot com
>> >> >>> >> >> VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux,
>> >> >>> >> >> PostgreSQL
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >> ------------------------------------------------------------------------------
>> >> >>> >> >> Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012,
>> >> >>> >> >> HTML5,
>> >> >>> >> >> CSS,
>> >> >>> >> >> MVC, Windows 8 Apps, JavaScript and much more. Keep your
>> >> >>> >> >> skills
>> >> >>> >> >> current
>> >> >>> >> >> with LearnDevNow - 3,200 step-by-step video tutorials by
>> >> >>> >> >> Microsoft
>> >> >>> >> >> MVPs and experts. ON SALE this month only -- learn more at:
>> >> >>> >> >> http://p.sf.net/sfu/learnmore_122712
>> >> >>> >> >> _______________________________________________
>> >> >>> >> >> Postgres-xc-developers mailing list
>> >> >>> >> >> Pos...@li...
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>> >> >>> >>
>> >> >>> >>
>> >> >>> >>
>> >> >>> >> --
>> >> >>> >> StormDB - http://www.stormdb.com
>> >> >>> >> The Database Cloud
>> >> >>> >> Postgres-XC Support and Service
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> --
>> >> >>> StormDB - http://www.stormdb.com
>> >> >>> The Database Cloud
>> >> >>> Postgres-XC Support and Service
>> >> >>>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > StormDB - http://www.stormdb.com
>> >> > The Database Cloud
>> >> > Postgres-XC Support and Service
>> >>
>> >>
>> >>
>> >> ------------------------------------------------------------------------------
>> >> The Go Parallel Website, sponsored by Intel - in partnership with
>> >> Geeknet,
>> >> is your hub for all things parallel software development, from weekly
>> >> thought
>> >> leadership blogs to news, videos, case studies, tutorials, tech docs,
>> >> whitepapers, evaluation guides, and opinion stories. Check out the most
>> >> recent posts - join the conversation now.
>> >> http://goparallel.sourceforge.net/
>> >>
>> >> _______________________________________________
>> >> Postgres-xc-developers mailing list
>> >> Pos...@li...
>> >> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>> >
>> >
>> >
>> >
>> > --
>> > Mason Sharp
>> >
>> >
>> > StormDB - http://www.stormdb.com
>> > The Database Cloud
>> > Postgres-XC Support and Services
>>
>>
>>
>> --
>> StormDB - http://www.stormdb.com
>> The Database Cloud
>> Postgres-XC Support and Service
>
>
>
>
> --
> Mason Sharp
>
> StormDB - http://www.stormdb.com
> The Database Cloud
> Postgres-XC Support and Services