postgres-xc-general Mailing List for Postgres-XC

As to StormDB, yes, there was a news that TransLattice bought StormDB.

Though, I found StormDB's site still valid so I wrote this way.

Hope somebody points out which I should refer to, StormDB or Translattice.

Regards;
---
Koichi Suzuki


2014-05-04 0:59 GMT+09:00 Dorian Hoxha <dor...@gm...>:
>> You just need commodity INTEL server runnign Linux.
>
> Are INTEL cpu required ? If not INTEL can be removed ? (also running typo)
>
>> For data­warehouse
>>
>> applications, you may need separate patch which devides complexed query
>> into smaller
>>
>> chunks which run in datanodes in parallel.    StormDB will provide such
>> patche.
>
> Wasn't stormdb bought by another company ? Is there an opensource
> alternative ? Fix the "patche" typo ?
>
> A way to make it simpler is by merging coordinator and datanode into 1 and
> making it possible for a 'node' to not hold data (be a coordinator only),
> like in elastic-search, but you probably already know that.
>
> What exact things does the gtm-proxy do? For example, a single row insert
> wouldn't need the gtm (coordinator just inserts it to the right
> data-node)(asumming no sequences, since for that the gtm is needed)?
>
> If multiple tables are sharded on the same key (example: user_id). Will all
> the rows, from the same user in different tables be in the same data-node ?
>
>
>
>
>
>
>
>
> On Fri, May 2, 2014 at 10:43 AM, Koichi Suzuki <koi...@gm...>
> wrote:
>>
>> Hello;
>>
>> Because there are many discussions here on Postgres-XC configuration
>> and operation, I drafted a paper "Pgcx_ctl primer" to provide how to
>> configure Postgres-XC cluster and how to operate it.
>>
>> https://sourceforge.net/projects/postgres-xc/files/Pgxc_ctl_primer/?
>> contains the document.
>>
>> Hope to have feedback on it.
>>
>> Regards;
>> ---
>> Koichi Suzuki
>>
>>
>> ------------------------------------------------------------------------------
>> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
>> Instantly run your Selenium tests across 300+ browser/OS combos.  Get
>> unparalleled scalability from the best Selenium testing platform
>> available.
>> Simple to use. Nothing to install. Get started now for free."
>> http://p.sf.net/sfu/SauceLabs
>> _______________________________________________
>> Postgres-xc-general mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
>

On Sun, May 4, 2014 at 7:05 PM, Dorian Hoxha <dor...@gm...> wrote:
> Probably even the gtm-proxy need to be merged with datanode+coordinator from
> what i read.
>
> If you make only local transactions (inside 1 datanode) + not using global
> sequences, will there be no traffic to the GTM for that transaction ?
You are forgetting global transaction ID, global snapshot and global
timestamp feeding across nodes to satisfy MVCC :)
-- 
Michael

Did you make any test before you did any action on your database?
Also could you share what you did to have this situation?   Anyway,
with this situation, I believe no DDL has been handled except for
temporary object, which is session specific.

So I believe you can restart GTM proxy gtm_pxy2  (you don't have to
reinitialize it as long as you maintain gtm_proxy.conf) and coord2.

Anyway, it is very essential to record and share what you did, and
more important thing is to test it with non-product environment and
see what goes on, and then review any step you are taking before you
do.

Hope to have more info on this.

Best Regards;
---
Koichi Suzuki


2014-05-03 14:31 GMT+09:00 Juned Khan <jkh...@gm...>:
> Hi Koichi,
>
> I tried to follow the steps removing datanode slave and adding it again but
> i no success. even earlier only datanode slave was not working but now gtm
> slave, gtm_pxy2, coord2 and dn_slave is not starting up. I have removed GTM
> slave and datanode slave
>
> here is my current status of pgxc
>
> PGXC monitor all
> Running: gtm master
> Running: gtm proxy gtm_pxy1
> Not running: gtm proxy gtm_pxy2
> Running: coordinator master coord1
> Not running: coordinator master coord2
> Running: datanode master datanode1
>
> Now each time i have to connect on db02 only( i.e PGXC Psql -h db02 -d
> mydatabase) but its not allowing me to modify table structure. and giving me
> below error.
>
> mydatabase=#  alter table invoice_summary_data add countrycode text not null
> default ''::character(1);
> ERROR:  Failed to get pooled connections
> CONTEXT:  SQL statement "EXECUTE DIRECT ON (coord2) 'SELECT
> pg_catalog.pg_try_advisory_xact_lock_shared(65535, 0)'"
>
> So as of now i am planning to remove those components which are not working
> (gtm_pxy2,coord2 ). so it will not go to connect cood2 which is down.
>
> what will be the impact of this, i don't want to loost database access
> completely.
>
> Please suggest.
>
> Regards
> Juned Khan
>
> ------------------------------------------------------------------------------
> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> Instantly run your Selenium tests across 300+ browser/OS combos.  Get
> unparalleled scalability from the best Selenium testing platform available.
> Simple to use. Nothing to install. Get started now for free."
> http://p.sf.net/sfu/SauceLabs
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>

As discussed in the last year's XC-day, GTM proxy should be integrated
as postmaster backend.     Maybe GTM can be.     Coordinator/Datanode
can also be integrated into one.

Apparently, this is the direction we should take.    At first, there
were no such good experience to start with.    Before version 1.0, we
determined that the datanode and the coordinator can share the same
binary.       It is true that we started with the idea to provide
cluster-wide MVCC and now we found the next direction.

With this integration and when start with only one node, we don't need
GTM, which looks identical to standalone PG.   When we add the server,
at present we do need GTM.   Only accumulating local transactions in
the nodes cannot maintain cluster-wide database consistency.

I'm still investigating an idea how to get rid of GTM.   We need to do
the following:

1) To provide cluster wide MVCC,
2) To provide good means to determine which row can be vacuumed.

My current idea is: if we associate any local XID to the root
transaction (the transaction which application created), we may be
able to provide cluster wide MVCC by calculating cluster-wide snapshot
when needed.   I don't know how efficient it is and t don't have good
idea how to determine if a given row can be vacuumed.

This is the current situation.

Hope to have much more input on this.

Anyway, hope my draft helps people who is trying to use Postgres-XC.

Best;
---
Koichi Suzuki


2014-05-04 19:05 GMT+09:00 Dorian Hoxha <dor...@gm...>:
> Probably even the gtm-proxy need to be merged with datanode+coordinator from
> what i read.
>
> If you make only local transactions (inside 1 datanode) + not using global
> sequences, will there be no traffic to the GTM for that transaction ?
>
>
> On Sun, May 4, 2014 at 6:24 AM, Michael Paquier <mic...@gm...>
> wrote:
>>
>> On Sun, May 4, 2014 at 12:59 AM, Dorian Hoxha <dor...@gm...>
>> wrote:
>> >> You just need commodity INTEL server runnign Linux.
>> > Are INTEL cpu required ? If not INTEL can be removed ? (also running
>> > typo)
>> Not really... I agree to what you mean here.
>>
>> >> For data­warehouse
>> >>
>> >> applications, you may need separate patch which devides complexed query
>> >> into smaller
>> >>
>> >> chunks which run in datanodes in parallel.    StormDB will provide such
>> >> patche.
>> >
>> > Wasn't stormdb bought by another company ? Is there an opensource
>> > alternative ? Fix the "patche" typo ?
>> >
>> > A way to make it simpler is by merging coordinator and datanode into 1
>> > and
>> > making it possible for a 'node' to not hold data (be a coordinator
>> > only),
>> > like in elastic-search, but you probably already know that.
>> +1. This would alleviate data transfer between cross-node joins where
>> Coordinator and Datanodes are on separate servers. You could always
>> have both nodes on the same server with the XC of now... But that's
>> double number of nodes to monitor.
>>
>> > What exact things does the gtm-proxy do? For example, a single row
>> > insert
>> > wouldn't need the gtm (coordinator just inserts it to the right
>> > data-node)(asumming no sequences, since for that the gtm is needed)?
>> Grouping messages between Coordinator/Datanode and GTM to reduce
>> package interferences and improve performance.
>>
>> > If multiple tables are sharded on the same key (example: user_id). Will
>> > all
>> > the rows, from the same user in different tables be in the same
>> > data-node ?
>> Yep. Node choice algorithm is based using the data type of the key.
>> --
>> Michael
>
>

Probably even the gtm-proxy need to be merged with datanode+coordinator
from what i read.

If you make only local transactions (inside 1 datanode) + not using global
sequences, will there be no traffic to the GTM for that transaction ?


On Sun, May 4, 2014 at 6:24 AM, Michael Paquier
<mic...@gm...>wrote:

> On Sun, May 4, 2014 at 12:59 AM, Dorian Hoxha <dor...@gm...>
> wrote:
> >> You just need commodity INTEL server runnign Linux.
> > Are INTEL cpu required ? If not INTEL can be removed ? (also running
> typo)
> Not really... I agree to what you mean here.
>
> >> For data­warehouse
> >>
> >> applications, you may need separate patch which devides complexed query
> >> into smaller
> >>
> >> chunks which run in datanodes in parallel.    StormDB will provide such
> >> patche.
> >
> > Wasn't stormdb bought by another company ? Is there an opensource
> > alternative ? Fix the "patche" typo ?
> >
> > A way to make it simpler is by merging coordinator and datanode into 1
> and
> > making it possible for a 'node' to not hold data (be a coordinator only),
> > like in elastic-search, but you probably already know that.
> +1. This would alleviate data transfer between cross-node joins where
> Coordinator and Datanodes are on separate servers. You could always
> have both nodes on the same server with the XC of now... But that's
> double number of nodes to monitor.
>
> > What exact things does the gtm-proxy do? For example, a single row insert
> > wouldn't need the gtm (coordinator just inserts it to the right
> > data-node)(asumming no sequences, since for that the gtm is needed)?
> Grouping messages between Coordinator/Datanode and GTM to reduce
> package interferences and improve performance.
>
> > If multiple tables are sharded on the same key (example: user_id). Will
> all
> > the rows, from the same user in different tables be in the same
> data-node ?
> Yep. Node choice algorithm is based using the data type of the key.
> --
> Michael
>

On Sun, May 4, 2014 at 12:59 AM, Dorian Hoxha <dor...@gm...> wrote:
>> You just need commodity INTEL server runnign Linux.
> Are INTEL cpu required ? If not INTEL can be removed ? (also running typo)
Not really... I agree to what you mean here.

>> For data­warehouse
>>
>> applications, you may need separate patch which devides complexed query
>> into smaller
>>
>> chunks which run in datanodes in parallel.    StormDB will provide such
>> patche.
>
> Wasn't stormdb bought by another company ? Is there an opensource
> alternative ? Fix the "patche" typo ?
>
> A way to make it simpler is by merging coordinator and datanode into 1 and
> making it possible for a 'node' to not hold data (be a coordinator only),
> like in elastic-search, but you probably already know that.
+1. This would alleviate data transfer between cross-node joins where
Coordinator and Datanodes are on separate servers. You could always
have both nodes on the same server with the XC of now... But that's
double number of nodes to monitor.

> What exact things does the gtm-proxy do? For example, a single row insert
> wouldn't need the gtm (coordinator just inserts it to the right
> data-node)(asumming no sequences, since for that the gtm is needed)?
Grouping messages between Coordinator/Datanode and GTM to reduce
package interferences and improve performance.

> If multiple tables are sharded on the same key (example: user_id). Will all
> the rows, from the same user in different tables be in the same data-node ?
Yep. Node choice algorithm is based using the data type of the key.
-- 
Michael