Re: [Postgres-xc-developers] Patch ready for review: online data redistribution

Something I am afraid is not possible with an external utility is control of redistribution at node level. For example, an external contrib module or utility will launch SQL queries to xc that have to be treated as global. However, redistribution needs to take care of cases like for example the reduction of nodes for replicated tables. In this case you just need to delete the data from removed nodes. Another easy example is the case of an increase of nodes for replicated tables. You need to pick up data on coordinator and then send it only to the new nodes. Those simple examples need a core management to minimize the work of redistribution inside cluster.

On 2012/07/13, at 15:09, Ashutosh Bapat <ash...@en...> wrote:

> Even, I am wondering if that would be better.
> 
> But, one thing that is essential is catalog updates. Are you suggesting that the catalog updates too should be done using some SQL?
> 
> Can you please expand more on your idea, may be providing some examples, pseudo-code etc.?
> 
> On Fri, Jul 13, 2012 at 10:36 AM, Nikhil Sontakke <ni...@st...> wrote:
> Just a thought.
> 
> If we have a utility which spews out all of these statements to
> redistribute a table across node modifications, then we can just wrap
> them inside a transaction block and just run that?
> 
> Wont it save all of the core changes?
> 
> Regards,
> Nikhils
> 
> On Fri, Jul 13, 2012 at 12:29 AM, Michael Paquier
> <mic...@gm...> wrote:
> > Hi all,
> >
> > Please find attached an updated patch adding redistribution optimizations
> > for replicated tables.
> > If the node subset of a replicated table is reduced, the necessary nodes are
> > simply truncated.
> > If it is increased, a COPY TO is done to fetch the data, and COPY FROM is
> > done only on the necessary nodes.
> > New regression tests have been added to test that.
> >
> > Regards,
> >
> >
> > On Thu, Jul 12, 2012 at 5:30 PM, Michael Paquier <mic...@gm...>
> > wrote:
> >>
> >> OK, here is the mammoth patch: 3000 lines including docs, implementation
> >> and  regressions.
> >> The code has been realigned with current master.
> >> This patch introduces the latest thing I am working on: the redistribution
> >> command tree planning and execution.
> >>
> >> As I explained before, a redistribution consists of a series of commands
> >> (TRUNCATE, REINDEX, DELETE, COPY FROM, COPY TO) that need to be determined
> >> depending on the new and old locator information of the relation. Each
> >> action can be done on a subset of nodes.
> >> This patch introduces the basic infrastructure of the command tree build
> >> and execution.
> >> For the time being, redistribution uses only what is called the default
> >> command tree consisting of:
> >> 1) COPY TO
> >> 2) TRUNCATE
> >> 3) COPY FROM
> >> 4) REINDEX
> >> But this structure can be easily completed with more complicated
> >> operations.
> >> In this patch there is still a small thing missing which is the
> >> possibility to launch a COPY FROM on a subset of nodes, particularly useful
> >> when redistribution consists of a replicated table whose set of nodes is
> >> increased.
> >> Compared to the last versions, the impact of redistribution in tablecmds.c
> >> is limited.
> >>
> >> Regards,
> >>
> >> --
> >> Michael Paquier
> >> http://michael.otacoo.com
> >
> >
> >
> >
> > --
> > Michael Paquier
> > http://michael.otacoo.com
> >
> > ------------------------------------------------------------------------------
> > Live Security Virtual Conference
> > Exclusive live event will cover all the ways today's security and
> > threat landscape has changed and how IT managers can respond. Discussions
> > will include endpoint security, mobile security and the latest in malware
> > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> > _______________________________________________
> > Postgres-xc-developers mailing list
> > Pos...@li...
> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
> >
> 
> 
> 
> --
> StormDB - http://www.stormdb.com
> The Database Cloud
> 
> 
> 
> -- 
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Enterprise Postgres Company
>