postgres-xc-committers Mailing List for Postgres-XC

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Project "Postgres-XC".

The branch, master has been updated
       via  d03ea805cef9375bee9b751e65d698c07c138bf5 (commit)
      from  baa8c4a51cdd7de321169f12ebfb47b02fed3afc (commit)

- Log -----------------------------------------------------------------
http://postgres-xc.git.sourceforge.net/git/gitweb.cgi?p=postgres-xc/postgres-xc;a=commitdiff;h=d03ea805cef9375bee9b751e65d698c07c138bf5

commit d03ea805cef9375bee9b751e65d698c07c138bf5
Author: Michael Paquier <mi...@ot...>
Date:   Tue Jul 24 16:13:55 2012 +0900

    Support for online data redistribution with ALTER TABLE

    Online data redistribution is the possibility for a user to change the distribution
    strategy of a table. There are no restrictions in the modifications possible, meaning
    that all types of tables with all possible node subsets can be completely changed in
    one command.

    The SQL command used for redistribution is an extension of ALTER TABLE with those
    clauses specific to XC and already available in CREATE TABLE:
    DISTRIBUTE BY { REPLICATION | ROUND ROBIN | { [HASH | MODULO ] ( column_name ) } }
    TO { GROUP groupname | NODE ( nodename [, ... ] ) }
    ADD NODE ( nodename [, ... ] )
    DELETE NODE ( nodename [, ... ] )
    Those commands can be combined together without limitations.

    Several redistribution scenarios are implemented depending on the old and new
    distribution type of the table:
    - Default scenario:
    1) Fetch the data of the table with a COPY TO and store it inside a tuplestore
    2) Perform a TRUNCATE on the Datanodes
    3) Perform a COPY TO with tuples inside tuplestore
    4) REINDEX table if necessary
    This default scenario could also be managed by an external tool, however all
    the following optimizations need a node-level control to perform with highest
    efficiency possible. The performance of this scenario is equivalent to running
    a COPY TO/COPY FROM sequence on a table, so here performance is not bounded by
    the redistribution mechanism itself but by the COPY protocol used for data exchanged
    in network.
    - Replicated to replicated:
    In case of nodes removed from the set of nodes, those nodes are simply truncated,
    so this is really quick even on large sets of data.
    For new nodes, data is fetched on Coordinator from one Datanode with COPY TO,
    data is stored in a tuplestore, and then COPY FROM is launched only on the new
    nodes.
    - Replicated to distributed:
    If new nodes are added, a fallback to default scenario is made.
    If nodes are removed, those nodes are truncated.
    Finally, on the remaining nodes a DELETE query removing only the necessary tuples
    is launched to each remote node. In this case there is no data exchanged between
    nodes so performance is maximized.

    In order to support all those scenarios, a couple of new internal mechanisms have
    been added to XC: materialization on Coordinator of tuple slots and possibility
    to reuse them for redistribution purposes, externalization of a portion of
    PostgreSQL COPY code used by redistribution, reuse and extension of Postgres-XC
    APIs for remote COPY management.
    The tuplestore used to store tuples if necessary can have its allowed cache
    controlled with work_mem. The only thing to take care of is that the tuplestore
    data needs to be stored on Coordinator once so some additional disk space might
    be necessary on this server to perform redistribution correctly.

    Documentation, as well as a new set of regression tests have been added.
    Regressions do checks on views, prepared statementsm, views, distribution types
    and subsets in a way completely transparent whatever the cluster configuration.

M	doc-xc/src/sgml/ref/alter_table.sgmlin
M	src/backend/access/hash/hashfunc.c
M	src/backend/catalog/heap.c
M	src/backend/catalog/pgxc_class.c
M	src/backend/commands/copy.c
M	src/backend/commands/tablecmds.c
M	src/backend/parser/gram.y
M	src/backend/parser/parse_utilcmd.c
M	src/backend/pgxc/copy/Makefile
A	src/backend/pgxc/copy/copyops.c
M	src/backend/pgxc/copy/remotecopy.c
M	src/backend/pgxc/locator/Makefile
M	src/backend/pgxc/locator/locator.c
A	src/backend/pgxc/locator/redistrib.c
M	src/backend/pgxc/nodemgr/nodemgr.c
M	src/backend/pgxc/pool/execRemote.c
M	src/include/access/hash.h
M	src/include/catalog/pgxc_class.h
M	src/include/nodes/parsenodes.h
A	src/include/pgxc/copyops.h
M	src/include/pgxc/execRemote.h
M	src/include/pgxc/locator.h
A	src/include/pgxc/redistrib.h
M	src/include/pgxc/remotecopy.h
M	src/include/utils/rel.h
M	src/test/regress/expected/xc_alter_table.out
M	src/test/regress/expected/xc_create_function.out
M	src/test/regress/sql/xc_alter_table.sql
M	src/test/regress/sql/xc_create_function.sql

-----------------------------------------------------------------------

Summary of changes:
 doc-xc/src/sgml/ref/alter_table.sgmlin           |  224 ++++++-
 src/backend/access/hash/hashfunc.c               |   81 ++-
 src/backend/catalog/heap.c                       |   28 +-
 src/backend/catalog/pgxc_class.c                 |  106 +++-
 src/backend/commands/copy.c                      |   22 +-
 src/backend/commands/tablecmds.c                 |  531 +++++++++++++-
 src/backend/parser/gram.y                        |   34 +
 src/backend/parser/parse_utilcmd.c               |    4 +-
 src/backend/pgxc/copy/Makefile                   |    2 +-
 src/backend/pgxc/copy/copyops.c                  |  496 ++++++++++++
 src/backend/pgxc/copy/remotecopy.c               |   22 +-
 src/backend/pgxc/locator/Makefile                |    4 +-
 src/backend/pgxc/locator/locator.c               |   37 +-
 src/backend/pgxc/locator/redistrib.c             |  871 ++++++++++++++++++++++
 src/backend/pgxc/nodemgr/nodemgr.c               |    3 +-
 src/backend/pgxc/pool/execRemote.c               |  133 +++-
 src/include/access/hash.h                        |    1 +
 src/include/catalog/pgxc_class.h                 |   25 +-
 src/include/nodes/parsenodes.h                   |    6 +
 src/include/pgxc/copyops.h                       |   27 +
 src/include/pgxc/execRemote.h                    |   17 +-
 src/include/pgxc/locator.h                       |    1 +
 src/include/pgxc/redistrib.h                     |   80 ++
 src/include/pgxc/remotecopy.h                    |    1 +
 src/include/utils/rel.h                          |    8 +
 src/test/regress/expected/xc_alter_table.out     |  408 ++++++++++
 src/test/regress/expected/xc_create_function.out |   84 +++
 src/test/regress/sql/xc_alter_table.sql          |  133 ++++
 src/test/regress/sql/xc_create_function.sql      |   85 +++
 29 files changed, 3416 insertions(+), 58 deletions(-)
 create mode 100644 src/backend/pgxc/copy/copyops.c
 create mode 100644 src/backend/pgxc/locator/redistrib.c
 create mode 100644 src/include/pgxc/copyops.h
 create mode 100644 src/include/pgxc/redistrib.h

hooks/post-receive
-- 
Postgres-XC

2010	Jan	Feb	Mar	Apr (4)	May (28)	Jun (12)	Jul (11)	Aug (12)	Sep (5)	Oct (19)	Nov (14)	Dec (12)
2011	Jan (18)	Feb (30)	Mar (115)	Apr (89)	May (50)	Jun (44)	Jul (22)	Aug (13)	Sep (11)	Oct (30)	Nov (28)	Dec (39)
2012	Jan (38)	Feb (18)	Mar (43)	Apr (91)	May (108)	Jun (46)	Jul (37)	Aug (44)	Sep (33)	Oct (29)	Nov (36)	Dec (15)
2013	Jan (35)	Feb (611)	Mar (5)	Apr (55)	May (30)	Jun (28)	Jul (458)	Aug (34)	Sep (9)	Oct (39)	Nov (22)	Dec (32)
2014	Jan (16)	Feb (16)	Mar (42)	Apr (179)	May (7)	Jun (6)	Jul (9)	Aug	Sep (4)	Oct	Nov (3)	Dec
2015	Jan	Feb	Mar	Apr (2)	May (4)	Jun	Jul	Aug	Sep	Oct	Nov	Dec

S	M	T	W	T	F	S
1	2 (2)	3 (2)	4 (2)	5	6 (4)	7
8	9 (2)	10 (1)	11 (3)	12 (10)	13 (2)	14
15	16	17 (2)	18 (3)	19	20	21
22	23 (3)	24 (1)	25	26	27	28
29	30	31

postgres-xc-committers Mailing List for Postgres-XC

postgres-xc-committers — Announces PG-XC repository commits