postgres-xc-general Mailing List for Postgres-XC

Hello,
we solved the problem with GTM - when we deleted the sequence it stopped 
writing extra data to log file. But the volume of inserted data per 
minute is still very small.

On 27. 11. 2013 10:52, Koichi Suzuki wrote:
> This could be a cause.   Are you using sequence heavily?  GTM backs up 
> its restart point every 2000 updates of Transaction ID or each 
> sequences value.
>
> Regards;
>
> ---
> Koichi Suzuki
>
>
> 2013/11/27 Matej Jellus <je...@ts... <mailto:je...@ts...>>
>
>     The value on the first line is increasing realy slow. The sequence
>     is increasing fast.
>     gtm.control :
>     prev :
>     727618
>     gps.public.ai_value\00  30957   1       1       1
>     9223372036854775806     f       t       1
>     next :
>     727618
>     gps.public.ai_value\00  30958   1       1       1
>     9223372036854775806     f       t       1
>
>
>     Thank you, best regards
>     Matej
>
>     On 27. 11. 2013 10:33, Koichi Suzuki wrote:
>>     Could you visit GTM's data directory and watch the file gtm.control?
>>     The first line is current restart point of GXID value.   If you
>>     watch this value periodically, it shows how many transactions are
>>     coming to GTM.
>>
>>     If the value does not increase so much, then please see another
>>     line, which are current value of sequences.   This will tell how
>>     often GTM has to backup itself.  If not there could be something
>>     wrong in GTM.
>>
>>     Best Regards;
>>
>>     ---
>>     Koichi Suzuki
>>
>>
>>     2013/11/27 Matej Jellus <je...@ts...
>>     <mailto:je...@ts...>>
>>
>>         Hello,
>>         I don't understand how it can create so many transactions. If
>>         I start phppgadmin, create non-partitioning table and insert
>>         there one row, nothing is written to log file. If I create
>>         partitioning table and insert there row, it writes
>>         'Saving transaction restoration info, backed-up gxid' to log
>>         file. If I send there "INSERT INTO <table> VALUES(...),
>>         (...), (...);" it writes that row three times to log file. I
>>         don't know how it is related to the partitioning, but it
>>         surely has dependencies.
>>
>>
>>         Thank you, best regards
>>         Matej
>>
>>         On 25. 11. 2013 8:47, Koichi Suzuki wrote:
>>>         Your experiment says that your application created more than
>>>         hundreds of thousand of transactions in a second.    Write
>>>         to GTM will be done once every two thousand transactions.  
>>>          I'm wondering how it happens and how it is related to the
>>>         partitioning.
>>>
>>>         Regards;
>>>
>>>         ---
>>>         Koichi Suzuki
>>>
>>>
>>>         2013/11/20 Matej Jellus <je...@ts...
>>>         <mailto:je...@ts...>>
>>>
>>>             Hello,
>>>             is it a problem of configuration? Or is it all bad?
>>>             Should we wait for some answers?
>>>
>>>
>>>             Thank you, best regards
>>>             Matej
>>>
>>>             On 11. 11. 2013 15:28, Matej Jellus wrote:
>>>>             Hello, here is the process.
>>>>
>>>>             When we created table with DISTRIBUTE BY REPLICATION TO
>>>>             NODE ( datanode1 ), it has been inserting maybe 100 000
>>>>             rows per minute.
>>>>             When we created table with DISTRIBUTE BY REPLICATION TO
>>>>             NODE ( datanode1, datanode2 ), it has been inserting
>>>>             maybe 50 000 rows per minute.
>>>>             When we created table witch  DISTRIBUTE BY MODULO (
>>>>             cluster_id ) TO NODE ( datanode1, datanode2 ) and has
>>>>             been writing to only one node, it was inserting 170 000
>>>>             rows per minute.
>>>>
>>>>             But when we created it with partitioning, it has been
>>>>             5000 rows per minute.
>>>>
>>>>             When we insert row into nonpartitioning table, no row
>>>>             was generated into the gtm log file. When we insert row
>>>>             into partitioning table, extra row is inserted into gtm
>>>>             log file : Saving transaction restoration info,
>>>>             backed-up gxid: 409215
>>>>             LOCATION: GTM_WriteRestorePointXid, gtm_txn.c:2649.
>>>>
>>>>             When deleting or updating from nonpartitioning table,
>>>>             everything is going quickly. But when we do the same
>>>>             thing with partitioning table, it is 3 - 4 times slower.
>>>>
>>>>             Below is example with partitioning. We are importing
>>>>             data to database, so first we insert unit_id into table
>>>>             units. This will generate schema for the year we are
>>>>             inserting, based on the column gps_timestamp. After
>>>>             that it create table for the unit in the schema, which
>>>>             inherits from primary talbe - units_data. Then we are
>>>>             inserting only to table units_data.
>>>>
>>>>             CREATE TABLE data.units ( unit_id uuid NOT NULL,
>>>>             cluster_id INT NOT NULL );
>>>>
>>>>             CREATE SEQUENCE ai_value;
>>>>
>>>>             CREATE TABLE data.units_data (
>>>>                 row_id INTEGER NOT NULL DEFAULT nextval('ai_value'),
>>>>                 unit_id uuid NOT NULL,
>>>>                 cluster_id INT NOT NULL,
>>>>                 gps_timestamp TIMESTAMP WITHOUT TIME ZONE NOT NULL,
>>>>                 gps_lat DOUBLE PRECISION NOT NULL,
>>>>                 gps_lng DOUBLE PRECISION NOT NULL,
>>>>                 others hstore,
>>>>                 PRIMARY KEY ( unit_id,cluster_id )
>>>>             );
>>>>
>>>>             CREATE OR REPLACE FUNCTION generate_all(IN cluster_id
>>>>             INTEGER,  IN unit_id UUID) RETURNS VOID AS $$
>>>>             DECLARE
>>>>                 ROK VARCHAR(4);
>>>>                 DATANODE_ID VARCHAR(1);
>>>>                 NAZOV_SCHEMY VARCHAR(13);
>>>>                 NAZOV_TABULKY VARCHAR;
>>>>                 NAZOV_END_TABULKY VARCHAR;
>>>>                 NAZOV_ROK_TABULKY VARCHAR;
>>>>             BEGIN
>>>>                 -- SELECT 2013 INTO ROK;
>>>>                 ROK := 2013;
>>>>                 SELECT cluster_id INTO DATANODE_ID;
>>>>                 NAZOV_SCHEMY := 'node_'||DATANODE_ID||'_y_'||ROK;
>>>>                 NAZOV_ROK_TABULKY := 'units_data_2013';
>>>>                 NAZOV_TABULKY := 'u_'||replace(unit_id::text,'-','');
>>>>                 NAZOV_END_TABULKY := NAZOV_TABULKY||'_m';
>>>>                 -- Ide sa vytvarat schema a tabulky
>>>>                     IF NOT EXISTS(SELECT * FROM
>>>>             information_schema.schemata WHERE
>>>>             schema_name=NAZOV_SCHEMY) THEN
>>>>                         EXECUTE 'CREATE SCHEMA '||NAZOV_SCHEMY||';';
>>>>                     END IF;
>>>>                     IF NOT EXISTS(SELECT * FROM
>>>>             information_schema.tables where
>>>>             table_name=NAZOV_ROK_TABULKY AND
>>>>             table_schema=NAZOV_SCHEMY) THEN
>>>>                         EXECUTE 'CREATE TABLE
>>>>             '||NAZOV_SCHEMY||'.'||NAZOV_ROK_TABULKY||'( CHECK (
>>>>             date_part(''year''::text, gps_timestamp)=2013 ) )
>>>>             INHERITS ( data.units_data );';
>>>>                     END IF;
>>>>                     EXECUTE 'CREATE OR REPLACE RULE
>>>>             "year2013_node'||DATANODE_ID||'_choice" AS ON INSERT TO
>>>>             data.units_data WHERE date_part(''year''::text,
>>>>             NEW.gps_timestamp)=2013 AND
>>>>             NEW.cluster_id='||DATANODE_ID||' DO INSTEAD INSERT INTO
>>>>             '||NAZOV_SCHEMY||'.units_data_2013 VALUES(NEW.*);';
>>>>
>>>>                     EXECUTE 'CREATE TABLE
>>>>             '||NAZOV_SCHEMY||'.'||NAZOV_TABULKY||'( CHECK (
>>>>             unit_id='''||unit_id||'''::uuid ) ) INHERITS
>>>>             ('||NAZOV_SCHEMY||'.'||NAZOV_ROK_TABULKY||' );';
>>>>
>>>>                     EXECUTE 'CREATE RULE
>>>>             "'||NAZOV_TABULKY||'_choice" AS ON INSERT TO
>>>>             '||NAZOV_SCHEMY||'.units_data_2013 WHERE
>>>>             unit_id='''||unit_id||''' DO INSTEAD INSERT INTO
>>>>             '||NAZOV_SCHEMY||'.'||NAZOV_TABULKY||' VALUES(NEW.*);';
>>>>                 RETURN;
>>>>             END
>>>>             $$ LANGUAGE plpgsql;
>>>>
>>>>             CREATE RULE "generate_all_rule" AS ON INSERT TO
>>>>             data.units DO ALSO SELECT generate_all(NEW.cluster_id,
>>>>             NEW.unit_id);
>>>>
>>>>             Thank you, best regards
>>>>             Matej
>>>>
>>>>
>>>>             On 11. 11. 2013 10:11, 鈴木 幸市 wrote:
>>>>>             This does not show anything special.   As I wrote, the
>>>>>             messages in question don’t appear so often.     Could
>>>>>             you test how many of these messages are written using
>>>>>             another case, for example, without using
>>>>>             partitioning.   I think partitioning table has nothing
>>>>>             to do with the issue but I’d like to know if the case
>>>>>             happens in other DMLs and DDLs.
>>>>>
>>>>>             Thank you.
>>>>>             ---
>>>>>             Koichi Suzuki
>>>>>
>>>>>             2013/11/11 17:49、Matej Jellus <je...@ts...>
>>>>>             <mailto:je...@ts...> のメール：
>>>>>
>>>>>>             Our situation :
>>>>>>             two machines
>>>>>>             first - primary GTM, coordinator, datanode
>>>>>>             second - standby GTM, coordinator, datanode
>>>>>>
>>>>>>             GTM configuration below. We are inserting 5 thousand
>>>>>>             rows per minute, which is too low for us. We need to
>>>>>>             insert 100 thousand rows per minute.
>>>>>>             Starting primary GTM : /gtm_ctl -Z gtm start -D
>>>>>>             /var/pgxc/data_gtm
>>>>>>             Starting standby GTM : /gtm_ctl -Z gtm start -D
>>>>>>             /var/pgxc/data_gtm
>>>>>>             Starting coord : pg_ctl start -D
>>>>>>             /var/pgxc/ubuntu2_coord -Z coordinator -l
>>>>>>             /tmp/logfile_ubuntu2_coord
>>>>>>
>>>>>>             Tables in database are using partitioning. So we have
>>>>>>             one primary table and we are creating child tables
>>>>>>             from it. For example :
>>>>>>             primary table :
>>>>>>             chema - data
>>>>>>             table - units_data
>>>>>>
>>>>>>             child tables :
>>>>>>             schema : y_2013_node1 - tables, which will be saved
>>>>>>             in node 1
>>>>>>                table : u_<unit_id> - child table from
>>>>>>             data.units_data
>>>>>>             schema : y_2013_node2 - tables, which will be saved
>>>>>>             in node 2
>>>>>>                table : u_<unit_id> - child table from
>>>>>>             data.units_data
>>>>>>
>>>>>>             Thank you, best regards
>>>>>>             Matej
>>>>>>
>>>>>>
>>>>>>             On 11. 11. 2013 7:49, 鈴木 幸市 wrote:
>>>>>>>             Message 1. will be written once every 2000
>>>>>>>             transactions. The number looks too large. Message 1
>>>>>>>             is written in the three cases:
>>>>>>>
>>>>>>>             1) When GTM starts,
>>>>>>>             2) Once every 2000 transactions,
>>>>>>>             3) When GTM standby is promoted to a master.
>>>>>>>
>>>>>>>             Could you let me know how you start your cluster?
>>>>>>>             We’ve tested XC with DBT-1 and DBT-2 but didn’t have
>>>>>>>             such frequent write at gtm. You are inserting 5000
>>>>>>>             raws in a second so 2) should happen twice or three
>>>>>>>             times a second.
>>>>>>>
>>>>>>>             Message 2. is written when GTM standby connects to
>>>>>>>             GTM.   Are you using GTM standby?   If so, how you
>>>>>>>             are starting it?
>>>>>>>
>>>>>>>             We’re testing XC in different environment but didn’t
>>>>>>>             have such GTM overload. I’m interested in the
>>>>>>>             situation.
>>>>>>>
>>>>>>>             Best;
>>>>>>>             ---
>>>>>>>             Koichi Suzuki
>>>>>>>
>>>>>>>             2013/11/09 0:06、Matej Jellus <je...@ts...>
>>>>>>>             <mailto:je...@ts...> のメール：
>>>>>>>
>>>>>>>>             Hello,
>>>>>>>>
>>>>>>>>             We have problem that postgres xc is very slow, gtm
>>>>>>>>             is writing to log a
>>>>>>>>             lot of data. Is it able to be disabled?
>>>>>>>>             The most repeated lines are :
>>>>>>>>
>>>>>>>>             1:140130870839040:2013-11-08 15:42:49.357 CET
>>>>>>>>             -LOG:  Saving transaction
>>>>>>>>             restoration info, backed-up gxid: 373106
>>>>>>>>             LOCATION: GTM_WriteRestorePointXid, gtm_txn.c:2649
>>>>>>>>             ( this is maybe 130 times per second )
>>>>>>>>
>>>>>>>>             1:140130879186688:2013-11-08 15:45:37.572 CET -LOG:
>>>>>>>>             Connection
>>>>>>>>             established with GTM standby. - 0x1f91398
>>>>>>>>             LOCATION: gtm_standby_connect_to_standby_int,
>>>>>>>>             gtm_standby.c:400
>>>>>>>>
>>>>>>>>             It is overloading the disk, i/o is 70%.
>>>>>>>>
>>>>>>>>             Now it is inserting 5 thousand inserts per minute.
>>>>>>>>             We need more inserts
>>>>>>>>             per minute.
>>>>>>>>
>>>>>>>>             Thank you, best regards
>>>>>>>>             Matej
>>>>>>>>
>>>>>>>>             ------------------------------------------------------------------------------
>>>>>>>>
>>>>>>>>             November Webinars for C, C++, Fortran Developers
>>>>>>>>             Accelerate application performance with scalable
>>>>>>>>             programming models. Explore
>>>>>>>>             techniques for threading, error checking, porting,
>>>>>>>>             and tuning. Get the most
>>>>>>>>             from the latest Intel processors and coprocessors.
>>>>>>>>             See abstracts and register
>>>>>>>>             http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
>>>>>>>>
>>>>>>>>             _______________________________________________
>>>>>>>>             Postgres-xc-general mailing list
>>>>>>>>             Pos...@li...
>>>>>>>>             <mailto:Pos...@li...>
>>>>>>>>             https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>             -- 
>>>
>>>             S pozdravom, best regards
>>>             *Matej Jellus*
>>>
>>>             TSS Group a.s.
>>>
>>>             *Technický pracovník - GPS monitoring*
>>>
>>>             TSS Group a.s.
>>>             K Zábraniu 1653
>>>             911 01 Trenčín
>>>             tel: +421 32 744 5921 <tel:%2B421%2032%20744%205921>
>>>             fax: +421 32 744 5922 <tel:%2B421%2032%20744%205922>
>>>             e-mail:je...@ts... <mailto:je...@ts...>
>>>             url: www.tssgroup.sk <http://www.tssgroup.sk>
>>>             url: www.gpsmonitoring.sk <http://www.gpsmonitoring.sk>
>>>
>>>
>>>             ------------------------------------------------------------------------------
>>>             Shape the Mobile Experience: Free Subscription
>>>             Software experts and developers: Be at the forefront of
>>>             tech innovation.
>>>             Intel(R) Software Adrenaline delivers strategic insight
>>>             and game-changing
>>>             conversations that shape the rapidly evolving mobile
>>>             landscape. Sign up now.
>>>             http://pubads.g.doubleclick.net/gampad/clk?id=63431311&iu=/4140/ostg.clktrk
>>>             _______________________________________________
>>>             Postgres-xc-general mailing list
>>>             Pos...@li...
>>>             <mailto:Pos...@li...>
>>>             https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>
>>>
>>
>>         -- 
>>
>>         S pozdravom, best regards
>>         *Matej Jellus*
>>
>>         TSS Group a.s.
>>
>>         *Technický pracovník - GPS monitoring*
>>
>>         TSS Group a.s.
>>         K Zábraniu 1653
>>         911 01 Trenčín
>>         tel: +421 32 744 5921 <tel:%2B421%2032%20744%205921>
>>         fax: +421 32 744 5922 <tel:%2B421%2032%20744%205922>
>>         e-mail:je...@ts... <mailto:je...@ts...>
>>         url: www.tssgroup.sk <http://www.tssgroup.sk>
>>         url: www.gpsmonitoring.sk <http://www.gpsmonitoring.sk>
>>
>>
>
>     -- 
>
>     S pozdravom, best regards
>     *Matej Jellus*
>
>     TSS Group a.s.
>
>     *Technický pracovník - GPS monitoring*
>
>     TSS Group a.s.
>     K Zábraniu 1653
>     911 01 Trenčín
>     tel: +421 32 744 5921 <tel:%2B421%2032%20744%205921>
>     fax: +421 32 744 5922 <tel:%2B421%2032%20744%205922>
>     e-mail:je...@ts... <mailto:je...@ts...>
>     url: www.tssgroup.sk <http://www.tssgroup.sk>
>     url: www.gpsmonitoring.sk <http://www.gpsmonitoring.sk>
>
>

Hello,
I don't understand how it can create so many transactions. If I start 
phppgadmin, create non-partitioning table and insert there one row, 
nothing is written to log file. If I create partitioning table and 
insert there row, it writes
'Saving transaction restoration info, backed-up gxid' to log file. If I 
send there "INSERT INTO <table> VALUES(...), (...), (...);" it writes 
that row three times to log file. I don't know how it is related to the 
partitioning, but it surely has dependencies.

Thank you, best regards
Matej

On 25. 11. 2013 8:47, Koichi Suzuki wrote:
> Your experiment says that your application created more than hundreds 
> of thousand of transactions in a second.  Write to GTM will be done 
> once every two thousand transactions.    I'm wondering how it happens 
> and how it is related to the partitioning.
>
> Regards;
>
> ---
> Koichi Suzuki
>
>
> 2013/11/20 Matej Jellus <je...@ts... <mailto:je...@ts...>>
>
>     Hello,
>     is it a problem of configuration? Or is it all bad? Should we wait
>     for some answers?
>
>
>     Thank you, best regards
>     Matej
>
>     On 11. 11. 2013 15:28, Matej Jellus wrote:
>>     Hello, here is the process.
>>
>>     When we created table with DISTRIBUTE BY REPLICATION TO NODE (
>>     datanode1 ), it has been inserting maybe 100 000 rows per minute.
>>     When we created table with DISTRIBUTE BY REPLICATION TO NODE (
>>     datanode1, datanode2 ), it has been inserting maybe 50 000 rows
>>     per minute.
>>     When we created table witch  DISTRIBUTE BY MODULO ( cluster_id )
>>     TO NODE ( datanode1, datanode2 ) and has been writing to only one
>>     node, it was inserting 170 000 rows per minute.
>>
>>     But when we created it with partitioning, it has been 5000 rows
>>     per minute.
>>
>>     When we insert row into nonpartitioning table, no row was
>>     generated into the gtm log file. When we insert row into
>>     partitioning table, extra row is inserted into gtm log file :
>>     Saving transaction restoration info, backed-up gxid: 409215
>>     LOCATION:  GTM_WriteRestorePointXid, gtm_txn.c:2649.
>>
>>     When deleting or updating from nonpartitioning table, everything
>>     is going quickly. But when we do the same thing with partitioning
>>     table, it is 3 - 4 times slower.
>>
>>     Below is example with partitioning. We are importing data to
>>     database, so first we insert unit_id into table units. This will
>>     generate schema for the year we are inserting, based on the
>>     column gps_timestamp. After that it create table for the unit in
>>     the schema, which inherits from primary talbe - units_data. Then
>>     we are inserting only to table units_data.
>>
>>     CREATE TABLE data.units ( unit_id uuid NOT NULL, cluster_id INT
>>     NOT NULL );
>>
>>     CREATE SEQUENCE ai_value;
>>
>>     CREATE TABLE data.units_data (
>>         row_id INTEGER NOT NULL DEFAULT nextval('ai_value'),
>>         unit_id uuid NOT NULL,
>>         cluster_id INT NOT NULL,
>>         gps_timestamp TIMESTAMP WITHOUT TIME ZONE NOT NULL,
>>         gps_lat DOUBLE PRECISION NOT NULL,
>>         gps_lng DOUBLE PRECISION NOT NULL,
>>         others hstore,
>>         PRIMARY KEY ( unit_id,cluster_id )
>>     );
>>
>>     CREATE OR REPLACE FUNCTION generate_all(IN cluster_id INTEGER, 
>>     IN unit_id UUID) RETURNS VOID AS $$
>>     DECLARE
>>         ROK VARCHAR(4);
>>         DATANODE_ID VARCHAR(1);
>>         NAZOV_SCHEMY VARCHAR(13);
>>         NAZOV_TABULKY VARCHAR;
>>         NAZOV_END_TABULKY VARCHAR;
>>         NAZOV_ROK_TABULKY VARCHAR;
>>     BEGIN
>>         -- SELECT 2013 INTO ROK;
>>         ROK := 2013;
>>         SELECT cluster_id INTO DATANODE_ID;
>>         NAZOV_SCHEMY := 'node_'||DATANODE_ID||'_y_'||ROK;
>>         NAZOV_ROK_TABULKY := 'units_data_2013';
>>         NAZOV_TABULKY := 'u_'||replace(unit_id::text,'-','');
>>         NAZOV_END_TABULKY := NAZOV_TABULKY||'_m';
>>         -- Ide sa vytvarat schema a tabulky
>>             IF NOT EXISTS(SELECT * FROM information_schema.schemata
>>     WHERE schema_name=NAZOV_SCHEMY) THEN
>>                 EXECUTE 'CREATE SCHEMA '||NAZOV_SCHEMY||';';
>>             END IF;
>>             IF NOT EXISTS(SELECT * FROM information_schema.tables
>>     where table_name=NAZOV_ROK_TABULKY AND table_schema=NAZOV_SCHEMY)
>>     THEN
>>                 EXECUTE 'CREATE TABLE
>>     '||NAZOV_SCHEMY||'.'||NAZOV_ROK_TABULKY||'( CHECK (
>>     date_part(''year''::text, gps_timestamp)=2013 ) ) INHERITS (
>>     data.units_data );';
>>             END IF;
>>             EXECUTE 'CREATE OR REPLACE RULE
>>     "year2013_node'||DATANODE_ID||'_choice" AS ON INSERT TO
>>     data.units_data WHERE date_part(''year''::text,
>>     NEW.gps_timestamp)=2013 AND NEW.cluster_id='||DATANODE_ID||' DO
>>     INSTEAD INSERT INTO '||NAZOV_SCHEMY||'.units_data_2013
>>     VALUES(NEW.*);';
>>
>>             EXECUTE 'CREATE TABLE
>>     '||NAZOV_SCHEMY||'.'||NAZOV_TABULKY||'( CHECK (
>>     unit_id='''||unit_id||'''::uuid ) ) INHERITS
>>     ('||NAZOV_SCHEMY||'.'||NAZOV_ROK_TABULKY||' );';
>>
>>             EXECUTE 'CREATE RULE "'||NAZOV_TABULKY||'_choice" AS ON
>>     INSERT TO '||NAZOV_SCHEMY||'.units_data_2013 WHERE
>>     unit_id='''||unit_id||''' DO INSTEAD INSERT INTO
>>     '||NAZOV_SCHEMY||'.'||NAZOV_TABULKY||' VALUES(NEW.*);';
>>         RETURN;
>>     END
>>     $$ LANGUAGE plpgsql;
>>
>>     CREATE RULE "generate_all_rule" AS ON INSERT TO data.units DO
>>     ALSO SELECT generate_all(NEW.cluster_id, NEW.unit_id);
>>
>>     Thank you, best regards
>>     Matej
>>
>>
>>     On 11. 11. 2013 10:11, 鈴木 幸市 wrote:
>>>     This does not show anything special.   As I wrote, the messages
>>>     in question don’t appear so often.     Could you test how many
>>>     of these messages are written using another case, for example,
>>>     without using partitioning.   I think partitioning table has
>>>     nothing to do with the issue but I’d like to know if the case
>>>     happens in other DMLs and DDLs.
>>>
>>>     Thank you.
>>>     ---
>>>     Koichi Suzuki
>>>
>>>     2013/11/11 17:49、Matej Jellus <je...@ts...>
>>>     <mailto:je...@ts...> のメール：
>>>
>>>>     Our situation :
>>>>     two machines
>>>>     first - primary GTM, coordinator, datanode
>>>>     second - standby GTM, coordinator, datanode
>>>>
>>>>     GTM configuration below. We are inserting 5 thousand rows per
>>>>     minute, which is too low for us. We need to insert 100 thousand
>>>>     rows per minute.
>>>>     Starting primary GTM : /gtm_ctl -Z gtm start -D /var/pgxc/data_gtm
>>>>     Starting standby GTM : /gtm_ctl -Z gtm start -D /var/pgxc/data_gtm
>>>>     Starting coord : pg_ctl start -D /var/pgxc/ubuntu2_coord -Z
>>>>     coordinator -l /tmp/logfile_ubuntu2_coord
>>>>
>>>>     Tables in database are using partitioning. So we have one
>>>>     primary table and we are creating child tables from it. For
>>>>     example :
>>>>     primary table :
>>>>     chema - data
>>>>     table - units_data
>>>>
>>>>     child tables :
>>>>     schema : y_2013_node1 - tables, which will be saved in node 1
>>>>        table : u_<unit_id> - child table from data.units_data
>>>>     schema : y_2013_node2 - tables, which will be saved in node 2
>>>>        table : u_<unit_id> - child table from data.units_data
>>>>
>>>>     Thank you, best regards
>>>>     Matej
>>>>
>>>>
>>>>     On 11. 11. 2013 7:49, 鈴木 幸市 wrote:
>>>>>     Message 1. will be written once every 2000 transactions.   
>>>>>     The number looks too large.   Message 1 is written in the
>>>>>     three cases:
>>>>>
>>>>>     1) When GTM starts,
>>>>>     2) Once every 2000 transactions,
>>>>>     3) When GTM standby is promoted to a master.
>>>>>
>>>>>     Could you let me know how you start your cluster?    We’ve
>>>>>     tested XC with DBT-1 and DBT-2 but didn’t have such frequent
>>>>>     write at gtm.   You are inserting 5000 raws in a second so 2)
>>>>>     should happen twice or three times a second.
>>>>>
>>>>>     Message 2. is written when GTM standby connects to GTM.   Are
>>>>>     you using GTM standby?   If so, how you are starting it?
>>>>>
>>>>>     We’re testing XC in different environment but didn’t have such
>>>>>     GTM overload.   I’m interested in the situation.
>>>>>
>>>>>     Best;
>>>>>     ---
>>>>>     Koichi Suzuki
>>>>>
>>>>>     2013/11/09 0:06、Matej Jellus <je...@ts...>
>>>>>     <mailto:je...@ts...> のメール：
>>>>>
>>>>>>     Hello,
>>>>>>
>>>>>>     We have problem that postgres xc is very slow, gtm is writing
>>>>>>     to log a
>>>>>>     lot of data. Is it able to be disabled?
>>>>>>     The most repeated lines are :
>>>>>>
>>>>>>     1:140130870839040:2013-11-08 15:42:49.357 CET -LOG:  Saving
>>>>>>     transaction
>>>>>>     restoration info, backed-up gxid: 373106
>>>>>>     LOCATION:  GTM_WriteRestorePointXid, gtm_txn.c:2649
>>>>>>     ( this is maybe 130 times per second )
>>>>>>
>>>>>>     1:140130879186688:2013-11-08 15:45:37.572 CET -LOG:  Connection
>>>>>>     established with GTM standby. - 0x1f91398
>>>>>>     LOCATION: gtm_standby_connect_to_standby_int, gtm_standby.c:400
>>>>>>
>>>>>>     It is overloading the disk, i/o is 70%.
>>>>>>
>>>>>>     Now it is inserting 5 thousand inserts per minute. We need
>>>>>>     more inserts
>>>>>>     per minute.
>>>>>>
>>>>>>     Thank you, best regards
>>>>>>     Matej
>>>>>>
>>>>>>     ------------------------------------------------------------------------------
>>>>>>
>>>>>>     November Webinars for C, C++, Fortran Developers
>>>>>>     Accelerate application performance with scalable programming
>>>>>>     models. Explore
>>>>>>     techniques for threading, error checking, porting, and
>>>>>>     tuning. Get the most
>>>>>>     from the latest Intel processors and coprocessors. See
>>>>>>     abstracts and register
>>>>>>     http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
>>>>>>
>>>>>>     _______________________________________________
>>>>>>     Postgres-xc-general mailing list
>>>>>>     Pos...@li...
>>>>>>     <mailto:Pos...@li...>
>>>>>>     https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>>>>
>>>>
>>
>
>     -- 
>
>     S pozdravom, best regards
>     *Matej Jellus*
>
>     TSS Group a.s.
>
>     *Technický pracovník - GPS monitoring*
>
>     TSS Group a.s.
>     K Zábraniu 1653
>     911 01 Trenčín
>     tel: +421 32 744 5921 <tel:%2B421%2032%20744%205921>
>     fax: +421 32 744 5922 <tel:%2B421%2032%20744%205922>
>     e-mail:je...@ts... <mailto:je...@ts...>
>     url: www.tssgroup.sk <http://www.tssgroup.sk>
>     url: www.gpsmonitoring.sk <http://www.gpsmonitoring.sk>
>
>
>     ------------------------------------------------------------------------------
>     Shape the Mobile Experience: Free Subscription
>     Software experts and developers: Be at the forefront of tech
>     innovation.
>     Intel(R) Software Adrenaline delivers strategic insight and
>     game-changing
>     conversations that shape the rapidly evolving mobile landscape.
>     Sign up now.
>     http://pubads.g.doubleclick.net/gampad/clk?id=63431311&iu=/4140/ostg.clktrk
>     _______________________________________________
>     Postgres-xc-general mailing list
>     Pos...@li...
>     <mailto:Pos...@li...>
>     https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
>

-- 

S pozdravom, best regards
*Matej Jellus*

TSS Group a.s.

*Technický pracovník - GPS monitoring*

TSS Group a.s.
K Zábraniu 1653
911 01 Trenčín
tel:  +421 32 744 5921
fax: +421 32 744 5922
e-mail:je...@ts... <mailto:je...@ts...>
url: www.tssgroup.sk <http://www.tssgroup.sk>
url: www.gpsmonitoring.sk <http://www.gpsmonitoring.sk>

I’m waiting for a opinion from the guy familiar with this.

Sorry for inconvenience.
---
Koichi Suzuki

2013/11/20 17:31、Matej Jellus <je...@ts...<mailto:je...@ts...>> のメール：

Hello,
is it a problem of configuration? Or is it all bad? Should we wait for some answers?

Thank you, best regards
Matej

On 11. 11. 2013 15:28, Matej Jellus wrote:
Hello, here is the process.

When we created table with DISTRIBUTE BY REPLICATION TO NODE ( datanode1 ), it has been inserting maybe 100 000 rows per minute.
When we created table with DISTRIBUTE BY REPLICATION TO NODE ( datanode1, datanode2 ), it has been inserting maybe 50 000 rows per minute.
When we created table witch  DISTRIBUTE BY MODULO ( cluster_id ) TO NODE ( datanode1, datanode2 ) and has been writing to only one node, it was inserting 170 000 rows per minute.

But when we created it with partitioning, it has been 5000 rows per minute.

When we insert row into nonpartitioning table, no row was generated into the gtm log file. When we insert row into partitioning table, extra row is inserted into gtm log file : Saving transaction restoration info, backed-up gxid: 409215
LOCATION:  GTM_WriteRestorePointXid, gtm_txn.c:2649.

When deleting or updating from nonpartitioning table, everything is going quickly. But when we do the same thing with partitioning table, it is 3 - 4 times slower.

Below is example with partitioning. We are importing data to database, so first we insert unit_id into table units. This will generate schema for the year we are inserting, based on the column gps_timestamp. After that it create table for the unit in the schema, which inherits from primary talbe - units_data. Then we are inserting only to table units_data.

CREATE TABLE data.units ( unit_id uuid NOT NULL, cluster_id INT NOT NULL );

CREATE SEQUENCE ai_value;

CREATE TABLE data.units_data (
    row_id INTEGER NOT NULL DEFAULT nextval('ai_value'),
    unit_id uuid NOT NULL,
    cluster_id INT NOT NULL,
    gps_timestamp TIMESTAMP WITHOUT TIME ZONE NOT NULL,
    gps_lat DOUBLE PRECISION NOT NULL,
    gps_lng DOUBLE PRECISION NOT NULL,
    others hstore,
    PRIMARY KEY ( unit_id,cluster_id )
);

CREATE OR REPLACE FUNCTION generate_all(IN cluster_id INTEGER,  IN unit_id UUID) RETURNS VOID AS $$
DECLARE
    ROK VARCHAR(4);
    DATANODE_ID VARCHAR(1);
    NAZOV_SCHEMY VARCHAR(13);
    NAZOV_TABULKY VARCHAR;
    NAZOV_END_TABULKY VARCHAR;
    NAZOV_ROK_TABULKY VARCHAR;
BEGIN
    -- SELECT 2013 INTO ROK;
    ROK := 2013;
    SELECT cluster_id INTO DATANODE_ID;
    NAZOV_SCHEMY := 'node_'||DATANODE_ID||'_y_'||ROK;
    NAZOV_ROK_TABULKY := 'units_data_2013';
    NAZOV_TABULKY := 'u_'||replace(unit_id::text,'-','');
    NAZOV_END_TABULKY := NAZOV_TABULKY||'_m';
    -- Ide sa vytvarat schema a tabulky
        IF NOT EXISTS(SELECT * FROM information_schema.schemata WHERE schema_name=NAZOV_SCHEMY) THEN
            EXECUTE 'CREATE SCHEMA '||NAZOV_SCHEMY||';';
        END IF;
        IF NOT EXISTS(SELECT * FROM information_schema.tables where table_name=NAZOV_ROK_TABULKY AND table_schema=NAZOV_SCHEMY) THEN
            EXECUTE 'CREATE TABLE '||NAZOV_SCHEMY||'.'||NAZOV_ROK_TABULKY||'( CHECK ( date_part(''year''::text, gps_timestamp)=2013 ) ) INHERITS ( data.units_data );';
        END IF;
        EXECUTE 'CREATE OR REPLACE RULE "year2013_node'||DATANODE_ID||'_choice" AS ON INSERT TO data.units_data WHERE date_part(''year''::text, NEW.gps_timestamp)=2013 AND NEW.cluster_id='||DATANODE_ID||' DO INSTEAD INSERT INTO '||NAZOV_SCHEMY||'.units_data_2013 VALUES(NEW.*);';

        EXECUTE 'CREATE TABLE '||NAZOV_SCHEMY||'.'||NAZOV_TABULKY||'( CHECK ( unit_id='''||unit_id||'''::uuid ) ) INHERITS ('||NAZOV_SCHEMY||'.'||NAZOV_ROK_TABULKY||' );';

        EXECUTE 'CREATE RULE "'||NAZOV_TABULKY||'_choice" AS ON INSERT TO '||NAZOV_SCHEMY||'.units_data_2013 WHERE unit_id='''||unit_id||''' DO INSTEAD INSERT INTO '||NAZOV_SCHEMY||'.'||NAZOV_TABULKY||' VALUES(NEW.*);';
    RETURN;
END
$$ LANGUAGE plpgsql;

CREATE RULE "generate_all_rule" AS ON INSERT TO data.units DO ALSO SELECT generate_all(NEW.cluster_id, NEW.unit_id);

Thank you, best regards
Matej


On 11. 11. 2013 10:11, 鈴木 幸市 wrote:
This does not show anything special.   As I wrote, the messages in question don’t appear so often.     Could you test how many of these messages are written using another case, for example, without using partitioning.   I think partitioning table has nothing to do with the issue but I’d like to know if the case happens in other DMLs and DDLs.

Thank you.
---
Koichi Suzuki

2013/11/11 17:49、Matej Jellus <je...@ts...><mailto:je...@ts...> のメール：

Our situation :
two machines
first - primary GTM, coordinator, datanode
second - standby GTM, coordinator, datanode

GTM configuration below. We are inserting 5 thousand rows per minute, which is too low for us. We need to insert 100 thousand rows per minute.
Starting primary GTM : /gtm_ctl -Z gtm start -D /var/pgxc/data_gtm
Starting standby GTM : /gtm_ctl -Z gtm start -D /var/pgxc/data_gtm
Starting coord : pg_ctl start -D /var/pgxc/ubuntu2_coord -Z coordinator -l /tmp/logfile_ubuntu2_coord

Tables in database are using partitioning. So we have one primary table and we are creating child tables from it. For example :
primary table :
chema - data
table - units_data

child tables :
schema : y_2013_node1 - tables, which will be saved in node 1
   table : u_<unit_id> - child table from data.units_data
schema : y_2013_node2 - tables, which will be saved in node 2
   table : u_<unit_id> - child table from data.units_data

Thank you, best regards
Matej


On 11. 11. 2013 7:49, 鈴木 幸市 wrote:
Message 1. will be written once every 2000 transactions.    The number looks too large.   Message 1 is written in the three cases:

1) When GTM starts,
2) Once every 2000 transactions,
3) When GTM standby is promoted to a master.

Could you let me know how you start your cluster?    We’ve tested XC with DBT-1 and DBT-2 but didn’t have such frequent write at gtm.   You are inserting 5000 raws in a second so 2) should happen twice or three times a second.

Message 2. is written when GTM standby connects to GTM.   Are you using GTM standby?   If so, how you are starting it?

We’re testing XC in different environment but didn’t have such GTM overload.   I’m interested in the situation.

Best;
---
Koichi Suzuki

2013/11/09 0:06、Matej Jellus <je...@ts...><mailto:je...@ts...> のメール：

Hello,

We have problem that postgres xc is very slow, gtm is writing to log a
lot of data. Is it able to be disabled?
The most repeated lines are :

1:140130870839040:2013-11-08 15:42:49.357 CET -LOG:  Saving transaction
restoration info, backed-up gxid: 373106
LOCATION:  GTM_WriteRestorePointXid, gtm_txn.c:2649
( this is maybe 130 times per second )

1:140130879186688:2013-11-08 15:45:37.572 CET -LOG:  Connection
established with GTM standby. - 0x1f91398
LOCATION:  gtm_standby_connect_to_standby_int, gtm_standby.c:400

It is overloading the disk, i/o is 70%.

Now it is inserting 5 thousand inserts per minute. We need more inserts
per minute.

Thank you, best regards
Matej

------------------------------------------------------------------------------
November Webinars for C, C++, Fortran Developers
Accelerate application performance with scalable programming models. Explore
techniques for threading, error checking, porting, and tuning. Get the most
from the latest Intel processors and coprocessors. See abstracts and register
http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
_______________________________________________
Postgres-xc-general mailing list
Pos...@li...<mailto:Pos...@li...>
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general




--

S pozdravom, best regards
                                                                                                               Matej Jellus

<mail_podpis.png>

Technický pracovník - GPS monitoring

TSS Group a.s.
K Zábraniu 1653
911 01 Trenčín
tel:  +421 32 744 5921
fax: +421 32 744 5922
e-mail:je...@ts...<mailto:je...@ts...>
url: www.tssgroup.sk<http://www.tssgroup.sk/>
url: www.gpsmonitoring.sk<http://www.gpsmonitoring.sk/>

Hello,
is it a problem of configuration? Or is it all bad? Should we wait for 
some answers?

Thank you, best regards
Matej

On 11. 11. 2013 15:28, Matej Jellus wrote:
> Hello, here is the process.
>
> When we created table with DISTRIBUTE BY REPLICATION TO NODE ( 
> datanode1 ), it has been inserting maybe 100 000 rows per minute.
> When we created table with DISTRIBUTE BY REPLICATION TO NODE ( 
> datanode1, datanode2 ), it has been inserting maybe 50 000 rows per 
> minute.
> When we created table witch  DISTRIBUTE BY MODULO ( cluster_id ) TO 
> NODE ( datanode1, datanode2 ) and has been writing to only one node, 
> it was inserting 170 000 rows per minute.
>
> But when we created it with partitioning, it has been 5000 rows per 
> minute.
>
> When we insert row into nonpartitioning table, no row was generated 
> into the gtm log file. When we insert row into partitioning table, 
> extra row is inserted into gtm log file : Saving transaction 
> restoration info, backed-up gxid: 409215
> LOCATION:  GTM_WriteRestorePointXid, gtm_txn.c:2649.
>
> When deleting or updating from nonpartitioning table, everything is 
> going quickly. But when we do the same thing with partitioning table, 
> it is 3 - 4 times slower.
>
> Below is example with partitioning. We are importing data to database, 
> so first we insert unit_id into table units. This will generate schema 
> for the year we are inserting, based on the column gps_timestamp. 
> After that it create table for the unit in the schema, which inherits 
> from primary talbe - units_data. Then we are inserting only to table 
> units_data.
>
> CREATE TABLE data.units ( unit_id uuid NOT NULL, cluster_id INT NOT 
> NULL );
>
> CREATE SEQUENCE ai_value;
>
> CREATE TABLE data.units_data (
>     row_id INTEGER NOT NULL DEFAULT nextval('ai_value'),
>     unit_id uuid NOT NULL,
>     cluster_id INT NOT NULL,
>     gps_timestamp TIMESTAMP WITHOUT TIME ZONE NOT NULL,
>     gps_lat DOUBLE PRECISION NOT NULL,
>     gps_lng DOUBLE PRECISION NOT NULL,
>     others hstore,
>     PRIMARY KEY ( unit_id,cluster_id )
> );
>
> CREATE OR REPLACE FUNCTION generate_all(IN cluster_id INTEGER,  IN 
> unit_id UUID) RETURNS VOID AS $$
> DECLARE
>     ROK VARCHAR(4);
>     DATANODE_ID VARCHAR(1);
>     NAZOV_SCHEMY VARCHAR(13);
>     NAZOV_TABULKY VARCHAR;
>     NAZOV_END_TABULKY VARCHAR;
>     NAZOV_ROK_TABULKY VARCHAR;
> BEGIN
>     -- SELECT 2013 INTO ROK;
>     ROK := 2013;
>     SELECT cluster_id INTO DATANODE_ID;
>     NAZOV_SCHEMY := 'node_'||DATANODE_ID||'_y_'||ROK;
>     NAZOV_ROK_TABULKY := 'units_data_2013';
>     NAZOV_TABULKY := 'u_'||replace(unit_id::text,'-','');
>     NAZOV_END_TABULKY := NAZOV_TABULKY||'_m';
>     -- Ide sa vytvarat schema a tabulky
>         IF NOT EXISTS(SELECT * FROM information_schema.schemata WHERE 
> schema_name=NAZOV_SCHEMY) THEN
>             EXECUTE 'CREATE SCHEMA '||NAZOV_SCHEMY||';';
>         END IF;
>         IF NOT EXISTS(SELECT * FROM information_schema.tables where 
> table_name=NAZOV_ROK_TABULKY AND table_schema=NAZOV_SCHEMY) THEN
>             EXECUTE 'CREATE TABLE 
> '||NAZOV_SCHEMY||'.'||NAZOV_ROK_TABULKY||'( CHECK ( 
> date_part(''year''::text, gps_timestamp)=2013 ) ) INHERITS ( 
> data.units_data );';
>         END IF;
>         EXECUTE 'CREATE OR REPLACE RULE 
> "year2013_node'||DATANODE_ID||'_choice" AS ON INSERT TO 
> data.units_data WHERE date_part(''year''::text, 
> NEW.gps_timestamp)=2013 AND NEW.cluster_id='||DATANODE_ID||' DO 
> INSTEAD INSERT INTO '||NAZOV_SCHEMY||'.units_data_2013 VALUES(NEW.*);';
>
>         EXECUTE 'CREATE TABLE '||NAZOV_SCHEMY||'.'||NAZOV_TABULKY||'( 
> CHECK ( unit_id='''||unit_id||'''::uuid ) ) INHERITS 
> ('||NAZOV_SCHEMY||'.'||NAZOV_ROK_TABULKY||' );';
>
>         EXECUTE 'CREATE RULE "'||NAZOV_TABULKY||'_choice" AS ON INSERT 
> TO '||NAZOV_SCHEMY||'.units_data_2013 WHERE unit_id='''||unit_id||''' 
> DO INSTEAD INSERT INTO '||NAZOV_SCHEMY||'.'||NAZOV_TABULKY||' 
> VALUES(NEW.*);';
>     RETURN;
> END
> $$ LANGUAGE plpgsql;
>
> CREATE RULE "generate_all_rule" AS ON INSERT TO data.units DO ALSO 
> SELECT generate_all(NEW.cluster_id, NEW.unit_id);
>
> Thank you, best regards
> Matej
>
>
> On 11. 11. 2013 10:11, 鈴木 幸市 wrote:
>> This does not show anything special.   As I wrote, the messages in 
>> question don’t appear so often. Could you test how many of these 
>> messages are written using another case, for example, without using 
>> partitioning.   I think partitioning table has nothing to do with the 
>> issue but I’d like to know if the case happens in other DMLs and DDLs.
>>
>> Thank you.
>> ---
>> Koichi Suzuki
>>
>> 2013/11/11 17:49、Matej Jellus <je...@ts...> のメール：
>>
>>> Our situation :
>>> two machines
>>> first - primary GTM, coordinator, datanode
>>> second - standby GTM, coordinator, datanode
>>>
>>> GTM configuration below. We are inserting 5 thousand rows per 
>>> minute, which is too low for us. We need to insert 100 thousand rows 
>>> per minute.
>>> Starting primary GTM : /gtm_ctl -Z gtm start -D /var/pgxc/data_gtm
>>> Starting standby GTM : /gtm_ctl -Z gtm start -D /var/pgxc/data_gtm
>>> Starting coord : pg_ctl start -D /var/pgxc/ubuntu2_coord -Z 
>>> coordinator -l /tmp/logfile_ubuntu2_coord
>>>
>>> Tables in database are using partitioning. So we have one primary 
>>> table and we are creating child tables from it. For example :
>>> primary table :
>>> chema - data
>>> table - units_data
>>>
>>> child tables :
>>> schema : y_2013_node1 - tables, which will be saved in node 1
>>>    table : u_<unit_id> - child table from data.units_data
>>> schema : y_2013_node2 - tables, which will be saved in node 2
>>>    table : u_<unit_id> - child table from data.units_data
>>>
>>> Thank you, best regards
>>> Matej
>>>
>>>
>>> On 11. 11. 2013 7:49, 鈴木 幸市 wrote:
>>>> Message 1. will be written once every 2000 transactions.    The 
>>>> number looks too large.   Message 1 is written in the three cases:
>>>>
>>>> 1) When GTM starts,
>>>> 2) Once every 2000 transactions,
>>>> 3) When GTM standby is promoted to a master.
>>>>
>>>> Could you let me know how you start your cluster?    We’ve tested 
>>>> XC with DBT-1 and DBT-2 but didn’t have such frequent write at 
>>>> gtm.   You are inserting 5000 raws in a second so 2) should happen 
>>>> twice or three times a second.
>>>>
>>>> Message 2. is written when GTM standby connects to GTM. Are you 
>>>> using GTM standby?   If so, how you are starting it?
>>>>
>>>> We’re testing XC in different environment but didn’t have such GTM 
>>>> overload.   I’m interested in the situation.
>>>>
>>>> Best;
>>>> ---
>>>> Koichi Suzuki
>>>>
>>>> 2013/11/09 0:06、Matej Jellus <je...@ts...> のメール：
>>>>
>>>>> Hello,
>>>>>
>>>>> We have problem that postgres xc is very slow, gtm is writing to 
>>>>> log a
>>>>> lot of data. Is it able to be disabled?
>>>>> The most repeated lines are :
>>>>>
>>>>> 1:140130870839040:2013-11-08 15:42:49.357 CET -LOG: Saving 
>>>>> transaction
>>>>> restoration info, backed-up gxid: 373106
>>>>> LOCATION:  GTM_WriteRestorePointXid, gtm_txn.c:2649
>>>>> ( this is maybe 130 times per second )
>>>>>
>>>>> 1:140130879186688:2013-11-08 15:45:37.572 CET -LOG: Connection
>>>>> established with GTM standby. - 0x1f91398
>>>>> LOCATION:  gtm_standby_connect_to_standby_int, gtm_standby.c:400
>>>>>
>>>>> It is overloading the disk, i/o is 70%.
>>>>>
>>>>> Now it is inserting 5 thousand inserts per minute. We need more 
>>>>> inserts
>>>>> per minute.
>>>>>
>>>>> Thank you, best regards
>>>>> Matej
>>>>>
>>>>> ------------------------------------------------------------------------------ 
>>>>>
>>>>> November Webinars for C, C++, Fortran Developers
>>>>> Accelerate application performance with scalable programming 
>>>>> models. Explore
>>>>> techniques for threading, error checking, porting, and tuning. Get 
>>>>> the most
>>>>> from the latest Intel processors and coprocessors. See abstracts 
>>>>> and register
>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk 
>>>>>
>>>>> _______________________________________________
>>>>> Postgres-xc-general mailing list
>>>>> Pos...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>>>
>>>
>

-- 

S pozdravom, best regards
*Matej Jellus*

TSS Group a.s.

*Technický pracovník - GPS monitoring*

TSS Group a.s.
K Zábraniu 1653
911 01 Trenčín
tel:  +421 32 744 5921
fax: +421 32 744 5922
e-mail:je...@ts... <mailto:je...@ts...>
url: www.tssgroup.sk <http://www.tssgroup.sk>
url: www.gpsmonitoring.sk <http://www.gpsmonitoring.sk>

Thanks.   I'll be looking forward to your test result.   I talked with
Masataka and he was interested that no relevant messages were found in
dmsg.

Regards;
---
Koichi Suzuki


2013/11/19 Sandeep Gupta <gup...@gm...>:
> Hi Koichi,
>
>  Thanks for taking a look. I did some further debugging and I won't rule
> out the possibility of the the OS killing some postgres processes which
> leads to copy execution reaching in a inconsistent state.
> Will do some further tests to confirm this as well.
>
> -Sandeep
>
>
>
>
> On Mon, Nov 18, 2013 at 1:06 AM, Koichi Suzuki <koi...@gm...>
> wrote:
>>
>> This happens when the message is not null-terminated properly.   Most
>> case is a bug.   In this case, different from vanilla PostgreSQL, copy
>> data is involved in.    Please let me find a time to look into it.
>>
>> Regards;
>> ---
>> Koichi Suzuki
>>
>>
>> 2013/11/16 Sandeep Gupta <gup...@gm...>:
>> > Forgot to mention  that the error is still the same "invalid string in
>> > message".
>> >
>> > -Sandeep
>> >
>> >
>> >
>> > On Fri, Nov 15, 2013 at 8:27 PM, Sandeep Gupta <gup...@gm...>
>> > wrote:
>> >>
>> >> Hi Koichi,
>> >>
>> >>  We tried the  copy with the patch but it did not fix the problem for
>> >> us.
>> >> If you need any additional details then let me know.
>> >>
>> >> Best,
>> >> -Sandeep
>> >>
>> >>
>> >> On Fri, Nov 15, 2013 at 6:13 AM, Sandeep Gupta
>> >> <gup...@gm...>
>> >> wrote:
>> >>>
>> >>> Hi Koichi,
>> >>>
>> >>>  Many thanks for sharing the patch. We would test this in our
>> >>> environment
>> >>> and would report back on the result.
>> >>>
>> >>> Regarding the very high memory consumption, it is not the coordinator
>> >>> that sees the high memory usage.
>> >>> The setup is following: We have one coordinator and 4 datanodes. All
>> >>> the
>> >>> datanodes run on separate machine
>> >>> then the coordinator.  It is the machine where the datanodes are
>> >>> running
>> >>> that have 48GB memory usage.
>> >>> The other caveat is that that "top" doesn't show that it is the
>> >>> datanodes
>> >>> processes using the memory.
>> >>>
>> >>> In any case, the last run of copy even with the high usage went
>> >>> through
>> >>> fine.
>> >>>
>> >>>
>> >>> Best,
>> >>> Sandeep
>> >>>
>> >>>
>> >>>
>> >>> On Fri, Nov 15, 2013 at 2:29 AM, Koichi Suzuki <koi...@gm...>
>> >>> wrote:
>> >>>>
>> >>>> Yes, I've already found this kind of failure.   When disk write is
>> >>>> quite slower than read, coordinator overshoots.  However, in this
>> >>>> case, I thought we will see that palloc() overflows.   If each piece
>> >>>> of COPY TO text data is less than 1GB, then each COPY TO succeeds.
>> >>>> If the write is much slower than read,  each coordinator will
>> >>>> allocate
>> >>>> memory for each COPY TO cache going to datanodes, ending up with
>> >>>> enormous amount of memory consumption and have a good chance to run
>> >>>> OOM killer.    Your memory usage looks enormous.    Maybe you have
>> >>>> many COPY TO instances each of which allocates too much memory.
>> >>>>
>> >>>> Attached patch fixes COPY TO.   Because we have a chance to have
>> >>>> similar overshoots in COPY FROM, I'd like to write a patch to fix
>> >>>> this
>> >>>> too before committing.   This is for the current master but will work
>> >>>> with 1.1.x (even with 1.0.x hopefully).
>> >>>>
>> >>>> The patch looks to be fine.   I asked to test it in such an
>> >>>> environment and they reported the patch fixed their issue.
>> >>>>
>> >>>> I'm not sure but hope this works in your case.   May reduce memory
>> >>>> consumption drastically.   Please let me know the result.
>> >>>>
>> >>>> I have to allocate some of my resource to fix the same issue in COPY
>> >>>> FROM.    I have another request to fix this so if the patch works for
>> >>>> you, I may be able to commit only COPY TO fix first.
>> >>>>
>> >>>> Regards;
>> >>>> ---
>> >>>> Koichi Suzuki
>> >>>>
>> >>>>
>> >>>> 2013/11/15 Sandeep Gupta <gup...@gm...>:
>> >>>> > Hi Koichi,
>> >>>> >
>> >>>> >  I am using the copy command over the text file to populate a
>> >>>> > table.
>> >>>> >
>> >>>> > Sorry for the confusion. I meant 628 -- 638 lines in the
>> >>>> > pqformat.c.
>> >>>> > Let me explain this a bit more. I get  error in some of the
>> >>>> > datanode
>> >>>> > logfiles
>> >>>> > as "invalid string in message". The message is mentioned only in
>> >>>> > pqformat.c
>> >>>> > at line 628. Although I am not able to trace back as to where
>> >>>> > error
>> >>>> > may
>> >>>> > have first ordered in the datanode that triggered this error.
>> >>>> >
>> >>>> > In order to debug this I started printing out the rows the
>> >>>> > coordinator
>> >>>> > is
>> >>>> > sending to the datanode (in function DataNodeCopyIn). For some
>> >>>> > reason
>> >>>> > the
>> >>>> > error invalid string in messge error has  not happened so far.  It
>> >>>> > seems
>> >>>> > there is some issue with the flow control of message between
>> >>>> > coordinator and
>> >>>> > datanode.
>> >>>> > Hope this helps in narrowing down what is causing the problem in
>> >>>> > the
>> >>>> > first
>> >>>> > place.
>> >>>> >
>> >>>> >
>> >>>> > However, the new problem I see is that the memory usage goes very
>> >>>> > high. Of
>> >>>> > the 48 GB almost all of the memory is getting used. Is this normal?
>> >>>> > If not can you give suggestion as to how to keep the memory
>> >>>> > consumption
>> >>>> > low.
>> >>>> >
>> >>>> > -Sandeep
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > On Thu, Nov 14, 2013 at 8:57 PM, Koichi Suzuki
>> >>>> > <koi...@gm...>
>> >>>> > wrote:
>> >>>> >>
>> >>>> >> Could you let me know what "line 682 -- 637" means?
>> >>>> >>
>> >>>> >> Because any communication between coordinator and datanode is done
>> >>>> >> using libpq, I don't think XC populates information using raw
>> >>>> >> stringinfo.   In fact, from datanodes, coordinator is just a
>> >>>> >> client
>> >>>> >> except for supplying XID and snapshot.   However,  COPY TO uses
>> >>>> >> send()/receive(), not through libpq, to send data to datanode.  In
>> >>>> >> this case, if incoming data file contains NULL in the data, such
>> >>>> >> data
>> >>>> >> may be sent via send() to datanodes.   COPY TO assumes incoming
>> >>>> >> data
>> >>>> >> in a text format.   Do you think you have such situations?
>> >>>> >>
>> >>>> >> Analysis may take a bit but let me try.
>> >>>> >>
>> >>>> >> Regards;
>> >>>> >> ---
>> >>>> >> Koichi Suzuki
>> >>>> >>
>> >>>> >>
>> >>>> >> 2013/11/15 Sandeep Gupta <gup...@gm...>:
>> >>>> >> > Hi Koichi, Masataka,
>> >>>> >> >
>> >>>> >> >  Thanks for taking a look. I double checked both memory usage
>> >>>> >> > and
>> >>>> >> > the
>> >>>> >> > dmesg
>> >>>> >> > log. The kill is not happening due to the out of memory. The
>> >>>> >> > kernel
>> >>>> >> > doesn't
>> >>>> >> > show any processes being killed.
>> >>>> >> >
>> >>>> >> > My current suspicion is that something amiss is happening over
>> >>>> >> > the
>> >>>> >> > network.
>> >>>> >> > This happens consistently when performing copy over large
>> >>>> >> > datasets.
>> >>>> >> > Few
>> >>>> >> > of
>> >>>> >> > datanodes end up with "invalid string in message".
>> >>>> >> > Going over the code in pqformat.c this happens because the
>> >>>> >> > invariant
>> >>>> >> > "StringInfo is guaranteed to have a trailing null byte" is not
>> >>>> >> > valid
>> >>>> >> > (line
>> >>>> >> > 682 -- 637). I am not sure why this happens.
>> >>>> >> >
>> >>>> >> >
>> >>>> >> > After this the datanodes do something illegal (probably buffer
>> >>>> >> > overflow
>> >>>> >> > on
>> >>>> >> > the port..just a guess) that the OS decides to issue a sigkill.
>> >>>> >> > I
>> >>>> >> > am
>> >>>> >> > also
>> >>>> >> > not sure why the OS is not logging this killed operation.
>> >>>> >> >
>> >>>> >> >
>> >>>> >> > Please let me know of  suggestions you may have regarding the
>> >>>> >> > "invalid
>> >>>> >> > string in message". In particular, where in the code is
>> >>>> >> > StringInfo
>> >>>> >> > msg
>> >>>> >> > is
>> >>>> >> > getting populated i.e. which code reads the message over the
>> >>>> >> > network and
>> >>>> >> > keeps it ready for the copy operation on the datanode. Any other
>> >>>> >> > suggestion
>> >>>> >> > would also be helpful to trace this bug.
>> >>>> >> >
>> >>>> >> > Best,
>> >>>> >> > Sandeep
>> >>>> >> >
>> >>>> >> >
>> >>>> >> >
>> >>>> >> >
>> >>>> >> >
>> >>>> >> >
>> >>>> >> > On Thu, Nov 14, 2013 at 10:25 AM, Masataka Saito
>> >>>> >> > <pg...@gm...>
>> >>>> >> > wrote:
>> >>>> >> >>
>> >>>> >> >> Hi,
>> >>>> >> >>
>> >>>> >> >> I think I may be onto something.
>> >>>> >> >>
>> >>>> >> >> When the memory is exhausted, the Linux kernel kills process
>> >>>> >> >> randomly
>> >>>> >> >> by SIGKILL.
>> >>>> >> >> The mechanism is called OOM(Out Of Memory) killer.
>> >>>> >> >> OOM killer logs its activity to kernel mesage buffer.
>> >>>> >> >>
>> >>>> >> >> Could you check display message (dmesg) and memory resource?
>> >>>> >> >>
>> >>>> >> >> Regards.
>> >>>> >> >>
>> >>>> >> >> On Tue, Nov 12, 2013 at 3:55 AM, Sandeep Gupta
>> >>>> >> >> <gup...@gm...>
>> >>>> >> >> wrote:
>> >>>> >> >> > Hi Koichi,
>> >>>> >> >> >
>> >>>> >> >> >    It is a bit of mystery  because it does not happen
>> >>>> >> >> > consistently.
>> >>>> >> >> > Thanks
>> >>>> >> >> > for the clarification though. It is indeed helpful.
>> >>>> >> >> >
>> >>>> >> >> > -Sandeep
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> > On Mon, Nov 11, 2013 at 1:58 AM, 鈴木 幸市
>> >>>> >> >> > <ko...@in...>
>> >>>> >> >> > wrote:
>> >>>> >> >> >>
>> >>>> >> >> >> Someone is sending SIGKILL to coordinator/datanode backend
>> >>>> >> >> >> processes.
>> >>>> >> >> >> Although XC (originally PG) code has a handler for SIGKILL,
>> >>>> >> >> >> I
>> >>>> >> >> >> didn’t
>> >>>> >> >> >> found
>> >>>> >> >> >> any code in XC sending SIGKILL to other processes.   I’m
>> >>>> >> >> >> afraid
>> >>>> >> >> >> there
>> >>>> >> >> >> could
>> >>>> >> >> >> be another process sending SIGKILL to them.
>> >>>> >> >> >>
>> >>>> >> >> >> Best Regards;
>> >>>> >> >> >> ---
>> >>>> >> >> >> Koichi Suzuki
>> >>>> >> >> >>
>> >>>> >> >> >> 2013/11/09 2:54、Sandeep Gupta <gup...@gm...>
>> >>>> >> >> >> のメール：
>> >>>> >> >> >>
>> >>>> >> >> >> > I am running single instance of postgres-xc with  several
>> >>>> >> >> >> > datanode
>> >>>> >> >> >> > Each data node runs on its own machine.
>> >>>> >> >> >> > After instantiating the cluster the database sits idle. I
>> >>>> >> >> >> > do
>> >>>> >> >> >> > not
>> >>>> >> >> >> > perform
>> >>>> >> >> >> > any create table or insert operations. However, after some
>> >>>> >> >> >> > time
>> >>>> >> >> >> > all
>> >>>> >> >> >> > the
>> >>>> >> >> >> > datanode  automatically shuts down (log messages
>> >>>> >> >> >> > attached).
>> >>>> >> >> >> >
>> >>>> >> >> >> > Any pointers as to why this maybe happening would be very
>> >>>> >> >> >> > useful.
>> >>>> >> >> >> >
>> >>>> >> >> >> > -Sandeep
>> >>>> >> >> >> >
>> >>>> >> >> >> > The log files across all datanode contain various error
>> >>>> >> >> >> > messages.
>> >>>> >> >> >> > These
>> >>>> >> >> >> > include:
>> >>>> >> >> >> >
>> >>>> >> >> >> >
>> >>>> >> >> >> > LOG:  statistics collector process (PID 23073) was
>> >>>> >> >> >> > terminated
>> >>>> >> >> >> > by
>> >>>> >> >> >> > signal
>> >>>> >> >> >> > 9: Killed
>> >>>> >> >> >> > LOG:  WAL writer process (PID 23071) was terminated by
>> >>>> >> >> >> > signal
>> >>>> >> >> >> > 9:
>> >>>> >> >> >> > Killed
>> >>>> >> >> >> > LOG:  terminating any other active server processes
>> >>>> >> >> >> > WARNING:  terminating connection because of crash of
>> >>>> >> >> >> > another
>> >>>> >> >> >> > server
>> >>>> >> >> >> > process
>> >>>> >> >> >> > DETAIL:  The postmaster has commanded this server process
>> >>>> >> >> >> > to
>> >>>> >> >> >> > roll
>> >>>> >> >> >> > back
>> >>>> >> >> >> > the current transaction and exit, because another server
>> >>>> >> >> >> > process
>> >>>> >> >> >> > exited
>> >>>> >> >> >> > abnormally and possibly corrupted shared memory.
>> >>>> >> >> >> > HINT:  In a moment you should be able to reconnect to the
>> >>>> >> >> >> > database
>> >>>> >> >> >> > and
>> >>>> >> >> >> > repeat your command.
>> >>>> >> >> >> > LOG:  all server processes terminated; reinitializing
>> >>>> >> >> >> >
>> >>>> >> >> >> >
>> >>>> >> >> >> > LOG:  checkpointer process (PID 21406) was terminated by
>> >>>> >> >> >> > signal 9:
>> >>>> >> >> >> > Killed
>> >>>> >> >> >> > LOG:  terminating any other active server processes
>> >>>> >> >> >> > WARNING:  terminating connection because of crash of
>> >>>> >> >> >> > another
>> >>>> >> >> >> > server
>> >>>> >> >> >> > process
>> >>>> >> >> >> > DETAIL:  The postmaster has commanded this server process
>> >>>> >> >> >> > to
>> >>>> >> >> >> > roll
>> >>>> >> >> >> > back
>> >>>> >> >> >> > the current transaction and exit, because another server
>> >>>> >> >> >> > process
>> >>>> >> >> >> > exited
>> >>>> >> >> >> > abnormally and possibly corrupted shared memory.
>> >>>> >> >> >> > LOG:  statistics collector process (PID 28881) was
>> >>>> >> >> >> > terminated
>> >>>> >> >> >> > by
>> >>>> >> >> >> > signal
>> >>>> >> >> >> > 9: Killed
>> >>>> >> >> >> > LOG:  autovacuum launcher process (PID 28880) was
>> >>>> >> >> >> > terminated
>> >>>> >> >> >> > by
>> >>>> >> >> >> > signal
>> >>>> >> >> >> > 9: Killed
>> >>>> >> >> >> > LOG:  terminating any other active server processes
>> >>>> >> >> >> >
>> >>>> >> >> >> >
>> >>>> >> >> >> >
>> >>>> >> >> >> >
>> >>>> >> >> >> >
>> >>>> >> >> >> >
>> >>>> >> >> >> >
>> >>>> >> >> >> >
>> >>>> >> >> >> >
>> >>>> >> >> >> >
>> >>>> >> >> >> > ------------------------------------------------------------------------------
>> >>>> >> >> >> > November Webinars for C, C++, Fortran Developers
>> >>>> >> >> >> > Accelerate application performance with scalable
>> >>>> >> >> >> > programming
>> >>>> >> >> >> > models.
>> >>>> >> >> >> > Explore
>> >>>> >> >> >> > techniques for threading, error checking, porting, and
>> >>>> >> >> >> > tuning. Get
>> >>>> >> >> >> > the
>> >>>> >> >> >> > most
>> >>>> >> >> >> > from the latest Intel processors and coprocessors. See
>> >>>> >> >> >> > abstracts
>> >>>> >> >> >> > and
>> >>>> >> >> >> > register
>> >>>> >> >> >> >
>> >>>> >> >> >> >
>> >>>> >> >> >> >
>> >>>> >> >> >> >
>> >>>> >> >> >> >
>> >>>> >> >> >> > http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk_______________________________________________
>> >>>> >> >> >> > Postgres-xc-general mailing list
>> >>>> >> >> >> > Pos...@li...
>> >>>> >> >> >> >
>> >>>> >> >> >> >
>> >>>> >> >> >> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>> >>>> >> >> >>
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> > ------------------------------------------------------------------------------
>> >>>> >> >> > November Webinars for C, C++, Fortran Developers
>> >>>> >> >> > Accelerate application performance with scalable programming
>> >>>> >> >> > models.
>> >>>> >> >> > Explore
>> >>>> >> >> > techniques for threading, error checking, porting, and
>> >>>> >> >> > tuning.
>> >>>> >> >> > Get
>> >>>> >> >> > the
>> >>>> >> >> > most
>> >>>> >> >> > from the latest Intel processors and coprocessors. See
>> >>>> >> >> > abstracts
>> >>>> >> >> > and
>> >>>> >> >> > register
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> > http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
>> >>>> >> >> > _______________________________________________
>> >>>> >> >> > Postgres-xc-general mailing list
>> >>>> >> >> > Pos...@li...
>> >>>> >> >> >
>> >>>> >> >> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>> >>>> >> >> >
>> >>>> >> >
>> >>>> >> >
>> >>>> >> >
>> >>>> >> >
>> >>>> >> >
>> >>>> >> >
>> >>>> >> > ------------------------------------------------------------------------------
>> >>>> >> > DreamFactory - Open Source REST & JSON Services for HTML5 &
>> >>>> >> > Native
>> >>>> >> > Apps
>> >>>> >> > OAuth, Users, Roles, SQL, NoSQL, BLOB Storage and External API
>> >>>> >> > Access
>> >>>> >> > Free app hosting. Or install the open source package on any LAMP
>> >>>> >> > server.
>> >>>> >> > Sign up and see examples for AngularJS, jQuery, Sencha Touch and
>> >>>> >> > Native!
>> >>>> >> >
>> >>>> >> >
>> >>>> >> >
>> >>>> >> > http://pubads.g.doubleclick.net/gampad/clk?id=63469471&iu=/4140/ostg.clktrk
>> >>>> >> > _______________________________________________
>> >>>> >> > Postgres-xc-general mailing list
>> >>>> >> > Pos...@li...
>> >>>> >> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>> >>>> >> >
>> >>>> >
>> >>>> >
>> >>>
>> >>>
>> >>
>> >
>
>

Hi Koichi,

 Thanks for taking a look. I did some further debugging and I won't rule
out the possibility of the the OS killing some postgres processes which
leads to copy execution reaching in a inconsistent state.
Will do some further tests to confirm this as well.

-Sandeep




On Mon, Nov 18, 2013 at 1:06 AM, Koichi Suzuki <koi...@gm...>wrote:

> This happens when the message is not null-terminated properly.   Most
> case is a bug.   In this case, different from vanilla PostgreSQL, copy
> data is involved in.    Please let me find a time to look into it.
>
> Regards;
> ---
> Koichi Suzuki
>
>
> 2013/11/16 Sandeep Gupta <gup...@gm...>:
> > Forgot to mention  that the error is still the same "invalid string in
> > message".
> >
> > -Sandeep
> >
> >
> >
> > On Fri, Nov 15, 2013 at 8:27 PM, Sandeep Gupta <gup...@gm...>
> > wrote:
> >>
> >> Hi Koichi,
> >>
> >>  We tried the  copy with the patch but it did not fix the problem for
> us.
> >> If you need any additional details then let me know.
> >>
> >> Best,
> >> -Sandeep
> >>
> >>
> >> On Fri, Nov 15, 2013 at 6:13 AM, Sandeep Gupta <gup...@gm...
> >
> >> wrote:
> >>>
> >>> Hi Koichi,
> >>>
> >>>  Many thanks for sharing the patch. We would test this in our
> environment
> >>> and would report back on the result.
> >>>
> >>> Regarding the very high memory consumption, it is not the coordinator
> >>> that sees the high memory usage.
> >>> The setup is following: We have one coordinator and 4 datanodes. All
> the
> >>> datanodes run on separate machine
> >>> then the coordinator.  It is the machine where the datanodes are
> running
> >>> that have 48GB memory usage.
> >>> The other caveat is that that "top" doesn't show that it is the
> datanodes
> >>> processes using the memory.
> >>>
> >>> In any case, the last run of copy even with the high usage went through
> >>> fine.
> >>>
> >>>
> >>> Best,
> >>> Sandeep
> >>>
> >>>
> >>>
> >>> On Fri, Nov 15, 2013 at 2:29 AM, Koichi Suzuki <koi...@gm...>
> >>> wrote:
> >>>>
> >>>> Yes, I've already found this kind of failure.   When disk write is
> >>>> quite slower than read, coordinator overshoots.  However, in this
> >>>> case, I thought we will see that palloc() overflows.   If each piece
> >>>> of COPY TO text data is less than 1GB, then each COPY TO succeeds.
> >>>> If the write is much slower than read,  each coordinator will allocate
> >>>> memory for each COPY TO cache going to datanodes, ending up with
> >>>> enormous amount of memory consumption and have a good chance to run
> >>>> OOM killer.    Your memory usage looks enormous.    Maybe you have
> >>>> many COPY TO instances each of which allocates too much memory.
> >>>>
> >>>> Attached patch fixes COPY TO.   Because we have a chance to have
> >>>> similar overshoots in COPY FROM, I'd like to write a patch to fix this
> >>>> too before committing.   This is for the current master but will work
> >>>> with 1.1.x (even with 1.0.x hopefully).
> >>>>
> >>>> The patch looks to be fine.   I asked to test it in such an
> >>>> environment and they reported the patch fixed their issue.
> >>>>
> >>>> I'm not sure but hope this works in your case.   May reduce memory
> >>>> consumption drastically.   Please let me know the result.
> >>>>
> >>>> I have to allocate some of my resource to fix the same issue in COPY
> >>>> FROM.    I have another request to fix this so if the patch works for
> >>>> you, I may be able to commit only COPY TO fix first.
> >>>>
> >>>> Regards;
> >>>> ---
> >>>> Koichi Suzuki
> >>>>
> >>>>
> >>>> 2013/11/15 Sandeep Gupta <gup...@gm...>:
> >>>> > Hi Koichi,
> >>>> >
> >>>> >  I am using the copy command over the text file to populate a table.
> >>>> >
> >>>> > Sorry for the confusion. I meant 628 -- 638 lines in the pqformat.c.
> >>>> > Let me explain this a bit more. I get  error in some of the datanode
> >>>> > logfiles
> >>>> > as "invalid string in message". The message is mentioned only in
> >>>> > pqformat.c
> >>>> > at line 628. Although I am not able to trace back as to where  error
> >>>> > may
> >>>> > have first ordered in the datanode that triggered this error.
> >>>> >
> >>>> > In order to debug this I started printing out the rows the
> coordinator
> >>>> > is
> >>>> > sending to the datanode (in function DataNodeCopyIn). For some
> reason
> >>>> > the
> >>>> > error invalid string in messge error has  not happened so far.  It
> >>>> > seems
> >>>> > there is some issue with the flow control of message between
> >>>> > coordinator and
> >>>> > datanode.
> >>>> > Hope this helps in narrowing down what is causing the problem in the
> >>>> > first
> >>>> > place.
> >>>> >
> >>>> >
> >>>> > However, the new problem I see is that the memory usage goes very
> >>>> > high. Of
> >>>> > the 48 GB almost all of the memory is getting used. Is this normal?
> >>>> > If not can you give suggestion as to how to keep the memory
> >>>> > consumption
> >>>> > low.
> >>>> >
> >>>> > -Sandeep
> >>>> >
> >>>> >
> >>>> >
> >>>> > On Thu, Nov 14, 2013 at 8:57 PM, Koichi Suzuki <
> koi...@gm...>
> >>>> > wrote:
> >>>> >>
> >>>> >> Could you let me know what "line 682 -- 637" means?
> >>>> >>
> >>>> >> Because any communication between coordinator and datanode is done
> >>>> >> using libpq, I don't think XC populates information using raw
> >>>> >> stringinfo.   In fact, from datanodes, coordinator is just a client
> >>>> >> except for supplying XID and snapshot.   However,  COPY TO uses
> >>>> >> send()/receive(), not through libpq, to send data to datanode.  In
> >>>> >> this case, if incoming data file contains NULL in the data, such
> data
> >>>> >> may be sent via send() to datanodes.   COPY TO assumes incoming
> data
> >>>> >> in a text format.   Do you think you have such situations?
> >>>> >>
> >>>> >> Analysis may take a bit but let me try.
> >>>> >>
> >>>> >> Regards;
> >>>> >> ---
> >>>> >> Koichi Suzuki
> >>>> >>
> >>>> >>
> >>>> >> 2013/11/15 Sandeep Gupta <gup...@gm...>:
> >>>> >> > Hi Koichi, Masataka,
> >>>> >> >
> >>>> >> >  Thanks for taking a look. I double checked both memory usage and
> >>>> >> > the
> >>>> >> > dmesg
> >>>> >> > log. The kill is not happening due to the out of memory. The
> kernel
> >>>> >> > doesn't
> >>>> >> > show any processes being killed.
> >>>> >> >
> >>>> >> > My current suspicion is that something amiss is happening over
> the
> >>>> >> > network.
> >>>> >> > This happens consistently when performing copy over large
> datasets.
> >>>> >> > Few
> >>>> >> > of
> >>>> >> > datanodes end up with "invalid string in message".
> >>>> >> > Going over the code in pqformat.c this happens because the
> >>>> >> > invariant
> >>>> >> > "StringInfo is guaranteed to have a trailing null byte" is not
> >>>> >> > valid
> >>>> >> > (line
> >>>> >> > 682 -- 637). I am not sure why this happens.
> >>>> >> >
> >>>> >> >
> >>>> >> > After this the datanodes do something illegal (probably buffer
> >>>> >> > overflow
> >>>> >> > on
> >>>> >> > the port..just a guess) that the OS decides to issue a sigkill.
>  I
> >>>> >> > am
> >>>> >> > also
> >>>> >> > not sure why the OS is not logging this killed operation.
> >>>> >> >
> >>>> >> >
> >>>> >> > Please let me know of  suggestions you may have regarding the
> >>>> >> > "invalid
> >>>> >> > string in message". In particular, where in the code is
> StringInfo
> >>>> >> > msg
> >>>> >> > is
> >>>> >> > getting populated i.e. which code reads the message over the
> >>>> >> > network and
> >>>> >> > keeps it ready for the copy operation on the datanode. Any other
> >>>> >> > suggestion
> >>>> >> > would also be helpful to trace this bug.
> >>>> >> >
> >>>> >> > Best,
> >>>> >> > Sandeep
> >>>> >> >
> >>>> >> >
> >>>> >> >
> >>>> >> >
> >>>> >> >
> >>>> >> >
> >>>> >> > On Thu, Nov 14, 2013 at 10:25 AM, Masataka Saito <
> pg...@gm...>
> >>>> >> > wrote:
> >>>> >> >>
> >>>> >> >> Hi,
> >>>> >> >>
> >>>> >> >> I think I may be onto something.
> >>>> >> >>
> >>>> >> >> When the memory is exhausted, the Linux kernel kills process
> >>>> >> >> randomly
> >>>> >> >> by SIGKILL.
> >>>> >> >> The mechanism is called OOM(Out Of Memory) killer.
> >>>> >> >> OOM killer logs its activity to kernel mesage buffer.
> >>>> >> >>
> >>>> >> >> Could you check display message (dmesg) and memory resource?
> >>>> >> >>
> >>>> >> >> Regards.
> >>>> >> >>
> >>>> >> >> On Tue, Nov 12, 2013 at 3:55 AM, Sandeep Gupta
> >>>> >> >> <gup...@gm...>
> >>>> >> >> wrote:
> >>>> >> >> > Hi Koichi,
> >>>> >> >> >
> >>>> >> >> >    It is a bit of mystery  because it does not happen
> >>>> >> >> > consistently.
> >>>> >> >> > Thanks
> >>>> >> >> > for the clarification though. It is indeed helpful.
> >>>> >> >> >
> >>>> >> >> > -Sandeep
> >>>> >> >> >
> >>>> >> >> >
> >>>> >> >> >
> >>>> >> >> >
> >>>> >> >> > On Mon, Nov 11, 2013 at 1:58 AM, 鈴木 幸市
> >>>> >> >> > <ko...@in...>
> >>>> >> >> > wrote:
> >>>> >> >> >>
> >>>> >> >> >> Someone is sending SIGKILL to coordinator/datanode backend
> >>>> >> >> >> processes.
> >>>> >> >> >> Although XC (originally PG) code has a handler for SIGKILL, I
> >>>> >> >> >> didn’t
> >>>> >> >> >> found
> >>>> >> >> >> any code in XC sending SIGKILL to other processes.   I’m
> afraid
> >>>> >> >> >> there
> >>>> >> >> >> could
> >>>> >> >> >> be another process sending SIGKILL to them.
> >>>> >> >> >>
> >>>> >> >> >> Best Regards;
> >>>> >> >> >> ---
> >>>> >> >> >> Koichi Suzuki
> >>>> >> >> >>
> >>>> >> >> >> 2013/11/09 2:54、Sandeep Gupta <gup...@gm...>
> のメール：
> >>>> >> >> >>
> >>>> >> >> >> > I am running single instance of postgres-xc with  several
> >>>> >> >> >> > datanode
> >>>> >> >> >> > Each data node runs on its own machine.
> >>>> >> >> >> > After instantiating the cluster the database sits idle. I
> do
> >>>> >> >> >> > not
> >>>> >> >> >> > perform
> >>>> >> >> >> > any create table or insert operations. However, after some
> >>>> >> >> >> > time
> >>>> >> >> >> > all
> >>>> >> >> >> > the
> >>>> >> >> >> > datanode  automatically shuts down (log messages attached).
> >>>> >> >> >> >
> >>>> >> >> >> > Any pointers as to why this maybe happening would be very
> >>>> >> >> >> > useful.
> >>>> >> >> >> >
> >>>> >> >> >> > -Sandeep
> >>>> >> >> >> >
> >>>> >> >> >> > The log files across all datanode contain various error
> >>>> >> >> >> > messages.
> >>>> >> >> >> > These
> >>>> >> >> >> > include:
> >>>> >> >> >> >
> >>>> >> >> >> >
> >>>> >> >> >> > LOG:  statistics collector process (PID 23073) was
> terminated
> >>>> >> >> >> > by
> >>>> >> >> >> > signal
> >>>> >> >> >> > 9: Killed
> >>>> >> >> >> > LOG:  WAL writer process (PID 23071) was terminated by
> signal
> >>>> >> >> >> > 9:
> >>>> >> >> >> > Killed
> >>>> >> >> >> > LOG:  terminating any other active server processes
> >>>> >> >> >> > WARNING:  terminating connection because of crash of
> another
> >>>> >> >> >> > server
> >>>> >> >> >> > process
> >>>> >> >> >> > DETAIL:  The postmaster has commanded this server process
> to
> >>>> >> >> >> > roll
> >>>> >> >> >> > back
> >>>> >> >> >> > the current transaction and exit, because another server
> >>>> >> >> >> > process
> >>>> >> >> >> > exited
> >>>> >> >> >> > abnormally and possibly corrupted shared memory.
> >>>> >> >> >> > HINT:  In a moment you should be able to reconnect to the
> >>>> >> >> >> > database
> >>>> >> >> >> > and
> >>>> >> >> >> > repeat your command.
> >>>> >> >> >> > LOG:  all server processes terminated; reinitializing
> >>>> >> >> >> >
> >>>> >> >> >> >
> >>>> >> >> >> > LOG:  checkpointer process (PID 21406) was terminated by
> >>>> >> >> >> > signal 9:
> >>>> >> >> >> > Killed
> >>>> >> >> >> > LOG:  terminating any other active server processes
> >>>> >> >> >> > WARNING:  terminating connection because of crash of
> another
> >>>> >> >> >> > server
> >>>> >> >> >> > process
> >>>> >> >> >> > DETAIL:  The postmaster has commanded this server process
> to
> >>>> >> >> >> > roll
> >>>> >> >> >> > back
> >>>> >> >> >> > the current transaction and exit, because another server
> >>>> >> >> >> > process
> >>>> >> >> >> > exited
> >>>> >> >> >> > abnormally and possibly corrupted shared memory.
> >>>> >> >> >> > LOG:  statistics collector process (PID 28881) was
> terminated
> >>>> >> >> >> > by
> >>>> >> >> >> > signal
> >>>> >> >> >> > 9: Killed
> >>>> >> >> >> > LOG:  autovacuum launcher process (PID 28880) was
> terminated
> >>>> >> >> >> > by
> >>>> >> >> >> > signal
> >>>> >> >> >> > 9: Killed
> >>>> >> >> >> > LOG:  terminating any other active server processes
> >>>> >> >> >> >
> >>>> >> >> >> >
> >>>> >> >> >> >
> >>>> >> >> >> >
> >>>> >> >> >> >
> >>>> >> >> >> >
> >>>> >> >> >> >
> >>>> >> >> >> >
> >>>> >> >> >> >
> >>>> >> >> >> >
> ------------------------------------------------------------------------------
> >>>> >> >> >> > November Webinars for C, C++, Fortran Developers
> >>>> >> >> >> > Accelerate application performance with scalable
> programming
> >>>> >> >> >> > models.
> >>>> >> >> >> > Explore
> >>>> >> >> >> > techniques for threading, error checking, porting, and
> >>>> >> >> >> > tuning. Get
> >>>> >> >> >> > the
> >>>> >> >> >> > most
> >>>> >> >> >> > from the latest Intel processors and coprocessors. See
> >>>> >> >> >> > abstracts
> >>>> >> >> >> > and
> >>>> >> >> >> > register
> >>>> >> >> >> >
> >>>> >> >> >> >
> >>>> >> >> >> >
> >>>> >> >> >> >
> >>>> >> >> >> >
> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk_______________________________________________
> >>>> >> >> >> > Postgres-xc-general mailing list
> >>>> >> >> >> > Pos...@li...
> >>>> >> >> >> >
> >>>> >> >> >> >
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> >>>> >> >> >>
> >>>> >> >> >
> >>>> >> >> >
> >>>> >> >> >
> >>>> >> >> >
> >>>> >> >> >
> >>>> >> >> >
> ------------------------------------------------------------------------------
> >>>> >> >> > November Webinars for C, C++, Fortran Developers
> >>>> >> >> > Accelerate application performance with scalable programming
> >>>> >> >> > models.
> >>>> >> >> > Explore
> >>>> >> >> > techniques for threading, error checking, porting, and tuning.
> >>>> >> >> > Get
> >>>> >> >> > the
> >>>> >> >> > most
> >>>> >> >> > from the latest Intel processors and coprocessors. See
> abstracts
> >>>> >> >> > and
> >>>> >> >> > register
> >>>> >> >> >
> >>>> >> >> >
> >>>> >> >> >
> >>>> >> >> >
> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
> >>>> >> >> > _______________________________________________
> >>>> >> >> > Postgres-xc-general mailing list
> >>>> >> >> > Pos...@li...
> >>>> >> >> >
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> >>>> >> >> >
> >>>> >> >
> >>>> >> >
> >>>> >> >
> >>>> >> >
> >>>> >> >
> >>>> >> >
> ------------------------------------------------------------------------------
> >>>> >> > DreamFactory - Open Source REST & JSON Services for HTML5 &
> Native
> >>>> >> > Apps
> >>>> >> > OAuth, Users, Roles, SQL, NoSQL, BLOB Storage and External API
> >>>> >> > Access
> >>>> >> > Free app hosting. Or install the open source package on any LAMP
> >>>> >> > server.
> >>>> >> > Sign up and see examples for AngularJS, jQuery, Sencha Touch and
> >>>> >> > Native!
> >>>> >> >
> >>>> >> >
> >>>> >> >
> http://pubads.g.doubleclick.net/gampad/clk?id=63469471&iu=/4140/ostg.clktrk
> >>>> >> > _______________________________________________
> >>>> >> > Postgres-xc-general mailing list
> >>>> >> > Pos...@li...
> >>>> >> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> >>>> >> >
> >>>> >
> >>>> >
> >>>
> >>>
> >>
> >
>

This happens when the message is not null-terminated properly.   Most
case is a bug.   In this case, different from vanilla PostgreSQL, copy
data is involved in.    Please let me find a time to look into it.

Regards;
---
Koichi Suzuki


2013/11/16 Sandeep Gupta <gup...@gm...>:
> Forgot to mention  that the error is still the same "invalid string in
> message".
>
> -Sandeep
>
>
>
> On Fri, Nov 15, 2013 at 8:27 PM, Sandeep Gupta <gup...@gm...>
> wrote:
>>
>> Hi Koichi,
>>
>>  We tried the  copy with the patch but it did not fix the problem for us.
>> If you need any additional details then let me know.
>>
>> Best,
>> -Sandeep
>>
>>
>> On Fri, Nov 15, 2013 at 6:13 AM, Sandeep Gupta <gup...@gm...>
>> wrote:
>>>
>>> Hi Koichi,
>>>
>>>  Many thanks for sharing the patch. We would test this in our environment
>>> and would report back on the result.
>>>
>>> Regarding the very high memory consumption, it is not the coordinator
>>> that sees the high memory usage.
>>> The setup is following: We have one coordinator and 4 datanodes. All the
>>> datanodes run on separate machine
>>> then the coordinator.  It is the machine where the datanodes are running
>>> that have 48GB memory usage.
>>> The other caveat is that that "top" doesn't show that it is the datanodes
>>> processes using the memory.
>>>
>>> In any case, the last run of copy even with the high usage went through
>>> fine.
>>>
>>>
>>> Best,
>>> Sandeep
>>>
>>>
>>>
>>> On Fri, Nov 15, 2013 at 2:29 AM, Koichi Suzuki <koi...@gm...>
>>> wrote:
>>>>
>>>> Yes, I've already found this kind of failure.   When disk write is
>>>> quite slower than read, coordinator overshoots.  However, in this
>>>> case, I thought we will see that palloc() overflows.   If each piece
>>>> of COPY TO text data is less than 1GB, then each COPY TO succeeds.
>>>> If the write is much slower than read,  each coordinator will allocate
>>>> memory for each COPY TO cache going to datanodes, ending up with
>>>> enormous amount of memory consumption and have a good chance to run
>>>> OOM killer.    Your memory usage looks enormous.    Maybe you have
>>>> many COPY TO instances each of which allocates too much memory.
>>>>
>>>> Attached patch fixes COPY TO.   Because we have a chance to have
>>>> similar overshoots in COPY FROM, I'd like to write a patch to fix this
>>>> too before committing.   This is for the current master but will work
>>>> with 1.1.x (even with 1.0.x hopefully).
>>>>
>>>> The patch looks to be fine.   I asked to test it in such an
>>>> environment and they reported the patch fixed their issue.
>>>>
>>>> I'm not sure but hope this works in your case.   May reduce memory
>>>> consumption drastically.   Please let me know the result.
>>>>
>>>> I have to allocate some of my resource to fix the same issue in COPY
>>>> FROM.    I have another request to fix this so if the patch works for
>>>> you, I may be able to commit only COPY TO fix first.
>>>>
>>>> Regards;
>>>> ---
>>>> Koichi Suzuki
>>>>
>>>>
>>>> 2013/11/15 Sandeep Gupta <gup...@gm...>:
>>>> > Hi Koichi,
>>>> >
>>>> >  I am using the copy command over the text file to populate a table.
>>>> >
>>>> > Sorry for the confusion. I meant 628 -- 638 lines in the pqformat.c.
>>>> > Let me explain this a bit more. I get  error in some of the datanode
>>>> > logfiles
>>>> > as "invalid string in message". The message is mentioned only in
>>>> > pqformat.c
>>>> > at line 628. Although I am not able to trace back as to where  error
>>>> > may
>>>> > have first ordered in the datanode that triggered this error.
>>>> >
>>>> > In order to debug this I started printing out the rows the coordinator
>>>> > is
>>>> > sending to the datanode (in function DataNodeCopyIn). For some reason
>>>> > the
>>>> > error invalid string in messge error has  not happened so far.  It
>>>> > seems
>>>> > there is some issue with the flow control of message between
>>>> > coordinator and
>>>> > datanode.
>>>> > Hope this helps in narrowing down what is causing the problem in the
>>>> > first
>>>> > place.
>>>> >
>>>> >
>>>> > However, the new problem I see is that the memory usage goes very
>>>> > high. Of
>>>> > the 48 GB almost all of the memory is getting used. Is this normal?
>>>> > If not can you give suggestion as to how to keep the memory
>>>> > consumption
>>>> > low.
>>>> >
>>>> > -Sandeep
>>>> >
>>>> >
>>>> >
>>>> > On Thu, Nov 14, 2013 at 8:57 PM, Koichi Suzuki <koi...@gm...>
>>>> > wrote:
>>>> >>
>>>> >> Could you let me know what "line 682 -- 637" means?
>>>> >>
>>>> >> Because any communication between coordinator and datanode is done
>>>> >> using libpq, I don't think XC populates information using raw
>>>> >> stringinfo.   In fact, from datanodes, coordinator is just a client
>>>> >> except for supplying XID and snapshot.   However,  COPY TO uses
>>>> >> send()/receive(), not through libpq, to send data to datanode.  In
>>>> >> this case, if incoming data file contains NULL in the data, such data
>>>> >> may be sent via send() to datanodes.   COPY TO assumes incoming data
>>>> >> in a text format.   Do you think you have such situations?
>>>> >>
>>>> >> Analysis may take a bit but let me try.
>>>> >>
>>>> >> Regards;
>>>> >> ---
>>>> >> Koichi Suzuki
>>>> >>
>>>> >>
>>>> >> 2013/11/15 Sandeep Gupta <gup...@gm...>:
>>>> >> > Hi Koichi, Masataka,
>>>> >> >
>>>> >> >  Thanks for taking a look. I double checked both memory usage and
>>>> >> > the
>>>> >> > dmesg
>>>> >> > log. The kill is not happening due to the out of memory. The kernel
>>>> >> > doesn't
>>>> >> > show any processes being killed.
>>>> >> >
>>>> >> > My current suspicion is that something amiss is happening over the
>>>> >> > network.
>>>> >> > This happens consistently when performing copy over large datasets.
>>>> >> > Few
>>>> >> > of
>>>> >> > datanodes end up with "invalid string in message".
>>>> >> > Going over the code in pqformat.c this happens because the
>>>> >> > invariant
>>>> >> > "StringInfo is guaranteed to have a trailing null byte" is not
>>>> >> > valid
>>>> >> > (line
>>>> >> > 682 -- 637). I am not sure why this happens.
>>>> >> >
>>>> >> >
>>>> >> > After this the datanodes do something illegal (probably buffer
>>>> >> > overflow
>>>> >> > on
>>>> >> > the port..just a guess) that the OS decides to issue a sigkill.  I
>>>> >> > am
>>>> >> > also
>>>> >> > not sure why the OS is not logging this killed operation.
>>>> >> >
>>>> >> >
>>>> >> > Please let me know of  suggestions you may have regarding the
>>>> >> > "invalid
>>>> >> > string in message". In particular, where in the code is StringInfo
>>>> >> > msg
>>>> >> > is
>>>> >> > getting populated i.e. which code reads the message over the
>>>> >> > network and
>>>> >> > keeps it ready for the copy operation on the datanode. Any other
>>>> >> > suggestion
>>>> >> > would also be helpful to trace this bug.
>>>> >> >
>>>> >> > Best,
>>>> >> > Sandeep
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > On Thu, Nov 14, 2013 at 10:25 AM, Masataka Saito <pg...@gm...>
>>>> >> > wrote:
>>>> >> >>
>>>> >> >> Hi,
>>>> >> >>
>>>> >> >> I think I may be onto something.
>>>> >> >>
>>>> >> >> When the memory is exhausted, the Linux kernel kills process
>>>> >> >> randomly
>>>> >> >> by SIGKILL.
>>>> >> >> The mechanism is called OOM(Out Of Memory) killer.
>>>> >> >> OOM killer logs its activity to kernel mesage buffer.
>>>> >> >>
>>>> >> >> Could you check display message (dmesg) and memory resource?
>>>> >> >>
>>>> >> >> Regards.
>>>> >> >>
>>>> >> >> On Tue, Nov 12, 2013 at 3:55 AM, Sandeep Gupta
>>>> >> >> <gup...@gm...>
>>>> >> >> wrote:
>>>> >> >> > Hi Koichi,
>>>> >> >> >
>>>> >> >> >    It is a bit of mystery  because it does not happen
>>>> >> >> > consistently.
>>>> >> >> > Thanks
>>>> >> >> > for the clarification though. It is indeed helpful.
>>>> >> >> >
>>>> >> >> > -Sandeep
>>>> >> >> >
>>>> >> >> >
>>>> >> >> >
>>>> >> >> >
>>>> >> >> > On Mon, Nov 11, 2013 at 1:58 AM, 鈴木 幸市
>>>> >> >> > <ko...@in...>
>>>> >> >> > wrote:
>>>> >> >> >>
>>>> >> >> >> Someone is sending SIGKILL to coordinator/datanode backend
>>>> >> >> >> processes.
>>>> >> >> >> Although XC (originally PG) code has a handler for SIGKILL, I
>>>> >> >> >> didn’t
>>>> >> >> >> found
>>>> >> >> >> any code in XC sending SIGKILL to other processes.   I’m afraid
>>>> >> >> >> there
>>>> >> >> >> could
>>>> >> >> >> be another process sending SIGKILL to them.
>>>> >> >> >>
>>>> >> >> >> Best Regards;
>>>> >> >> >> ---
>>>> >> >> >> Koichi Suzuki
>>>> >> >> >>
>>>> >> >> >> 2013/11/09 2:54、Sandeep Gupta <gup...@gm...> のメール：
>>>> >> >> >>
>>>> >> >> >> > I am running single instance of postgres-xc with  several
>>>> >> >> >> > datanode
>>>> >> >> >> > Each data node runs on its own machine.
>>>> >> >> >> > After instantiating the cluster the database sits idle. I do
>>>> >> >> >> > not
>>>> >> >> >> > perform
>>>> >> >> >> > any create table or insert operations. However, after some
>>>> >> >> >> > time
>>>> >> >> >> > all
>>>> >> >> >> > the
>>>> >> >> >> > datanode  automatically shuts down (log messages attached).
>>>> >> >> >> >
>>>> >> >> >> > Any pointers as to why this maybe happening would be very
>>>> >> >> >> > useful.
>>>> >> >> >> >
>>>> >> >> >> > -Sandeep
>>>> >> >> >> >
>>>> >> >> >> > The log files across all datanode contain various error
>>>> >> >> >> > messages.
>>>> >> >> >> > These
>>>> >> >> >> > include:
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> > LOG:  statistics collector process (PID 23073) was terminated
>>>> >> >> >> > by
>>>> >> >> >> > signal
>>>> >> >> >> > 9: Killed
>>>> >> >> >> > LOG:  WAL writer process (PID 23071) was terminated by signal
>>>> >> >> >> > 9:
>>>> >> >> >> > Killed
>>>> >> >> >> > LOG:  terminating any other active server processes
>>>> >> >> >> > WARNING:  terminating connection because of crash of another
>>>> >> >> >> > server
>>>> >> >> >> > process
>>>> >> >> >> > DETAIL:  The postmaster has commanded this server process to
>>>> >> >> >> > roll
>>>> >> >> >> > back
>>>> >> >> >> > the current transaction and exit, because another server
>>>> >> >> >> > process
>>>> >> >> >> > exited
>>>> >> >> >> > abnormally and possibly corrupted shared memory.
>>>> >> >> >> > HINT:  In a moment you should be able to reconnect to the
>>>> >> >> >> > database
>>>> >> >> >> > and
>>>> >> >> >> > repeat your command.
>>>> >> >> >> > LOG:  all server processes terminated; reinitializing
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> > LOG:  checkpointer process (PID 21406) was terminated by
>>>> >> >> >> > signal 9:
>>>> >> >> >> > Killed
>>>> >> >> >> > LOG:  terminating any other active server processes
>>>> >> >> >> > WARNING:  terminating connection because of crash of another
>>>> >> >> >> > server
>>>> >> >> >> > process
>>>> >> >> >> > DETAIL:  The postmaster has commanded this server process to
>>>> >> >> >> > roll
>>>> >> >> >> > back
>>>> >> >> >> > the current transaction and exit, because another server
>>>> >> >> >> > process
>>>> >> >> >> > exited
>>>> >> >> >> > abnormally and possibly corrupted shared memory.
>>>> >> >> >> > LOG:  statistics collector process (PID 28881) was terminated
>>>> >> >> >> > by
>>>> >> >> >> > signal
>>>> >> >> >> > 9: Killed
>>>> >> >> >> > LOG:  autovacuum launcher process (PID 28880) was terminated
>>>> >> >> >> > by
>>>> >> >> >> > signal
>>>> >> >> >> > 9: Killed
>>>> >> >> >> > LOG:  terminating any other active server processes
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> > ------------------------------------------------------------------------------
>>>> >> >> >> > November Webinars for C, C++, Fortran Developers
>>>> >> >> >> > Accelerate application performance with scalable programming
>>>> >> >> >> > models.
>>>> >> >> >> > Explore
>>>> >> >> >> > techniques for threading, error checking, porting, and
>>>> >> >> >> > tuning. Get
>>>> >> >> >> > the
>>>> >> >> >> > most
>>>> >> >> >> > from the latest Intel processors and coprocessors. See
>>>> >> >> >> > abstracts
>>>> >> >> >> > and
>>>> >> >> >> > register
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> > http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk_______________________________________________
>>>> >> >> >> > Postgres-xc-general mailing list
>>>> >> >> >> > Pos...@li...
>>>> >> >> >> >
>>>> >> >> >> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>> >> >> >>
>>>> >> >> >
>>>> >> >> >
>>>> >> >> >
>>>> >> >> >
>>>> >> >> >
>>>> >> >> > ------------------------------------------------------------------------------
>>>> >> >> > November Webinars for C, C++, Fortran Developers
>>>> >> >> > Accelerate application performance with scalable programming
>>>> >> >> > models.
>>>> >> >> > Explore
>>>> >> >> > techniques for threading, error checking, porting, and tuning.
>>>> >> >> > Get
>>>> >> >> > the
>>>> >> >> > most
>>>> >> >> > from the latest Intel processors and coprocessors. See abstracts
>>>> >> >> > and
>>>> >> >> > register
>>>> >> >> >
>>>> >> >> >
>>>> >> >> >
>>>> >> >> > http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
>>>> >> >> > _______________________________________________
>>>> >> >> > Postgres-xc-general mailing list
>>>> >> >> > Pos...@li...
>>>> >> >> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>> >> >> >
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > ------------------------------------------------------------------------------
>>>> >> > DreamFactory - Open Source REST & JSON Services for HTML5 & Native
>>>> >> > Apps
>>>> >> > OAuth, Users, Roles, SQL, NoSQL, BLOB Storage and External API
>>>> >> > Access
>>>> >> > Free app hosting. Or install the open source package on any LAMP
>>>> >> > server.
>>>> >> > Sign up and see examples for AngularJS, jQuery, Sencha Touch and
>>>> >> > Native!
>>>> >> >
>>>> >> >
>>>> >> > http://pubads.g.doubleclick.net/gampad/clk?id=63469471&iu=/4140/ostg.clktrk
>>>> >> > _______________________________________________
>>>> >> > Postgres-xc-general mailing list
>>>> >> > Pos...@li...
>>>> >> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>> >> >
>>>> >
>>>> >
>>>
>>>
>>
>

Forgot to mention  that the error is still the same "invalid string in
message".

-Sandeep



On Fri, Nov 15, 2013 at 8:27 PM, Sandeep Gupta <gup...@gm...>wrote:

> Hi Koichi,
>
>  We tried the  copy with the patch but it did not fix the problem for us.
> If you need any additional details then let me know.
>
> Best,
> -Sandeep
>
>
> On Fri, Nov 15, 2013 at 6:13 AM, Sandeep Gupta <gup...@gm...>wrote:
>
>> Hi Koichi,
>>
>>  Many thanks for sharing the patch. We would test this in our environment
>> and would report back on the result.
>>
>> Regarding the very high memory consumption, it is not the coordinator
>>  that sees the high memory usage.
>> The setup is following: We have one coordinator and 4 datanodes. All the
>> datanodes run on separate machine
>> then the coordinator.  It is the machine where the datanodes are running
>> that have 48GB memory usage.
>> The other caveat is that that "top" doesn't show that it is the datanodes
>> processes using the memory.
>>
>> In any case, the last run of copy even with the high usage went through
>> fine.
>>
>>
>> Best,
>> Sandeep
>>
>>
>>
>> On Fri, Nov 15, 2013 at 2:29 AM, Koichi Suzuki <koi...@gm...>wrote:
>>
>>> Yes, I've already found this kind of failure.   When disk write is
>>> quite slower than read, coordinator overshoots.  However, in this
>>> case, I thought we will see that palloc() overflows.   If each piece
>>> of COPY TO text data is less than 1GB, then each COPY TO succeeds.
>>> If the write is much slower than read,  each coordinator will allocate
>>> memory for each COPY TO cache going to datanodes, ending up with
>>> enormous amount of memory consumption and have a good chance to run
>>> OOM killer.    Your memory usage looks enormous.    Maybe you have
>>> many COPY TO instances each of which allocates too much memory.
>>>
>>> Attached patch fixes COPY TO.   Because we have a chance to have
>>> similar overshoots in COPY FROM, I'd like to write a patch to fix this
>>> too before committing.   This is for the current master but will work
>>> with 1.1.x (even with 1.0.x hopefully).
>>>
>>> The patch looks to be fine.   I asked to test it in such an
>>> environment and they reported the patch fixed their issue.
>>>
>>> I'm not sure but hope this works in your case.   May reduce memory
>>> consumption drastically.   Please let me know the result.
>>>
>>> I have to allocate some of my resource to fix the same issue in COPY
>>> FROM.    I have another request to fix this so if the patch works for
>>> you, I may be able to commit only COPY TO fix first.
>>>
>>> Regards;
>>> ---
>>> Koichi Suzuki
>>>
>>>
>>> 2013/11/15 Sandeep Gupta <gup...@gm...>:
>>> > Hi Koichi,
>>> >
>>> >  I am using the copy command over the text file to populate a table.
>>> >
>>> > Sorry for the confusion. I meant 628 -- 638 lines in the pqformat.c.
>>> > Let me explain this a bit more. I get  error in some of the datanode
>>> > logfiles
>>> > as "invalid string in message". The message is mentioned only in
>>> pqformat.c
>>> > at line 628. Although I am not able to trace back as to where  error
>>> may
>>> > have first ordered in the datanode that triggered this error.
>>> >
>>> > In order to debug this I started printing out the rows the coordinator
>>> is
>>> > sending to the datanode (in function DataNodeCopyIn). For some reason
>>> the
>>> > error invalid string in messge error has  not happened so far.  It
>>> seems
>>> > there is some issue with the flow control of message between
>>> coordinator and
>>> > datanode.
>>> > Hope this helps in narrowing down what is causing the problem in the
>>> first
>>> > place.
>>> >
>>> >
>>> > However, the new problem I see is that the memory usage goes very
>>> high. Of
>>> > the 48 GB almost all of the memory is getting used. Is this normal?
>>> > If not can you give suggestion as to how to keep the memory
>>>  consumption
>>> > low.
>>> >
>>> > -Sandeep
>>> >
>>> >
>>> >
>>> > On Thu, Nov 14, 2013 at 8:57 PM, Koichi Suzuki <koi...@gm...>
>>> > wrote:
>>> >>
>>> >> Could you let me know what "line 682 -- 637" means?
>>> >>
>>> >> Because any communication between coordinator and datanode is done
>>> >> using libpq, I don't think XC populates information using raw
>>> >> stringinfo.   In fact, from datanodes, coordinator is just a client
>>> >> except for supplying XID and snapshot.   However,  COPY TO uses
>>> >> send()/receive(), not through libpq, to send data to datanode.  In
>>> >> this case, if incoming data file contains NULL in the data, such data
>>> >> may be sent via send() to datanodes.   COPY TO assumes incoming data
>>> >> in a text format.   Do you think you have such situations?
>>> >>
>>> >> Analysis may take a bit but let me try.
>>> >>
>>> >> Regards;
>>> >> ---
>>> >> Koichi Suzuki
>>> >>
>>> >>
>>> >> 2013/11/15 Sandeep Gupta <gup...@gm...>:
>>> >> > Hi Koichi, Masataka,
>>> >> >
>>> >> >  Thanks for taking a look. I double checked both memory usage and
>>> the
>>> >> > dmesg
>>> >> > log. The kill is not happening due to the out of memory. The kernel
>>> >> > doesn't
>>> >> > show any processes being killed.
>>> >> >
>>> >> > My current suspicion is that something amiss is happening over the
>>> >> > network.
>>> >> > This happens consistently when performing copy over large datasets.
>>> Few
>>> >> > of
>>> >> > datanodes end up with "invalid string in message".
>>> >> > Going over the code in pqformat.c this happens because the invariant
>>> >> > "StringInfo is guaranteed to have a trailing null byte" is not valid
>>> >> > (line
>>> >> > 682 -- 637). I am not sure why this happens.
>>> >> >
>>> >> >
>>> >> > After this the datanodes do something illegal (probably buffer
>>> overflow
>>> >> > on
>>> >> > the port..just a guess) that the OS decides to issue a sigkill.  I
>>> am
>>> >> > also
>>> >> > not sure why the OS is not logging this killed operation.
>>> >> >
>>> >> >
>>> >> > Please let me know of  suggestions you may have regarding the
>>> "invalid
>>> >> > string in message". In particular, where in the code is StringInfo
>>> msg
>>> >> > is
>>> >> > getting populated i.e. which code reads the message over the
>>> network and
>>> >> > keeps it ready for the copy operation on the datanode. Any other
>>> >> > suggestion
>>> >> > would also be helpful to trace this bug.
>>> >> >
>>> >> > Best,
>>> >> > Sandeep
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> > On Thu, Nov 14, 2013 at 10:25 AM, Masataka Saito <pg...@gm...>
>>> >> > wrote:
>>> >> >>
>>> >> >> Hi,
>>> >> >>
>>> >> >> I think I may be onto something.
>>> >> >>
>>> >> >> When the memory is exhausted, the Linux kernel kills process
>>> randomly
>>> >> >> by SIGKILL.
>>> >> >> The mechanism is called OOM(Out Of Memory) killer.
>>> >> >> OOM killer logs its activity to kernel mesage buffer.
>>> >> >>
>>> >> >> Could you check display message (dmesg) and memory resource?
>>> >> >>
>>> >> >> Regards.
>>> >> >>
>>> >> >> On Tue, Nov 12, 2013 at 3:55 AM, Sandeep Gupta
>>> >> >> <gup...@gm...>
>>> >> >> wrote:
>>> >> >> > Hi Koichi,
>>> >> >> >
>>> >> >> >    It is a bit of mystery  because it does not happen
>>> consistently.
>>> >> >> > Thanks
>>> >> >> > for the clarification though. It is indeed helpful.
>>> >> >> >
>>> >> >> > -Sandeep
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > On Mon, Nov 11, 2013 at 1:58 AM, 鈴木 幸市 <ko...@in...
>>> >
>>> >> >> > wrote:
>>> >> >> >>
>>> >> >> >> Someone is sending SIGKILL to coordinator/datanode backend
>>> >> >> >> processes.
>>> >> >> >> Although XC (originally PG) code has a handler for SIGKILL, I
>>> didn’t
>>> >> >> >> found
>>> >> >> >> any code in XC sending SIGKILL to other processes.   I’m afraid
>>> >> >> >> there
>>> >> >> >> could
>>> >> >> >> be another process sending SIGKILL to them.
>>> >> >> >>
>>> >> >> >> Best Regards;
>>> >> >> >> ---
>>> >> >> >> Koichi Suzuki
>>> >> >> >>
>>> >> >> >> 2013/11/09 2:54、Sandeep Gupta <gup...@gm...> のメール：
>>> >> >> >>
>>> >> >> >> > I am running single instance of postgres-xc with  several
>>> datanode
>>> >> >> >> > Each data node runs on its own machine.
>>> >> >> >> > After instantiating the cluster the database sits idle. I do
>>> not
>>> >> >> >> > perform
>>> >> >> >> > any create table or insert operations. However, after some
>>> time
>>> >> >> >> > all
>>> >> >> >> > the
>>> >> >> >> > datanode  automatically shuts down (log messages attached).
>>> >> >> >> >
>>> >> >> >> > Any pointers as to why this maybe happening would be very
>>> useful.
>>> >> >> >> >
>>> >> >> >> > -Sandeep
>>> >> >> >> >
>>> >> >> >> > The log files across all datanode contain various error
>>> messages.
>>> >> >> >> > These
>>> >> >> >> > include:
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> > LOG:  statistics collector process (PID 23073) was terminated
>>> by
>>> >> >> >> > signal
>>> >> >> >> > 9: Killed
>>> >> >> >> > LOG:  WAL writer process (PID 23071) was terminated by signal
>>> 9:
>>> >> >> >> > Killed
>>> >> >> >> > LOG:  terminating any other active server processes
>>> >> >> >> > WARNING:  terminating connection because of crash of another
>>> >> >> >> > server
>>> >> >> >> > process
>>> >> >> >> > DETAIL:  The postmaster has commanded this server process to
>>> roll
>>> >> >> >> > back
>>> >> >> >> > the current transaction and exit, because another server
>>> process
>>> >> >> >> > exited
>>> >> >> >> > abnormally and possibly corrupted shared memory.
>>> >> >> >> > HINT:  In a moment you should be able to reconnect to the
>>> database
>>> >> >> >> > and
>>> >> >> >> > repeat your command.
>>> >> >> >> > LOG:  all server processes terminated; reinitializing
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> > LOG:  checkpointer process (PID 21406) was terminated by
>>> signal 9:
>>> >> >> >> > Killed
>>> >> >> >> > LOG:  terminating any other active server processes
>>> >> >> >> > WARNING:  terminating connection because of crash of another
>>> >> >> >> > server
>>> >> >> >> > process
>>> >> >> >> > DETAIL:  The postmaster has commanded this server process to
>>> roll
>>> >> >> >> > back
>>> >> >> >> > the current transaction and exit, because another server
>>> process
>>> >> >> >> > exited
>>> >> >> >> > abnormally and possibly corrupted shared memory.
>>> >> >> >> > LOG:  statistics collector process (PID 28881) was terminated
>>> by
>>> >> >> >> > signal
>>> >> >> >> > 9: Killed
>>> >> >> >> > LOG:  autovacuum launcher process (PID 28880) was terminated
>>> by
>>> >> >> >> > signal
>>> >> >> >> > 9: Killed
>>> >> >> >> > LOG:  terminating any other active server processes
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> ------------------------------------------------------------------------------
>>> >> >> >> > November Webinars for C, C++, Fortran Developers
>>> >> >> >> > Accelerate application performance with scalable programming
>>> >> >> >> > models.
>>> >> >> >> > Explore
>>> >> >> >> > techniques for threading, error checking, porting, and
>>> tuning. Get
>>> >> >> >> > the
>>> >> >> >> > most
>>> >> >> >> > from the latest Intel processors and coprocessors. See
>>> abstracts
>>> >> >> >> > and
>>> >> >> >> > register
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk_______________________________________________
>>> >> >> >> > Postgres-xc-general mailing list
>>> >> >> >> > Pos...@li...
>>> >> >> >> >
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>> >> >> >>
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> ------------------------------------------------------------------------------
>>> >> >> > November Webinars for C, C++, Fortran Developers
>>> >> >> > Accelerate application performance with scalable programming
>>> models.
>>> >> >> > Explore
>>> >> >> > techniques for threading, error checking, porting, and tuning.
>>> Get
>>> >> >> > the
>>> >> >> > most
>>> >> >> > from the latest Intel processors and coprocessors. See abstracts
>>> and
>>> >> >> > register
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
>>> >> >> > _______________________________________________
>>> >> >> > Postgres-xc-general mailing list
>>> >> >> > Pos...@li...
>>> >> >> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>> >> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> ------------------------------------------------------------------------------
>>> >> > DreamFactory - Open Source REST & JSON Services for HTML5 & Native
>>> Apps
>>> >> > OAuth, Users, Roles, SQL, NoSQL, BLOB Storage and External API
>>> Access
>>> >> > Free app hosting. Or install the open source package on any LAMP
>>> server.
>>> >> > Sign up and see examples for AngularJS, jQuery, Sencha Touch and
>>> Native!
>>> >> >
>>> >> >
>>> http://pubads.g.doubleclick.net/gampad/clk?id=63469471&iu=/4140/ostg.clktrk
>>> >> > _______________________________________________
>>> >> > Postgres-xc-general mailing list
>>> >> > Pos...@li...
>>> >> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>> >> >
>>> >
>>> >
>>>
>>
>>
>

Hi Koichi,

 We tried the  copy with the patch but it did not fix the problem for us.
If you need any additional details then let me know.

Best,
-Sandeep


On Fri, Nov 15, 2013 at 6:13 AM, Sandeep Gupta <gup...@gm...>wrote:

> Hi Koichi,
>
>  Many thanks for sharing the patch. We would test this in our environment
> and would report back on the result.
>
> Regarding the very high memory consumption, it is not the coordinator
>  that sees the high memory usage.
> The setup is following: We have one coordinator and 4 datanodes. All the
> datanodes run on separate machine
> then the coordinator.  It is the machine where the datanodes are running
> that have 48GB memory usage.
> The other caveat is that that "top" doesn't show that it is the datanodes
> processes using the memory.
>
> In any case, the last run of copy even with the high usage went through
> fine.
>
>
> Best,
> Sandeep
>
>
>
> On Fri, Nov 15, 2013 at 2:29 AM, Koichi Suzuki <koi...@gm...>wrote:
>
>> Yes, I've already found this kind of failure.   When disk write is
>> quite slower than read, coordinator overshoots.  However, in this
>> case, I thought we will see that palloc() overflows.   If each piece
>> of COPY TO text data is less than 1GB, then each COPY TO succeeds.
>> If the write is much slower than read,  each coordinator will allocate
>> memory for each COPY TO cache going to datanodes, ending up with
>> enormous amount of memory consumption and have a good chance to run
>> OOM killer.    Your memory usage looks enormous.    Maybe you have
>> many COPY TO instances each of which allocates too much memory.
>>
>> Attached patch fixes COPY TO.   Because we have a chance to have
>> similar overshoots in COPY FROM, I'd like to write a patch to fix this
>> too before committing.   This is for the current master but will work
>> with 1.1.x (even with 1.0.x hopefully).
>>
>> The patch looks to be fine.   I asked to test it in such an
>> environment and they reported the patch fixed their issue.
>>
>> I'm not sure but hope this works in your case.   May reduce memory
>> consumption drastically.   Please let me know the result.
>>
>> I have to allocate some of my resource to fix the same issue in COPY
>> FROM.    I have another request to fix this so if the patch works for
>> you, I may be able to commit only COPY TO fix first.
>>
>> Regards;
>> ---
>> Koichi Suzuki
>>
>>
>> 2013/11/15 Sandeep Gupta <gup...@gm...>:
>> > Hi Koichi,
>> >
>> >  I am using the copy command over the text file to populate a table.
>> >
>> > Sorry for the confusion. I meant 628 -- 638 lines in the pqformat.c.
>> > Let me explain this a bit more. I get  error in some of the datanode
>> > logfiles
>> > as "invalid string in message". The message is mentioned only in
>> pqformat.c
>> > at line 628. Although I am not able to trace back as to where  error may
>> > have first ordered in the datanode that triggered this error.
>> >
>> > In order to debug this I started printing out the rows the coordinator
>> is
>> > sending to the datanode (in function DataNodeCopyIn). For some reason
>> the
>> > error invalid string in messge error has  not happened so far.  It seems
>> > there is some issue with the flow control of message between
>> coordinator and
>> > datanode.
>> > Hope this helps in narrowing down what is causing the problem in the
>> first
>> > place.
>> >
>> >
>> > However, the new problem I see is that the memory usage goes very high.
>> Of
>> > the 48 GB almost all of the memory is getting used. Is this normal?
>> > If not can you give suggestion as to how to keep the memory  consumption
>> > low.
>> >
>> > -Sandeep
>> >
>> >
>> >
>> > On Thu, Nov 14, 2013 at 8:57 PM, Koichi Suzuki <koi...@gm...>
>> > wrote:
>> >>
>> >> Could you let me know what "line 682 -- 637" means?
>> >>
>> >> Because any communication between coordinator and datanode is done
>> >> using libpq, I don't think XC populates information using raw
>> >> stringinfo.   In fact, from datanodes, coordinator is just a client
>> >> except for supplying XID and snapshot.   However,  COPY TO uses
>> >> send()/receive(), not through libpq, to send data to datanode.  In
>> >> this case, if incoming data file contains NULL in the data, such data
>> >> may be sent via send() to datanodes.   COPY TO assumes incoming data
>> >> in a text format.   Do you think you have such situations?
>> >>
>> >> Analysis may take a bit but let me try.
>> >>
>> >> Regards;
>> >> ---
>> >> Koichi Suzuki
>> >>
>> >>
>> >> 2013/11/15 Sandeep Gupta <gup...@gm...>:
>> >> > Hi Koichi, Masataka,
>> >> >
>> >> >  Thanks for taking a look. I double checked both memory usage and the
>> >> > dmesg
>> >> > log. The kill is not happening due to the out of memory. The kernel
>> >> > doesn't
>> >> > show any processes being killed.
>> >> >
>> >> > My current suspicion is that something amiss is happening over the
>> >> > network.
>> >> > This happens consistently when performing copy over large datasets.
>> Few
>> >> > of
>> >> > datanodes end up with "invalid string in message".
>> >> > Going over the code in pqformat.c this happens because the invariant
>> >> > "StringInfo is guaranteed to have a trailing null byte" is not valid
>> >> > (line
>> >> > 682 -- 637). I am not sure why this happens.
>> >> >
>> >> >
>> >> > After this the datanodes do something illegal (probably buffer
>> overflow
>> >> > on
>> >> > the port..just a guess) that the OS decides to issue a sigkill.  I am
>> >> > also
>> >> > not sure why the OS is not logging this killed operation.
>> >> >
>> >> >
>> >> > Please let me know of  suggestions you may have regarding the
>> "invalid
>> >> > string in message". In particular, where in the code is StringInfo
>> msg
>> >> > is
>> >> > getting populated i.e. which code reads the message over the network
>> and
>> >> > keeps it ready for the copy operation on the datanode. Any other
>> >> > suggestion
>> >> > would also be helpful to trace this bug.
>> >> >
>> >> > Best,
>> >> > Sandeep
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On Thu, Nov 14, 2013 at 10:25 AM, Masataka Saito <pg...@gm...>
>> >> > wrote:
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> I think I may be onto something.
>> >> >>
>> >> >> When the memory is exhausted, the Linux kernel kills process
>> randomly
>> >> >> by SIGKILL.
>> >> >> The mechanism is called OOM(Out Of Memory) killer.
>> >> >> OOM killer logs its activity to kernel mesage buffer.
>> >> >>
>> >> >> Could you check display message (dmesg) and memory resource?
>> >> >>
>> >> >> Regards.
>> >> >>
>> >> >> On Tue, Nov 12, 2013 at 3:55 AM, Sandeep Gupta
>> >> >> <gup...@gm...>
>> >> >> wrote:
>> >> >> > Hi Koichi,
>> >> >> >
>> >> >> >    It is a bit of mystery  because it does not happen
>> consistently.
>> >> >> > Thanks
>> >> >> > for the clarification though. It is indeed helpful.
>> >> >> >
>> >> >> > -Sandeep
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > On Mon, Nov 11, 2013 at 1:58 AM, 鈴木 幸市 <ko...@in...>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> Someone is sending SIGKILL to coordinator/datanode backend
>> >> >> >> processes.
>> >> >> >> Although XC (originally PG) code has a handler for SIGKILL, I
>> didn’t
>> >> >> >> found
>> >> >> >> any code in XC sending SIGKILL to other processes.   I’m afraid
>> >> >> >> there
>> >> >> >> could
>> >> >> >> be another process sending SIGKILL to them.
>> >> >> >>
>> >> >> >> Best Regards;
>> >> >> >> ---
>> >> >> >> Koichi Suzuki
>> >> >> >>
>> >> >> >> 2013/11/09 2:54、Sandeep Gupta <gup...@gm...> のメール：
>> >> >> >>
>> >> >> >> > I am running single instance of postgres-xc with  several
>> datanode
>> >> >> >> > Each data node runs on its own machine.
>> >> >> >> > After instantiating the cluster the database sits idle. I do
>> not
>> >> >> >> > perform
>> >> >> >> > any create table or insert operations. However, after some time
>> >> >> >> > all
>> >> >> >> > the
>> >> >> >> > datanode  automatically shuts down (log messages attached).
>> >> >> >> >
>> >> >> >> > Any pointers as to why this maybe happening would be very
>> useful.
>> >> >> >> >
>> >> >> >> > -Sandeep
>> >> >> >> >
>> >> >> >> > The log files across all datanode contain various error
>> messages.
>> >> >> >> > These
>> >> >> >> > include:
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > LOG:  statistics collector process (PID 23073) was terminated
>> by
>> >> >> >> > signal
>> >> >> >> > 9: Killed
>> >> >> >> > LOG:  WAL writer process (PID 23071) was terminated by signal
>> 9:
>> >> >> >> > Killed
>> >> >> >> > LOG:  terminating any other active server processes
>> >> >> >> > WARNING:  terminating connection because of crash of another
>> >> >> >> > server
>> >> >> >> > process
>> >> >> >> > DETAIL:  The postmaster has commanded this server process to
>> roll
>> >> >> >> > back
>> >> >> >> > the current transaction and exit, because another server
>> process
>> >> >> >> > exited
>> >> >> >> > abnormally and possibly corrupted shared memory.
>> >> >> >> > HINT:  In a moment you should be able to reconnect to the
>> database
>> >> >> >> > and
>> >> >> >> > repeat your command.
>> >> >> >> > LOG:  all server processes terminated; reinitializing
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > LOG:  checkpointer process (PID 21406) was terminated by
>> signal 9:
>> >> >> >> > Killed
>> >> >> >> > LOG:  terminating any other active server processes
>> >> >> >> > WARNING:  terminating connection because of crash of another
>> >> >> >> > server
>> >> >> >> > process
>> >> >> >> > DETAIL:  The postmaster has commanded this server process to
>> roll
>> >> >> >> > back
>> >> >> >> > the current transaction and exit, because another server
>> process
>> >> >> >> > exited
>> >> >> >> > abnormally and possibly corrupted shared memory.
>> >> >> >> > LOG:  statistics collector process (PID 28881) was terminated
>> by
>> >> >> >> > signal
>> >> >> >> > 9: Killed
>> >> >> >> > LOG:  autovacuum launcher process (PID 28880) was terminated by
>> >> >> >> > signal
>> >> >> >> > 9: Killed
>> >> >> >> > LOG:  terminating any other active server processes
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> ------------------------------------------------------------------------------
>> >> >> >> > November Webinars for C, C++, Fortran Developers
>> >> >> >> > Accelerate application performance with scalable programming
>> >> >> >> > models.
>> >> >> >> > Explore
>> >> >> >> > techniques for threading, error checking, porting, and tuning.
>> Get
>> >> >> >> > the
>> >> >> >> > most
>> >> >> >> > from the latest Intel processors and coprocessors. See
>> abstracts
>> >> >> >> > and
>> >> >> >> > register
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk_______________________________________________
>> >> >> >> > Postgres-xc-general mailing list
>> >> >> >> > Pos...@li...
>> >> >> >> >
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>> >> >> >>
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> ------------------------------------------------------------------------------
>> >> >> > November Webinars for C, C++, Fortran Developers
>> >> >> > Accelerate application performance with scalable programming
>> models.
>> >> >> > Explore
>> >> >> > techniques for threading, error checking, porting, and tuning. Get
>> >> >> > the
>> >> >> > most
>> >> >> > from the latest Intel processors and coprocessors. See abstracts
>> and
>> >> >> > register
>> >> >> >
>> >> >> >
>> >> >> >
>> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
>> >> >> > _______________________________________________
>> >> >> > Postgres-xc-general mailing list
>> >> >> > Pos...@li...
>> >> >> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>> >> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> ------------------------------------------------------------------------------
>> >> > DreamFactory - Open Source REST & JSON Services for HTML5 & Native
>> Apps
>> >> > OAuth, Users, Roles, SQL, NoSQL, BLOB Storage and External API Access
>> >> > Free app hosting. Or install the open source package on any LAMP
>> server.
>> >> > Sign up and see examples for AngularJS, jQuery, Sencha Touch and
>> Native!
>> >> >
>> >> >
>> http://pubads.g.doubleclick.net/gampad/clk?id=63469471&iu=/4140/ostg.clktrk
>> >> > _______________________________________________
>> >> > Postgres-xc-general mailing list
>> >> > Pos...@li...
>> >> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>> >> >
>> >
>> >
>>
>
>

Hi Koichi,

 Many thanks for sharing the patch. We would test this in our environment
and would report back on the result.

Regarding the very high memory consumption, it is not the coordinator  that
sees the high memory usage.
The setup is following: We have one coordinator and 4 datanodes. All the
datanodes run on separate machine
then the coordinator.  It is the machine where the datanodes are running
that have 48GB memory usage.
The other caveat is that that "top" doesn't show that it is the datanodes
processes using the memory.

In any case, the last run of copy even with the high usage went through
fine.


Best,
Sandeep



On Fri, Nov 15, 2013 at 2:29 AM, Koichi Suzuki <koi...@gm...>wrote:

> Yes, I've already found this kind of failure.   When disk write is
> quite slower than read, coordinator overshoots.  However, in this
> case, I thought we will see that palloc() overflows.   If each piece
> of COPY TO text data is less than 1GB, then each COPY TO succeeds.
> If the write is much slower than read,  each coordinator will allocate
> memory for each COPY TO cache going to datanodes, ending up with
> enormous amount of memory consumption and have a good chance to run
> OOM killer.    Your memory usage looks enormous.    Maybe you have
> many COPY TO instances each of which allocates too much memory.
>
> Attached patch fixes COPY TO.   Because we have a chance to have
> similar overshoots in COPY FROM, I'd like to write a patch to fix this
> too before committing.   This is for the current master but will work
> with 1.1.x (even with 1.0.x hopefully).
>
> The patch looks to be fine.   I asked to test it in such an
> environment and they reported the patch fixed their issue.
>
> I'm not sure but hope this works in your case.   May reduce memory
> consumption drastically.   Please let me know the result.
>
> I have to allocate some of my resource to fix the same issue in COPY
> FROM.    I have another request to fix this so if the patch works for
> you, I may be able to commit only COPY TO fix first.
>
> Regards;
> ---
> Koichi Suzuki
>
>
> 2013/11/15 Sandeep Gupta <gup...@gm...>:
> > Hi Koichi,
> >
> >  I am using the copy command over the text file to populate a table.
> >
> > Sorry for the confusion. I meant 628 -- 638 lines in the pqformat.c.
> > Let me explain this a bit more. I get  error in some of the datanode
> > logfiles
> > as "invalid string in message". The message is mentioned only in
> pqformat.c
> > at line 628. Although I am not able to trace back as to where  error may
> > have first ordered in the datanode that triggered this error.
> >
> > In order to debug this I started printing out the rows the coordinator is
> > sending to the datanode (in function DataNodeCopyIn). For some reason the
> > error invalid string in messge error has  not happened so far.  It seems
> > there is some issue with the flow control of message between coordinator
> and
> > datanode.
> > Hope this helps in narrowing down what is causing the problem in the
> first
> > place.
> >
> >
> > However, the new problem I see is that the memory usage goes very high.
> Of
> > the 48 GB almost all of the memory is getting used. Is this normal?
> > If not can you give suggestion as to how to keep the memory  consumption
> > low.
> >
> > -Sandeep
> >
> >
> >
> > On Thu, Nov 14, 2013 at 8:57 PM, Koichi Suzuki <koi...@gm...>
> > wrote:
> >>
> >> Could you let me know what "line 682 -- 637" means?
> >>
> >> Because any communication between coordinator and datanode is done
> >> using libpq, I don't think XC populates information using raw
> >> stringinfo.   In fact, from datanodes, coordinator is just a client
> >> except for supplying XID and snapshot.   However,  COPY TO uses
> >> send()/receive(), not through libpq, to send data to datanode.  In
> >> this case, if incoming data file contains NULL in the data, such data
> >> may be sent via send() to datanodes.   COPY TO assumes incoming data
> >> in a text format.   Do you think you have such situations?
> >>
> >> Analysis may take a bit but let me try.
> >>
> >> Regards;
> >> ---
> >> Koichi Suzuki
> >>
> >>
> >> 2013/11/15 Sandeep Gupta <gup...@gm...>:
> >> > Hi Koichi, Masataka,
> >> >
> >> >  Thanks for taking a look. I double checked both memory usage and the
> >> > dmesg
> >> > log. The kill is not happening due to the out of memory. The kernel
> >> > doesn't
> >> > show any processes being killed.
> >> >
> >> > My current suspicion is that something amiss is happening over the
> >> > network.
> >> > This happens consistently when performing copy over large datasets.
> Few
> >> > of
> >> > datanodes end up with "invalid string in message".
> >> > Going over the code in pqformat.c this happens because the invariant
> >> > "StringInfo is guaranteed to have a trailing null byte" is not valid
> >> > (line
> >> > 682 -- 637). I am not sure why this happens.
> >> >
> >> >
> >> > After this the datanodes do something illegal (probably buffer
> overflow
> >> > on
> >> > the port..just a guess) that the OS decides to issue a sigkill.  I am
> >> > also
> >> > not sure why the OS is not logging this killed operation.
> >> >
> >> >
> >> > Please let me know of  suggestions you may have regarding the "invalid
> >> > string in message". In particular, where in the code is StringInfo msg
> >> > is
> >> > getting populated i.e. which code reads the message over the network
> and
> >> > keeps it ready for the copy operation on the datanode. Any other
> >> > suggestion
> >> > would also be helpful to trace this bug.
> >> >
> >> > Best,
> >> > Sandeep
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Thu, Nov 14, 2013 at 10:25 AM, Masataka Saito <pg...@gm...>
> >> > wrote:
> >> >>
> >> >> Hi,
> >> >>
> >> >> I think I may be onto something.
> >> >>
> >> >> When the memory is exhausted, the Linux kernel kills process randomly
> >> >> by SIGKILL.
> >> >> The mechanism is called OOM(Out Of Memory) killer.
> >> >> OOM killer logs its activity to kernel mesage buffer.
> >> >>
> >> >> Could you check display message (dmesg) and memory resource?
> >> >>
> >> >> Regards.
> >> >>
> >> >> On Tue, Nov 12, 2013 at 3:55 AM, Sandeep Gupta
> >> >> <gup...@gm...>
> >> >> wrote:
> >> >> > Hi Koichi,
> >> >> >
> >> >> >    It is a bit of mystery  because it does not happen consistently.
> >> >> > Thanks
> >> >> > for the clarification though. It is indeed helpful.
> >> >> >
> >> >> > -Sandeep
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > On Mon, Nov 11, 2013 at 1:58 AM, 鈴木 幸市 <ko...@in...>
> >> >> > wrote:
> >> >> >>
> >> >> >> Someone is sending SIGKILL to coordinator/datanode backend
> >> >> >> processes.
> >> >> >> Although XC (originally PG) code has a handler for SIGKILL, I
> didn’t
> >> >> >> found
> >> >> >> any code in XC sending SIGKILL to other processes.   I’m afraid
> >> >> >> there
> >> >> >> could
> >> >> >> be another process sending SIGKILL to them.
> >> >> >>
> >> >> >> Best Regards;
> >> >> >> ---
> >> >> >> Koichi Suzuki
> >> >> >>
> >> >> >> 2013/11/09 2:54、Sandeep Gupta <gup...@gm...> のメール：
> >> >> >>
> >> >> >> > I am running single instance of postgres-xc with  several
> datanode
> >> >> >> > Each data node runs on its own machine.
> >> >> >> > After instantiating the cluster the database sits idle. I do not
> >> >> >> > perform
> >> >> >> > any create table or insert operations. However, after some time
> >> >> >> > all
> >> >> >> > the
> >> >> >> > datanode  automatically shuts down (log messages attached).
> >> >> >> >
> >> >> >> > Any pointers as to why this maybe happening would be very
> useful.
> >> >> >> >
> >> >> >> > -Sandeep
> >> >> >> >
> >> >> >> > The log files across all datanode contain various error
> messages.
> >> >> >> > These
> >> >> >> > include:
> >> >> >> >
> >> >> >> >
> >> >> >> > LOG:  statistics collector process (PID 23073) was terminated by
> >> >> >> > signal
> >> >> >> > 9: Killed
> >> >> >> > LOG:  WAL writer process (PID 23071) was terminated by signal 9:
> >> >> >> > Killed
> >> >> >> > LOG:  terminating any other active server processes
> >> >> >> > WARNING:  terminating connection because of crash of another
> >> >> >> > server
> >> >> >> > process
> >> >> >> > DETAIL:  The postmaster has commanded this server process to
> roll
> >> >> >> > back
> >> >> >> > the current transaction and exit, because another server process
> >> >> >> > exited
> >> >> >> > abnormally and possibly corrupted shared memory.
> >> >> >> > HINT:  In a moment you should be able to reconnect to the
> database
> >> >> >> > and
> >> >> >> > repeat your command.
> >> >> >> > LOG:  all server processes terminated; reinitializing
> >> >> >> >
> >> >> >> >
> >> >> >> > LOG:  checkpointer process (PID 21406) was terminated by signal
> 9:
> >> >> >> > Killed
> >> >> >> > LOG:  terminating any other active server processes
> >> >> >> > WARNING:  terminating connection because of crash of another
> >> >> >> > server
> >> >> >> > process
> >> >> >> > DETAIL:  The postmaster has commanded this server process to
> roll
> >> >> >> > back
> >> >> >> > the current transaction and exit, because another server process
> >> >> >> > exited
> >> >> >> > abnormally and possibly corrupted shared memory.
> >> >> >> > LOG:  statistics collector process (PID 28881) was terminated by
> >> >> >> > signal
> >> >> >> > 9: Killed
> >> >> >> > LOG:  autovacuum launcher process (PID 28880) was terminated by
> >> >> >> > signal
> >> >> >> > 9: Killed
> >> >> >> > LOG:  terminating any other active server processes
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> ------------------------------------------------------------------------------
> >> >> >> > November Webinars for C, C++, Fortran Developers
> >> >> >> > Accelerate application performance with scalable programming
> >> >> >> > models.
> >> >> >> > Explore
> >> >> >> > techniques for threading, error checking, porting, and tuning.
> Get
> >> >> >> > the
> >> >> >> > most
> >> >> >> > from the latest Intel processors and coprocessors. See abstracts
> >> >> >> > and
> >> >> >> > register
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk_______________________________________________
> >> >> >> > Postgres-xc-general mailing list
> >> >> >> > Pos...@li...
> >> >> >> >
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> >> >> >>
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> ------------------------------------------------------------------------------
> >> >> > November Webinars for C, C++, Fortran Developers
> >> >> > Accelerate application performance with scalable programming
> models.
> >> >> > Explore
> >> >> > techniques for threading, error checking, porting, and tuning. Get
> >> >> > the
> >> >> > most
> >> >> > from the latest Intel processors and coprocessors. See abstracts
> and
> >> >> > register
> >> >> >
> >> >> >
> >> >> >
> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
> >> >> > _______________________________________________
> >> >> > Postgres-xc-general mailing list
> >> >> > Pos...@li...
> >> >> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> >> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> ------------------------------------------------------------------------------
> >> > DreamFactory - Open Source REST & JSON Services for HTML5 & Native
> Apps
> >> > OAuth, Users, Roles, SQL, NoSQL, BLOB Storage and External API Access
> >> > Free app hosting. Or install the open source package on any LAMP
> server.
> >> > Sign up and see examples for AngularJS, jQuery, Sencha Touch and
> Native!
> >> >
> >> >
> http://pubads.g.doubleclick.net/gampad/clk?id=63469471&iu=/4140/ostg.clktrk
> >> > _______________________________________________
> >> > Postgres-xc-general mailing list
> >> > Pos...@li...
> >> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> >> >
> >
> >
>

Yes, I've already found this kind of failure.   When disk write is
quite slower than read, coordinator overshoots.  However, in this
case, I thought we will see that palloc() overflows.   If each piece
of COPY TO text data is less than 1GB, then each COPY TO succeeds.
If the write is much slower than read,  each coordinator will allocate
memory for each COPY TO cache going to datanodes, ending up with
enormous amount of memory consumption and have a good chance to run
OOM killer.    Your memory usage looks enormous.    Maybe you have
many COPY TO instances each of which allocates too much memory.

Attached patch fixes COPY TO.   Because we have a chance to have
similar overshoots in COPY FROM, I'd like to write a patch to fix this
too before committing.   This is for the current master but will work
with 1.1.x (even with 1.0.x hopefully).

The patch looks to be fine.   I asked to test it in such an
environment and they reported the patch fixed their issue.

I'm not sure but hope this works in your case.   May reduce memory
consumption drastically.   Please let me know the result.

I have to allocate some of my resource to fix the same issue in COPY
FROM.    I have another request to fix this so if the patch works for
you, I may be able to commit only COPY TO fix first.

Regards;
---
Koichi Suzuki


2013/11/15 Sandeep Gupta <gup...@gm...>:
> Hi Koichi,
>
>  I am using the copy command over the text file to populate a table.
>
> Sorry for the confusion. I meant 628 -- 638 lines in the pqformat.c.
> Let me explain this a bit more. I get  error in some of the datanode
> logfiles
> as "invalid string in message". The message is mentioned only in pqformat.c
> at line 628. Although I am not able to trace back as to where  error may
> have first ordered in the datanode that triggered this error.
>
> In order to debug this I started printing out the rows the coordinator is
> sending to the datanode (in function DataNodeCopyIn). For some reason the
> error invalid string in messge error has  not happened so far.  It seems
> there is some issue with the flow control of message between coordinator and
> datanode.
> Hope this helps in narrowing down what is causing the problem in the first
> place.
>
>
> However, the new problem I see is that the memory usage goes very high. Of
> the 48 GB almost all of the memory is getting used. Is this normal?
> If not can you give suggestion as to how to keep the memory  consumption
> low.
>
> -Sandeep
>
>
>
> On Thu, Nov 14, 2013 at 8:57 PM, Koichi Suzuki <koi...@gm...>
> wrote:
>>
>> Could you let me know what "line 682 -- 637" means?
>>
>> Because any communication between coordinator and datanode is done
>> using libpq, I don't think XC populates information using raw
>> stringinfo.   In fact, from datanodes, coordinator is just a client
>> except for supplying XID and snapshot.   However,  COPY TO uses
>> send()/receive(), not through libpq, to send data to datanode.  In
>> this case, if incoming data file contains NULL in the data, such data
>> may be sent via send() to datanodes.   COPY TO assumes incoming data
>> in a text format.   Do you think you have such situations?
>>
>> Analysis may take a bit but let me try.
>>
>> Regards;
>> ---
>> Koichi Suzuki
>>
>>
>> 2013/11/15 Sandeep Gupta <gup...@gm...>:
>> > Hi Koichi, Masataka,
>> >
>> >  Thanks for taking a look. I double checked both memory usage and the
>> > dmesg
>> > log. The kill is not happening due to the out of memory. The kernel
>> > doesn't
>> > show any processes being killed.
>> >
>> > My current suspicion is that something amiss is happening over the
>> > network.
>> > This happens consistently when performing copy over large datasets. Few
>> > of
>> > datanodes end up with "invalid string in message".
>> > Going over the code in pqformat.c this happens because the invariant
>> > "StringInfo is guaranteed to have a trailing null byte" is not valid
>> > (line
>> > 682 -- 637). I am not sure why this happens.
>> >
>> >
>> > After this the datanodes do something illegal (probably buffer overflow
>> > on
>> > the port..just a guess) that the OS decides to issue a sigkill.  I am
>> > also
>> > not sure why the OS is not logging this killed operation.
>> >
>> >
>> > Please let me know of  suggestions you may have regarding the "invalid
>> > string in message". In particular, where in the code is StringInfo msg
>> > is
>> > getting populated i.e. which code reads the message over the network and
>> > keeps it ready for the copy operation on the datanode. Any other
>> > suggestion
>> > would also be helpful to trace this bug.
>> >
>> > Best,
>> > Sandeep
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Nov 14, 2013 at 10:25 AM, Masataka Saito <pg...@gm...>
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> I think I may be onto something.
>> >>
>> >> When the memory is exhausted, the Linux kernel kills process randomly
>> >> by SIGKILL.
>> >> The mechanism is called OOM(Out Of Memory) killer.
>> >> OOM killer logs its activity to kernel mesage buffer.
>> >>
>> >> Could you check display message (dmesg) and memory resource?
>> >>
>> >> Regards.
>> >>
>> >> On Tue, Nov 12, 2013 at 3:55 AM, Sandeep Gupta
>> >> <gup...@gm...>
>> >> wrote:
>> >> > Hi Koichi,
>> >> >
>> >> >    It is a bit of mystery  because it does not happen consistently.
>> >> > Thanks
>> >> > for the clarification though. It is indeed helpful.
>> >> >
>> >> > -Sandeep
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On Mon, Nov 11, 2013 at 1:58 AM, 鈴木 幸市 <ko...@in...>
>> >> > wrote:
>> >> >>
>> >> >> Someone is sending SIGKILL to coordinator/datanode backend
>> >> >> processes.
>> >> >> Although XC (originally PG) code has a handler for SIGKILL, I didn’t
>> >> >> found
>> >> >> any code in XC sending SIGKILL to other processes.   I’m afraid
>> >> >> there
>> >> >> could
>> >> >> be another process sending SIGKILL to them.
>> >> >>
>> >> >> Best Regards;
>> >> >> ---
>> >> >> Koichi Suzuki
>> >> >>
>> >> >> 2013/11/09 2:54、Sandeep Gupta <gup...@gm...> のメール：
>> >> >>
>> >> >> > I am running single instance of postgres-xc with  several datanode
>> >> >> > Each data node runs on its own machine.
>> >> >> > After instantiating the cluster the database sits idle. I do not
>> >> >> > perform
>> >> >> > any create table or insert operations. However, after some time
>> >> >> > all
>> >> >> > the
>> >> >> > datanode  automatically shuts down (log messages attached).
>> >> >> >
>> >> >> > Any pointers as to why this maybe happening would be very useful.
>> >> >> >
>> >> >> > -Sandeep
>> >> >> >
>> >> >> > The log files across all datanode contain various error messages.
>> >> >> > These
>> >> >> > include:
>> >> >> >
>> >> >> >
>> >> >> > LOG:  statistics collector process (PID 23073) was terminated by
>> >> >> > signal
>> >> >> > 9: Killed
>> >> >> > LOG:  WAL writer process (PID 23071) was terminated by signal 9:
>> >> >> > Killed
>> >> >> > LOG:  terminating any other active server processes
>> >> >> > WARNING:  terminating connection because of crash of another
>> >> >> > server
>> >> >> > process
>> >> >> > DETAIL:  The postmaster has commanded this server process to roll
>> >> >> > back
>> >> >> > the current transaction and exit, because another server process
>> >> >> > exited
>> >> >> > abnormally and possibly corrupted shared memory.
>> >> >> > HINT:  In a moment you should be able to reconnect to the database
>> >> >> > and
>> >> >> > repeat your command.
>> >> >> > LOG:  all server processes terminated; reinitializing
>> >> >> >
>> >> >> >
>> >> >> > LOG:  checkpointer process (PID 21406) was terminated by signal 9:
>> >> >> > Killed
>> >> >> > LOG:  terminating any other active server processes
>> >> >> > WARNING:  terminating connection because of crash of another
>> >> >> > server
>> >> >> > process
>> >> >> > DETAIL:  The postmaster has commanded this server process to roll
>> >> >> > back
>> >> >> > the current transaction and exit, because another server process
>> >> >> > exited
>> >> >> > abnormally and possibly corrupted shared memory.
>> >> >> > LOG:  statistics collector process (PID 28881) was terminated by
>> >> >> > signal
>> >> >> > 9: Killed
>> >> >> > LOG:  autovacuum launcher process (PID 28880) was terminated by
>> >> >> > signal
>> >> >> > 9: Killed
>> >> >> > LOG:  terminating any other active server processes
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > ------------------------------------------------------------------------------
>> >> >> > November Webinars for C, C++, Fortran Developers
>> >> >> > Accelerate application performance with scalable programming
>> >> >> > models.
>> >> >> > Explore
>> >> >> > techniques for threading, error checking, porting, and tuning. Get
>> >> >> > the
>> >> >> > most
>> >> >> > from the latest Intel processors and coprocessors. See abstracts
>> >> >> > and
>> >> >> > register
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk_______________________________________________
>> >> >> > Postgres-xc-general mailing list
>> >> >> > Pos...@li...
>> >> >> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > ------------------------------------------------------------------------------
>> >> > November Webinars for C, C++, Fortran Developers
>> >> > Accelerate application performance with scalable programming models.
>> >> > Explore
>> >> > techniques for threading, error checking, porting, and tuning. Get
>> >> > the
>> >> > most
>> >> > from the latest Intel processors and coprocessors. See abstracts and
>> >> > register
>> >> >
>> >> >
>> >> > http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
>> >> > _______________________________________________
>> >> > Postgres-xc-general mailing list
>> >> > Pos...@li...
>> >> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>> >> >
>> >
>> >
>> >
>> >
>> > ------------------------------------------------------------------------------
>> > DreamFactory - Open Source REST & JSON Services for HTML5 & Native Apps
>> > OAuth, Users, Roles, SQL, NoSQL, BLOB Storage and External API Access
>> > Free app hosting. Or install the open source package on any LAMP server.
>> > Sign up and see examples for AngularJS, jQuery, Sencha Touch and Native!
>> >
>> > http://pubads.g.doubleclick.net/gampad/clk?id=63469471&iu=/4140/ostg.clktrk
>> > _______________________________________________
>> > Postgres-xc-general mailing list
>> > Pos...@li...
>> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>> >
>
>

Hi Koichi,

 I am using the copy command over the text file to populate a table.

Sorry for the confusion. I meant 628 -- 638 lines in the pqformat.c.
Let me explain this a bit more. I get  error in some of the datanode
logfiles
as "invalid string in message". The message is mentioned only in pqformat.c
at line 628. Although I am not able to trace back as to where  error may
have first ordered in the datanode that triggered this error.

In order to debug this I started printing out the rows the coordinator is
sending to the datanode (in function DataNodeCopyIn). For some reason the
error invalid string in messge error has  not happened so far.  It seems
there is some issue with the flow control of message between coordinator
and datanode.
Hope this helps in narrowing down what is causing the problem in the first
place.


However, the new problem I see is that the memory usage goes very high. Of
the 48 GB almost all of the memory is getting used. Is this normal?
If not can you give suggestion as to how to keep the memory  consumption
low.

-Sandeep



On Thu, Nov 14, 2013 at 8:57 PM, Koichi Suzuki <koi...@gm...>wrote:

> Could you let me know what "line 682 -- 637" means?
>
> Because any communication between coordinator and datanode is done
> using libpq, I don't think XC populates information using raw
> stringinfo.   In fact, from datanodes, coordinator is just a client
> except for supplying XID and snapshot.   However,  COPY TO uses
> send()/receive(), not through libpq, to send data to datanode.  In
> this case, if incoming data file contains NULL in the data, such data
> may be sent via send() to datanodes.   COPY TO assumes incoming data
> in a text format.   Do you think you have such situations?
>
> Analysis may take a bit but let me try.
>
> Regards;
> ---
> Koichi Suzuki
>
>
> 2013/11/15 Sandeep Gupta <gup...@gm...>:
> > Hi Koichi, Masataka,
> >
> >  Thanks for taking a look. I double checked both memory usage and the
> dmesg
> > log. The kill is not happening due to the out of memory. The kernel
> doesn't
> > show any processes being killed.
> >
> > My current suspicion is that something amiss is happening over the
> network.
> > This happens consistently when performing copy over large datasets. Few
> of
> > datanodes end up with "invalid string in message".
> > Going over the code in pqformat.c this happens because the invariant
> > "StringInfo is guaranteed to have a trailing null byte" is not valid
> (line
> > 682 -- 637). I am not sure why this happens.
> >
> >
> > After this the datanodes do something illegal (probably buffer overflow
> on
> > the port..just a guess) that the OS decides to issue a sigkill.  I am
> also
> > not sure why the OS is not logging this killed operation.
> >
> >
> > Please let me know of  suggestions you may have regarding the "invalid
> > string in message". In particular, where in the code is StringInfo msg is
> > getting populated i.e. which code reads the message over the network and
> > keeps it ready for the copy operation on the datanode. Any other
> suggestion
> > would also be helpful to trace this bug.
> >
> > Best,
> > Sandeep
> >
> >
> >
> >
> >
> >
> > On Thu, Nov 14, 2013 at 10:25 AM, Masataka Saito <pg...@gm...>
> wrote:
> >>
> >> Hi,
> >>
> >> I think I may be onto something.
> >>
> >> When the memory is exhausted, the Linux kernel kills process randomly
> >> by SIGKILL.
> >> The mechanism is called OOM(Out Of Memory) killer.
> >> OOM killer logs its activity to kernel mesage buffer.
> >>
> >> Could you check display message (dmesg) and memory resource?
> >>
> >> Regards.
> >>
> >> On Tue, Nov 12, 2013 at 3:55 AM, Sandeep Gupta <gup...@gm...
> >
> >> wrote:
> >> > Hi Koichi,
> >> >
> >> >    It is a bit of mystery  because it does not happen consistently.
> >> > Thanks
> >> > for the clarification though. It is indeed helpful.
> >> >
> >> > -Sandeep
> >> >
> >> >
> >> >
> >> >
> >> > On Mon, Nov 11, 2013 at 1:58 AM, 鈴木 幸市 <ko...@in...>
> wrote:
> >> >>
> >> >> Someone is sending SIGKILL to coordinator/datanode backend processes.
> >> >> Although XC (originally PG) code has a handler for SIGKILL, I didn’t
> >> >> found
> >> >> any code in XC sending SIGKILL to other processes.   I’m afraid there
> >> >> could
> >> >> be another process sending SIGKILL to them.
> >> >>
> >> >> Best Regards;
> >> >> ---
> >> >> Koichi Suzuki
> >> >>
> >> >> 2013/11/09 2:54、Sandeep Gupta <gup...@gm...> のメール：
> >> >>
> >> >> > I am running single instance of postgres-xc with  several datanode
> >> >> > Each data node runs on its own machine.
> >> >> > After instantiating the cluster the database sits idle. I do not
> >> >> > perform
> >> >> > any create table or insert operations. However, after some time all
> >> >> > the
> >> >> > datanode  automatically shuts down (log messages attached).
> >> >> >
> >> >> > Any pointers as to why this maybe happening would be very useful.
> >> >> >
> >> >> > -Sandeep
> >> >> >
> >> >> > The log files across all datanode contain various error messages.
> >> >> > These
> >> >> > include:
> >> >> >
> >> >> >
> >> >> > LOG:  statistics collector process (PID 23073) was terminated by
> >> >> > signal
> >> >> > 9: Killed
> >> >> > LOG:  WAL writer process (PID 23071) was terminated by signal 9:
> >> >> > Killed
> >> >> > LOG:  terminating any other active server processes
> >> >> > WARNING:  terminating connection because of crash of another server
> >> >> > process
> >> >> > DETAIL:  The postmaster has commanded this server process to roll
> >> >> > back
> >> >> > the current transaction and exit, because another server process
> >> >> > exited
> >> >> > abnormally and possibly corrupted shared memory.
> >> >> > HINT:  In a moment you should be able to reconnect to the database
> >> >> > and
> >> >> > repeat your command.
> >> >> > LOG:  all server processes terminated; reinitializing
> >> >> >
> >> >> >
> >> >> > LOG:  checkpointer process (PID 21406) was terminated by signal 9:
> >> >> > Killed
> >> >> > LOG:  terminating any other active server processes
> >> >> > WARNING:  terminating connection because of crash of another server
> >> >> > process
> >> >> > DETAIL:  The postmaster has commanded this server process to roll
> >> >> > back
> >> >> > the current transaction and exit, because another server process
> >> >> > exited
> >> >> > abnormally and possibly corrupted shared memory.
> >> >> > LOG:  statistics collector process (PID 28881) was terminated by
> >> >> > signal
> >> >> > 9: Killed
> >> >> > LOG:  autovacuum launcher process (PID 28880) was terminated by
> >> >> > signal
> >> >> > 9: Killed
> >> >> > LOG:  terminating any other active server processes
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> ------------------------------------------------------------------------------
> >> >> > November Webinars for C, C++, Fortran Developers
> >> >> > Accelerate application performance with scalable programming
> models.
> >> >> > Explore
> >> >> > techniques for threading, error checking, porting, and tuning. Get
> >> >> > the
> >> >> > most
> >> >> > from the latest Intel processors and coprocessors. See abstracts
> and
> >> >> > register
> >> >> >
> >> >> >
> >> >> >
> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk_______________________________________________
> >> >> > Postgres-xc-general mailing list
> >> >> > Pos...@li...
> >> >> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> >> >>
> >> >
> >> >
> >> >
> >> >
> ------------------------------------------------------------------------------
> >> > November Webinars for C, C++, Fortran Developers
> >> > Accelerate application performance with scalable programming models.
> >> > Explore
> >> > techniques for threading, error checking, porting, and tuning. Get the
> >> > most
> >> > from the latest Intel processors and coprocessors. See abstracts and
> >> > register
> >> >
> >> >
> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
> >> > _______________________________________________
> >> > Postgres-xc-general mailing list
> >> > Pos...@li...
> >> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> >> >
> >
> >
> >
> >
> ------------------------------------------------------------------------------
> > DreamFactory - Open Source REST & JSON Services for HTML5 & Native Apps
> > OAuth, Users, Roles, SQL, NoSQL, BLOB Storage and External API Access
> > Free app hosting. Or install the open source package on any LAMP server.
> > Sign up and see examples for AngularJS, jQuery, Sencha Touch and Native!
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=63469471&iu=/4140/ostg.clktrk
> > _______________________________________________
> > Postgres-xc-general mailing list
> > Pos...@li...
> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> >
>

Could you let me know what "line 682 -- 637" means?

Because any communication between coordinator and datanode is done
using libpq, I don't think XC populates information using raw
stringinfo.   In fact, from datanodes, coordinator is just a client
except for supplying XID and snapshot.   However,  COPY TO uses
send()/receive(), not through libpq, to send data to datanode.  In
this case, if incoming data file contains NULL in the data, such data
may be sent via send() to datanodes.   COPY TO assumes incoming data
in a text format.   Do you think you have such situations?

Analysis may take a bit but let me try.

Regards;
---
Koichi Suzuki


2013/11/15 Sandeep Gupta <gup...@gm...>:
> Hi Koichi, Masataka,
>
>  Thanks for taking a look. I double checked both memory usage and the dmesg
> log. The kill is not happening due to the out of memory. The kernel doesn't
> show any processes being killed.
>
> My current suspicion is that something amiss is happening over the network.
> This happens consistently when performing copy over large datasets. Few of
> datanodes end up with "invalid string in message".
> Going over the code in pqformat.c this happens because the invariant
> "StringInfo is guaranteed to have a trailing null byte" is not valid (line
> 682 -- 637). I am not sure why this happens.
>
>
> After this the datanodes do something illegal (probably buffer overflow on
> the port..just a guess) that the OS decides to issue a sigkill.  I am also
> not sure why the OS is not logging this killed operation.
>
>
> Please let me know of  suggestions you may have regarding the "invalid
> string in message". In particular, where in the code is StringInfo msg is
> getting populated i.e. which code reads the message over the network and
> keeps it ready for the copy operation on the datanode. Any other suggestion
> would also be helpful to trace this bug.
>
> Best,
> Sandeep
>
>
>
>
>
>
> On Thu, Nov 14, 2013 at 10:25 AM, Masataka Saito <pg...@gm...> wrote:
>>
>> Hi,
>>
>> I think I may be onto something.
>>
>> When the memory is exhausted, the Linux kernel kills process randomly
>> by SIGKILL.
>> The mechanism is called OOM(Out Of Memory) killer.
>> OOM killer logs its activity to kernel mesage buffer.
>>
>> Could you check display message (dmesg) and memory resource?
>>
>> Regards.
>>
>> On Tue, Nov 12, 2013 at 3:55 AM, Sandeep Gupta <gup...@gm...>
>> wrote:
>> > Hi Koichi,
>> >
>> >    It is a bit of mystery  because it does not happen consistently.
>> > Thanks
>> > for the clarification though. It is indeed helpful.
>> >
>> > -Sandeep
>> >
>> >
>> >
>> >
>> > On Mon, Nov 11, 2013 at 1:58 AM, 鈴木 幸市 <ko...@in...> wrote:
>> >>
>> >> Someone is sending SIGKILL to coordinator/datanode backend processes.
>> >> Although XC (originally PG) code has a handler for SIGKILL, I didn’t
>> >> found
>> >> any code in XC sending SIGKILL to other processes.   I’m afraid there
>> >> could
>> >> be another process sending SIGKILL to them.
>> >>
>> >> Best Regards;
>> >> ---
>> >> Koichi Suzuki
>> >>
>> >> 2013/11/09 2:54、Sandeep Gupta <gup...@gm...> のメール：
>> >>
>> >> > I am running single instance of postgres-xc with  several datanode
>> >> > Each data node runs on its own machine.
>> >> > After instantiating the cluster the database sits idle. I do not
>> >> > perform
>> >> > any create table or insert operations. However, after some time all
>> >> > the
>> >> > datanode  automatically shuts down (log messages attached).
>> >> >
>> >> > Any pointers as to why this maybe happening would be very useful.
>> >> >
>> >> > -Sandeep
>> >> >
>> >> > The log files across all datanode contain various error messages.
>> >> > These
>> >> > include:
>> >> >
>> >> >
>> >> > LOG:  statistics collector process (PID 23073) was terminated by
>> >> > signal
>> >> > 9: Killed
>> >> > LOG:  WAL writer process (PID 23071) was terminated by signal 9:
>> >> > Killed
>> >> > LOG:  terminating any other active server processes
>> >> > WARNING:  terminating connection because of crash of another server
>> >> > process
>> >> > DETAIL:  The postmaster has commanded this server process to roll
>> >> > back
>> >> > the current transaction and exit, because another server process
>> >> > exited
>> >> > abnormally and possibly corrupted shared memory.
>> >> > HINT:  In a moment you should be able to reconnect to the database
>> >> > and
>> >> > repeat your command.
>> >> > LOG:  all server processes terminated; reinitializing
>> >> >
>> >> >
>> >> > LOG:  checkpointer process (PID 21406) was terminated by signal 9:
>> >> > Killed
>> >> > LOG:  terminating any other active server processes
>> >> > WARNING:  terminating connection because of crash of another server
>> >> > process
>> >> > DETAIL:  The postmaster has commanded this server process to roll
>> >> > back
>> >> > the current transaction and exit, because another server process
>> >> > exited
>> >> > abnormally and possibly corrupted shared memory.
>> >> > LOG:  statistics collector process (PID 28881) was terminated by
>> >> > signal
>> >> > 9: Killed
>> >> > LOG:  autovacuum launcher process (PID 28880) was terminated by
>> >> > signal
>> >> > 9: Killed
>> >> > LOG:  terminating any other active server processes
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > ------------------------------------------------------------------------------
>> >> > November Webinars for C, C++, Fortran Developers
>> >> > Accelerate application performance with scalable programming models.
>> >> > Explore
>> >> > techniques for threading, error checking, porting, and tuning. Get
>> >> > the
>> >> > most
>> >> > from the latest Intel processors and coprocessors. See abstracts and
>> >> > register
>> >> >
>> >> >
>> >> > http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk_______________________________________________
>> >> > Postgres-xc-general mailing list
>> >> > Pos...@li...
>> >> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>> >>
>> >
>> >
>> >
>> > ------------------------------------------------------------------------------
>> > November Webinars for C, C++, Fortran Developers
>> > Accelerate application performance with scalable programming models.
>> > Explore
>> > techniques for threading, error checking, porting, and tuning. Get the
>> > most
>> > from the latest Intel processors and coprocessors. See abstracts and
>> > register
>> >
>> > http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
>> > _______________________________________________
>> > Postgres-xc-general mailing list
>> > Pos...@li...
>> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>> >
>
>
>
> ------------------------------------------------------------------------------
> DreamFactory - Open Source REST & JSON Services for HTML5 & Native Apps
> OAuth, Users, Roles, SQL, NoSQL, BLOB Storage and External API Access
> Free app hosting. Or install the open source package on any LAMP server.
> Sign up and see examples for AngularJS, jQuery, Sencha Touch and Native!
> http://pubads.g.doubleclick.net/gampad/clk?id=63469471&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>

pgxc_ctl contrib module comes with this capability.   To use this, you
should configure and operate XC cluster with this utility.   The
document will be found in XC documentation.   Online documentation
will be found at http://postgres-xc.sourceforge.net/docs/1_1/

If you'd like to do this manually, the following pgxc_ctl source will
be helpful:

add_coordinatorMaster() in coord_cmd.c: describes how to add coordinator master.
add_datanodeMaster() in datanode_cmd.c: describes how to add datanode master.

Hope they help.

Regards;
---
Koichi Suzuki


2013/11/14 Casper, Stephan <Ste...@se...>:
> Hello,
>
>
>
> we want to use a postgres-xc cluster with two or more master nodes.
>
> If we start the cluster configuration with a GTM, 2 GTM-proxys, 2
> master-coordinators and 2 master-datanodes , we can initialize and use it.
>
>
>
> Now we want to add or remove datanodes while the database is still running.
>
> If we add a new node, we add that one to the coordinators node list. It
> seems the datanode now knows the database schema, but didn’t contain any
> data. The node is also in write protection mode.
>
> If we remove an existing  datanode (shutdown its kvm and remove it from the
> coordinators node list), now the database is not able to execute queries.
>
>
>
> We are using Postgres-XC 1.1 beta.
>
>
>
> So, we would like to know which is the right way to add more new master
> nodes to a running cluster and how can we use the cluster after one of the
> nodes has crashed?
>
>
>
> Best regards,
>
>
>
> Stephan Casper
>
>
> ------------------------------------------------------------------------------
> DreamFactory - Open Source REST & JSON Services for HTML5 & Native Apps
> OAuth, Users, Roles, SQL, NoSQL, BLOB Storage and External API Access
> Free app hosting. Or install the open source package on any LAMP server.
> Sign up and see examples for AngularJS, jQuery, Sencha Touch and Native!
> http://pubads.g.doubleclick.net/gampad/clk?id=63469471&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>

Hi Koichi, Masataka,

 Thanks for taking a look. I double checked both memory usage and the dmesg
log. The kill is not happening due to the out of memory. The kernel doesn't
show any processes being killed.

My current suspicion is that something amiss is happening over the network.
This happens consistently when performing copy over large datasets. Few of
datanodes end up with "invalid string in message".
Going over the code in pqformat.c this happens because the invariant
"StringInfo is guaranteed to have a trailing null byte" is not valid (line
682 -- 637). I am not sure why this happens.


After this the datanodes do something illegal (probably buffer overflow on
the port..just a guess) that the OS decides to issue a sigkill.  I am also
not sure why the OS is not logging this killed operation.


Please let me know of  suggestions you may have regarding the "invalid
string in message". In particular, where in the code is StringInfo msg is
getting populated i.e. which code reads the message over the network and
keeps it ready for the copy operation on the datanode. Any other suggestion
would also be helpful to trace this bug.

Best,
Sandeep






On Thu, Nov 14, 2013 at 10:25 AM, Masataka Saito <pg...@gm...> wrote:

> Hi,
>
> I think I may be onto something.
>
> When the memory is exhausted, the Linux kernel kills process randomly
> by SIGKILL.
> The mechanism is called OOM(Out Of Memory) killer.
> OOM killer logs its activity to kernel mesage buffer.
>
> Could you check display message (dmesg) and memory resource?
>
> Regards.
>
> On Tue, Nov 12, 2013 at 3:55 AM, Sandeep Gupta <gup...@gm...>
> wrote:
> > Hi Koichi,
> >
> >    It is a bit of mystery  because it does not happen consistently.
> Thanks
> > for the clarification though. It is indeed helpful.
> >
> > -Sandeep
> >
> >
> >
> >
> > On Mon, Nov 11, 2013 at 1:58 AM, 鈴木 幸市 <ko...@in...> wrote:
> >>
> >> Someone is sending SIGKILL to coordinator/datanode backend processes.
> >> Although XC (originally PG) code has a handler for SIGKILL, I didn’t
> found
> >> any code in XC sending SIGKILL to other processes.   I’m afraid there
> could
> >> be another process sending SIGKILL to them.
> >>
> >> Best Regards;
> >> ---
> >> Koichi Suzuki
> >>
> >> 2013/11/09 2:54、Sandeep Gupta <gup...@gm...> のメール：
> >>
> >> > I am running single instance of postgres-xc with  several datanode
> >> > Each data node runs on its own machine.
> >> > After instantiating the cluster the database sits idle. I do not
> perform
> >> > any create table or insert operations. However, after some time all
> the
> >> > datanode  automatically shuts down (log messages attached).
> >> >
> >> > Any pointers as to why this maybe happening would be very useful.
> >> >
> >> > -Sandeep
> >> >
> >> > The log files across all datanode contain various error messages.
> These
> >> > include:
> >> >
> >> >
> >> > LOG:  statistics collector process (PID 23073) was terminated by
> signal
> >> > 9: Killed
> >> > LOG:  WAL writer process (PID 23071) was terminated by signal 9:
> Killed
> >> > LOG:  terminating any other active server processes
> >> > WARNING:  terminating connection because of crash of another server
> >> > process
> >> > DETAIL:  The postmaster has commanded this server process to roll back
> >> > the current transaction and exit, because another server process
> exited
> >> > abnormally and possibly corrupted shared memory.
> >> > HINT:  In a moment you should be able to reconnect to the database and
> >> > repeat your command.
> >> > LOG:  all server processes terminated; reinitializing
> >> >
> >> >
> >> > LOG:  checkpointer process (PID 21406) was terminated by signal 9:
> >> > Killed
> >> > LOG:  terminating any other active server processes
> >> > WARNING:  terminating connection because of crash of another server
> >> > process
> >> > DETAIL:  The postmaster has commanded this server process to roll back
> >> > the current transaction and exit, because another server process
> exited
> >> > abnormally and possibly corrupted shared memory.
> >> > LOG:  statistics collector process (PID 28881) was terminated by
> signal
> >> > 9: Killed
> >> > LOG:  autovacuum launcher process (PID 28880) was terminated by signal
> >> > 9: Killed
> >> > LOG:  terminating any other active server processes
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> ------------------------------------------------------------------------------
> >> > November Webinars for C, C++, Fortran Developers
> >> > Accelerate application performance with scalable programming models.
> >> > Explore
> >> > techniques for threading, error checking, porting, and tuning. Get the
> >> > most
> >> > from the latest Intel processors and coprocessors. See abstracts and
> >> > register
> >> >
> >> >
> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk_______________________________________________
> >> > Postgres-xc-general mailing list
> >> > Pos...@li...
> >> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> >>
> >
> >
> >
> ------------------------------------------------------------------------------
> > November Webinars for C, C++, Fortran Developers
> > Accelerate application performance with scalable programming models.
> Explore
> > techniques for threading, error checking, porting, and tuning. Get the
> most
> > from the latest Intel processors and coprocessors. See abstracts and
> > register
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
> > _______________________________________________
> > Postgres-xc-general mailing list
> > Pos...@li...
> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> >
>

Hi,

I think I may be onto something.

When the memory is exhausted, the Linux kernel kills process randomly
by SIGKILL.
The mechanism is called OOM(Out Of Memory) killer.
OOM killer logs its activity to kernel mesage buffer.

Could you check display message (dmesg) and memory resource?

Regards.

On Tue, Nov 12, 2013 at 3:55 AM, Sandeep Gupta <gup...@gm...> wrote:
> Hi Koichi,
>
>    It is a bit of mystery  because it does not happen consistently. Thanks
> for the clarification though. It is indeed helpful.
>
> -Sandeep
>
>
>
>
> On Mon, Nov 11, 2013 at 1:58 AM, 鈴木 幸市 <ko...@in...> wrote:
>>
>> Someone is sending SIGKILL to coordinator/datanode backend processes.
>> Although XC (originally PG) code has a handler for SIGKILL, I didn’t found
>> any code in XC sending SIGKILL to other processes.   I’m afraid there could
>> be another process sending SIGKILL to them.
>>
>> Best Regards;
>> ---
>> Koichi Suzuki
>>
>> 2013/11/09 2:54、Sandeep Gupta <gup...@gm...> のメール：
>>
>> > I am running single instance of postgres-xc with  several datanode
>> > Each data node runs on its own machine.
>> > After instantiating the cluster the database sits idle. I do not perform
>> > any create table or insert operations. However, after some time all the
>> > datanode  automatically shuts down (log messages attached).
>> >
>> > Any pointers as to why this maybe happening would be very useful.
>> >
>> > -Sandeep
>> >
>> > The log files across all datanode contain various error messages. These
>> > include:
>> >
>> >
>> > LOG:  statistics collector process (PID 23073) was terminated by signal
>> > 9: Killed
>> > LOG:  WAL writer process (PID 23071) was terminated by signal 9: Killed
>> > LOG:  terminating any other active server processes
>> > WARNING:  terminating connection because of crash of another server
>> > process
>> > DETAIL:  The postmaster has commanded this server process to roll back
>> > the current transaction and exit, because another server process exited
>> > abnormally and possibly corrupted shared memory.
>> > HINT:  In a moment you should be able to reconnect to the database and
>> > repeat your command.
>> > LOG:  all server processes terminated; reinitializing
>> >
>> >
>> > LOG:  checkpointer process (PID 21406) was terminated by signal 9:
>> > Killed
>> > LOG:  terminating any other active server processes
>> > WARNING:  terminating connection because of crash of another server
>> > process
>> > DETAIL:  The postmaster has commanded this server process to roll back
>> > the current transaction and exit, because another server process exited
>> > abnormally and possibly corrupted shared memory.
>> > LOG:  statistics collector process (PID 28881) was terminated by signal
>> > 9: Killed
>> > LOG:  autovacuum launcher process (PID 28880) was terminated by signal
>> > 9: Killed
>> > LOG:  terminating any other active server processes
>> >
>> >
>> >
>> >
>> >
>> >
>> > ------------------------------------------------------------------------------
>> > November Webinars for C, C++, Fortran Developers
>> > Accelerate application performance with scalable programming models.
>> > Explore
>> > techniques for threading, error checking, porting, and tuning. Get the
>> > most
>> > from the latest Intel processors and coprocessors. See abstracts and
>> > register
>> >
>> > http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk_______________________________________________
>> > Postgres-xc-general mailing list
>> > Pos...@li...
>> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>
>
>
> ------------------------------------------------------------------------------
> November Webinars for C, C++, Fortran Developers
> Accelerate application performance with scalable programming models. Explore
> techniques for threading, error checking, porting, and tuning. Get the most
> from the latest Intel processors and coprocessors. See abstracts and
> register
> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>

Hello,

 

we want to use a postgres-xc cluster with two or more master nodes.

If we start the cluster configuration with a GTM, 2 GTM-proxys, 2
master-coordinators and 2 master-datanodes , we can initialize and use
it.

 

Now we want to add or remove datanodes while the database is still
running. 

If we add a new node, we add that one to the coordinators node list. It
seems the datanode now knows the database schema, but didn't contain any
data. The node is also in write protection mode.

If we remove an existing  datanode (shutdown its kvm and remove it from
the coordinators node list), now the database is not able to execute
queries.

 

We are using Postgres-XC 1.1 beta.

 

So, we would like to know which is the right way to add more new master
nodes to a running cluster and how can we use the cluster after one of
the nodes has crashed? 

 

Best regards,

 

Stephan Casper

Hi Koichi,

   It is a bit of mystery  because it does not happen consistently. Thanks
for the clarification though. It is indeed helpful.

-Sandeep




On Mon, Nov 11, 2013 at 1:58 AM, 鈴木 幸市 <ko...@in...> wrote:

> Someone is sending SIGKILL to coordinator/datanode backend processes.
>  Although XC (originally PG) code has a handler for SIGKILL, I didn’t found
> any code in XC sending SIGKILL to other processes.   I’m afraid there could
> be another process sending SIGKILL to them.
>
> Best Regards;
> ---
> Koichi Suzuki
>
> 2013/11/09 2:54、Sandeep Gupta <gup...@gm...> のメール：
>
> > I am running single instance of postgres-xc with  several datanode
> > Each data node runs on its own machine.
> > After instantiating the cluster the database sits idle. I do not perform
> any create table or insert operations. However, after some time all the
> datanode  automatically shuts down (log messages attached).
> >
> > Any pointers as to why this maybe happening would be very useful.
> >
> > -Sandeep
> >
> > The log files across all datanode contain various error messages. These
> include:
> >
> >
> > LOG:  statistics collector process (PID 23073) was terminated by signal
> 9: Killed
> > LOG:  WAL writer process (PID 23071) was terminated by signal 9: Killed
> > LOG:  terminating any other active server processes
> > WARNING:  terminating connection because of crash of another server
> process
> > DETAIL:  The postmaster has commanded this server process to roll back
> the current transaction and exit, because another server process exited
> abnormally and possibly corrupted shared memory.
> > HINT:  In a moment you should be able to reconnect to the database and
> repeat your command.
> > LOG:  all server processes terminated; reinitializing
> >
> >
> > LOG:  checkpointer process (PID 21406) was terminated by signal 9: Killed
> > LOG:  terminating any other active server processes
> > WARNING:  terminating connection because of crash of another server
> process
> > DETAIL:  The postmaster has commanded this server process to roll back
> the current transaction and exit, because another server process exited
> abnormally and possibly corrupted shared memory.
> > LOG:  statistics collector process (PID 28881) was terminated by signal
> 9: Killed
> > LOG:  autovacuum launcher process (PID 28880) was terminated by signal
> 9: Killed
> > LOG:  terminating any other active server processes
> >
> >
> >
> >
> >
> >
> ------------------------------------------------------------------------------
> > November Webinars for C, C++, Fortran Developers
> > Accelerate application performance with scalable programming models.
> Explore
> > techniques for threading, error checking, porting, and tuning. Get the
> most
> > from the latest Intel processors and coprocessors. See abstracts and
> register
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk_______________________________________________
> > Postgres-xc-general mailing list
> > Pos...@li...
> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
>

Hello, here is the process.

When we created table with DISTRIBUTE BY REPLICATION TO NODE ( datanode1 
), it has been inserting maybe 100 000 rows per minute.
When we created table with DISTRIBUTE BY REPLICATION TO NODE ( 
datanode1, datanode2 ), it has been inserting maybe 50 000 rows per minute.
When we created table witch  DISTRIBUTE BY MODULO ( cluster_id ) TO NODE 
( datanode1, datanode2 ) and has been writing to only one node, it was 
inserting 170 000 rows per minute.

But when we created it with partitioning, it has been 5000 rows per minute.

When we insert row into nonpartitioning table, no row was generated into 
the gtm log file. When we insert row into partitioning table, extra row 
is inserted into gtm log file : Saving transaction restoration info, 
backed-up gxid: 409215
LOCATION:  GTM_WriteRestorePointXid, gtm_txn.c:2649.

When deleting or updating from nonpartitioning table, everything is 
going quickly. But when we do the same thing with partitioning table, it 
is 3 - 4 times slower.

Below is example with partitioning. We are importing data to database, 
so first we insert unit_id into table units. This will generate schema 
for the year we are inserting, based on the column gps_timestamp. After 
that it create table for the unit in the schema, which inherits from 
primary talbe - units_data. Then we are inserting only to table units_data.

CREATE TABLE data.units ( unit_id uuid NOT NULL, cluster_id INT NOT NULL );

CREATE SEQUENCE ai_value;

CREATE TABLE data.units_data (
     row_id INTEGER NOT NULL DEFAULT nextval('ai_value'),
     unit_id uuid NOT NULL,
     cluster_id INT NOT NULL,
     gps_timestamp TIMESTAMP WITHOUT TIME ZONE NOT NULL,
     gps_lat DOUBLE PRECISION NOT NULL,
     gps_lng DOUBLE PRECISION NOT NULL,
     others hstore,
     PRIMARY KEY ( unit_id,cluster_id )
);

CREATE OR REPLACE FUNCTION generate_all(IN cluster_id INTEGER,  IN 
unit_id UUID) RETURNS VOID AS $$
DECLARE
     ROK VARCHAR(4);
     DATANODE_ID VARCHAR(1);
     NAZOV_SCHEMY VARCHAR(13);
     NAZOV_TABULKY VARCHAR;
     NAZOV_END_TABULKY VARCHAR;
     NAZOV_ROK_TABULKY VARCHAR;
BEGIN
     -- SELECT 2013 INTO ROK;
     ROK := 2013;
     SELECT cluster_id INTO DATANODE_ID;
     NAZOV_SCHEMY := 'node_'||DATANODE_ID||'_y_'||ROK;
     NAZOV_ROK_TABULKY := 'units_data_2013';
     NAZOV_TABULKY := 'u_'||replace(unit_id::text,'-','');
     NAZOV_END_TABULKY := NAZOV_TABULKY||'_m';
     -- Ide sa vytvarat schema a tabulky
         IF NOT EXISTS(SELECT * FROM information_schema.schemata WHERE 
schema_name=NAZOV_SCHEMY) THEN
             EXECUTE 'CREATE SCHEMA '||NAZOV_SCHEMY||';';
         END IF;
         IF NOT EXISTS(SELECT * FROM information_schema.tables where 
table_name=NAZOV_ROK_TABULKY AND table_schema=NAZOV_SCHEMY) THEN
             EXECUTE 'CREATE TABLE 
'||NAZOV_SCHEMY||'.'||NAZOV_ROK_TABULKY||'( CHECK ( 
date_part(''year''::text, gps_timestamp)=2013 ) ) INHERITS ( 
data.units_data );';
         END IF;
         EXECUTE 'CREATE OR REPLACE RULE 
"year2013_node'||DATANODE_ID||'_choice" AS ON INSERT TO data.units_data 
WHERE date_part(''year''::text, NEW.gps_timestamp)=2013 AND 
NEW.cluster_id='||DATANODE_ID||' DO INSTEAD INSERT INTO 
'||NAZOV_SCHEMY||'.units_data_2013 VALUES(NEW.*);';

         EXECUTE 'CREATE TABLE '||NAZOV_SCHEMY||'.'||NAZOV_TABULKY||'( 
CHECK ( unit_id='''||unit_id||'''::uuid ) ) INHERITS 
('||NAZOV_SCHEMY||'.'||NAZOV_ROK_TABULKY||' );';

         EXECUTE 'CREATE RULE "'||NAZOV_TABULKY||'_choice" AS ON INSERT 
TO '||NAZOV_SCHEMY||'.units_data_2013 WHERE unit_id='''||unit_id||''' DO 
INSTEAD INSERT INTO '||NAZOV_SCHEMY||'.'||NAZOV_TABULKY||' VALUES(NEW.*);';
     RETURN;
END
$$ LANGUAGE plpgsql;

CREATE RULE "generate_all_rule" AS ON INSERT TO data.units DO ALSO 
SELECT generate_all(NEW.cluster_id, NEW.unit_id);

Thank you, best regards
Matej


On 11. 11. 2013 10:11, 鈴木 幸市 wrote:
> This does not show anything special.   As I wrote, the messages in question don’t appear so often.     Could you test how many of these messages are written using another case, for example, without using partitioning.   I think partitioning table has nothing to do with the issue but I’d like to know if the case happens in other DMLs and DDLs.
>
> Thank you.
> ---
> Koichi Suzuki
>
> 2013/11/11 17:49、Matej Jellus <je...@ts...> のメール：
>
>> Our situation :
>> two machines
>> first - primary GTM, coordinator, datanode
>> second - standby GTM, coordinator, datanode
>>
>> GTM configuration below. We are inserting 5 thousand rows per minute, which is too low for us. We need to insert 100 thousand rows per minute.
>> Starting primary GTM : /gtm_ctl -Z gtm start -D /var/pgxc/data_gtm
>> Starting standby GTM : /gtm_ctl -Z gtm start -D /var/pgxc/data_gtm
>> Starting coord : pg_ctl start -D /var/pgxc/ubuntu2_coord -Z coordinator -l /tmp/logfile_ubuntu2_coord
>>
>> Tables in database are using partitioning. So we have one primary table and we are creating child tables from it. For example :
>> primary table :
>> chema - data
>> table - units_data
>>
>> child tables :
>> schema : y_2013_node1 - tables, which will be saved in node 1
>>    table : u_<unit_id> - child table from data.units_data
>> schema : y_2013_node2 - tables, which will be saved in node 2
>>    table : u_<unit_id> - child table from data.units_data
>>
>> Thank you, best regards
>> Matej
>>
>>
>> On 11. 11. 2013 7:49, 鈴木 幸市 wrote:
>>> Message 1. will be written once every 2000 transactions.    The number looks too large.   Message 1 is written in the three cases:
>>>
>>> 1) When GTM starts,
>>> 2) Once every 2000 transactions,
>>> 3) When GTM standby is promoted to a master.
>>>
>>> Could you let me know how you start your cluster?    We’ve tested XC with DBT-1 and DBT-2 but didn’t have such frequent write at gtm.   You are inserting 5000 raws in a second so 2) should happen twice or three times a second.
>>>
>>> Message 2. is written when GTM standby connects to GTM.   Are you using GTM standby?   If so, how you are starting it?
>>>
>>> We’re testing XC in different environment but didn’t have such GTM overload.   I’m interested in the situation.
>>>
>>> Best;
>>> ---
>>> Koichi Suzuki
>>>
>>> 2013/11/09 0:06、Matej Jellus <je...@ts...> のメール：
>>>
>>>> Hello,
>>>>
>>>> We have problem that postgres xc is very slow, gtm is writing to log a
>>>> lot of data. Is it able to be disabled?
>>>> The most repeated lines are :
>>>>
>>>> 1:140130870839040:2013-11-08 15:42:49.357 CET -LOG:  Saving transaction
>>>> restoration info, backed-up gxid: 373106
>>>> LOCATION:  GTM_WriteRestorePointXid, gtm_txn.c:2649
>>>> ( this is maybe 130 times per second )
>>>>
>>>> 1:140130879186688:2013-11-08 15:45:37.572 CET -LOG:  Connection
>>>> established with GTM standby. - 0x1f91398
>>>> LOCATION:  gtm_standby_connect_to_standby_int, gtm_standby.c:400
>>>>
>>>> It is overloading the disk, i/o is 70%.
>>>>
>>>> Now it is inserting 5 thousand inserts per minute. We need more inserts
>>>> per minute.
>>>>
>>>> Thank you, best regards
>>>> Matej
>>>>
>>>> ------------------------------------------------------------------------------
>>>> November Webinars for C, C++, Fortran Developers
>>>> Accelerate application performance with scalable programming models. Explore
>>>> techniques for threading, error checking, porting, and tuning. Get the most
>>>> from the latest Intel processors and coprocessors. See abstracts and register
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
>>>> _______________________________________________
>>>> Postgres-xc-general mailing list
>>>> Pos...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>>
>>

Someone is sending SIGKILL to coordinator/datanode backend processes.    Although XC (originally PG) code has a handler for SIGKILL, I didn’t found any code in XC sending SIGKILL to other processes.   I’m afraid there could be another process sending SIGKILL to them.

Best Regards;
---
Koichi Suzuki

2013/11/09 2:54、Sandeep Gupta <gup...@gm...> のメール：

> I am running single instance of postgres-xc with  several datanode
> Each data node runs on its own machine. 
> After instantiating the cluster the database sits idle. I do not perform any create table or insert operations. However, after some time all the datanode  automatically shuts down (log messages attached). 
> 
> Any pointers as to why this maybe happening would be very useful. 
> 
> -Sandeep
> 
> The log files across all datanode contain various error messages. These include:
> 
> 
> LOG:  statistics collector process (PID 23073) was terminated by signal 9: Killed
> LOG:  WAL writer process (PID 23071) was terminated by signal 9: Killed
> LOG:  terminating any other active server processes
> WARNING:  terminating connection because of crash of another server process
> DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
> HINT:  In a moment you should be able to reconnect to the database and repeat your command.
> LOG:  all server processes terminated; reinitializing
> 
> 
> LOG:  checkpointer process (PID 21406) was terminated by signal 9: Killed
> LOG:  terminating any other active server processes
> WARNING:  terminating connection because of crash of another server process
> DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
> LOG:  statistics collector process (PID 28881) was terminated by signal 9: Killed
> LOG:  autovacuum launcher process (PID 28880) was terminated by signal 9: Killed
> LOG:  terminating any other active server processes
> 
> 
> 
> 
> 
> ------------------------------------------------------------------------------
> November Webinars for C, C++, Fortran Developers
> Accelerate application performance with scalable programming models. Explore
> techniques for threading, error checking, porting, and tuning. Get the most 
> from the latest Intel processors and coprocessors. See abstracts and register
> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk_______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

Message 1. will be written once every 2000 transactions.    The number looks too large.   Message 1 is written in the three cases:

1) When GTM starts,
2) Once every 2000 transactions,
3) When GTM standby is promoted to a master.

Could you let me know how you start your cluster?    We’ve tested XC with DBT-1 and DBT-2 but didn’t have such frequent write at gtm.   You are inserting 5000 raws in a second so 2) should happen twice or three times a second.

Message 2. is written when GTM standby connects to GTM.   Are you using GTM standby?   If so, how you are starting it?

We’re testing XC in different environment but didn’t have such GTM overload.   I’m interested in the situation.

Best;
---
Koichi Suzuki

2013/11/09 0:06、Matej Jellus <je...@ts...> のメール：

> Hello,
> 
> We have problem that postgres xc is very slow, gtm is writing to log a 
> lot of data. Is it able to be disabled?
> The most repeated lines are :
> 
> 1:140130870839040:2013-11-08 15:42:49.357 CET -LOG:  Saving transaction 
> restoration info, backed-up gxid: 373106
> LOCATION:  GTM_WriteRestorePointXid, gtm_txn.c:2649
> ( this is maybe 130 times per second )
> 
> 1:140130879186688:2013-11-08 15:45:37.572 CET -LOG:  Connection 
> established with GTM standby. - 0x1f91398
> LOCATION:  gtm_standby_connect_to_standby_int, gtm_standby.c:400
> 
> It is overloading the disk, i/o is 70%.
> 
> Now it is inserting 5 thousand inserts per minute. We need more inserts 
> per minute.
> 
> Thank you, best regards
> Matej
> 
> ------------------------------------------------------------------------------
> November Webinars for C, C++, Fortran Developers
> Accelerate application performance with scalable programming models. Explore
> techniques for threading, error checking, porting, and tuning. Get the most 
> from the latest Intel processors and coprocessors. See abstracts and register
> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> 

I am running single instance of postgres-xc with  several datanode
Each data node runs on its own machine.
After instantiating the cluster the database sits idle. I do not perform
any create table or insert operations. However, after some time all the
datanode  automatically shuts down (log messages attached).

Any pointers as to why this maybe happening would be very useful.

-Sandeep

The log files across all datanode contain various error messages. These
include:


LOG:  statistics collector process (PID 23073) was terminated by signal 9:
Killed
LOG:  WAL writer process (PID 23071) was terminated by signal 9: Killed
LOG:  terminating any other active server processes
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the
current transaction and exit, because another server process exited
abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and
repeat your command.
LOG:  all server processes terminated; reinitializing


LOG:  checkpointer process (PID 21406) was terminated by signal 9: Killed
LOG:  terminating any other active server processes
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the
current transaction and exit, because another server process exited
abnormally and possibly corrupted shared memory.
LOG:  statistics collector process (PID 28881) was terminated by signal 9:
Killed
LOG:  autovacuum launcher process (PID 28880) was terminated by signal 9:
Killed
LOG:  terminating any other active server processes

Hello,

We have problem that postgres xc is very slow, gtm is writing to log a 
lot of data. Is it able to be disabled?
The most repeated lines are :

1:140130870839040:2013-11-08 15:42:49.357 CET -LOG:  Saving transaction 
restoration info, backed-up gxid: 373106
LOCATION:  GTM_WriteRestorePointXid, gtm_txn.c:2649
( this is maybe 130 times per second )

1:140130879186688:2013-11-08 15:45:37.572 CET -LOG:  Connection 
established with GTM standby. - 0x1f91398
LOCATION:  gtm_standby_connect_to_standby_int, gtm_standby.c:400

It is overloading the disk, i/o is 70%.

Now it is inserting 5 thousand inserts per minute. We need more inserts 
per minute.

Thank you, best regards
Matej

If you find any inconsistency in replicated table replica, here's another means to repair them.   If you find inconsistent replica on datanode D, you can issue ALTER TABLE twice to repair them.

ALTER TABLE T DROP NODE D;
ALTER TABLE T ADD NODE D;

If you have more than one inconsistent replica, you should drop all such replica with ALTER TABLE DROP NODE and then repair them by ALTER TABLE ADD NODE.

This is available in release 1.1 or later.

Regards;
---
Koichi Suzuki

On 2013/11/07, at 2:16, Alex Hudson <ale...@au...<mailto:ale...@au...>> wrote:

Thanks a lot Koichi!!

We are working with replicated tables in a 3 nodes cluster. We’ll use "EXECUTE DIRECT" statement to be sure if all the replica are identical.

Thanks and Regards!

Alex Hudson S.

De: Koichi Suzuki [mailto:koi...@gm...<http://gmail.com>]
Enviado el: martes, 05 de noviembre de 2013 22:48
Para: Alex Hudson
CC: Postgres-XC mailing list
Asunto: Re: [Postgres-xc-general] consistency check on SPI tuple count failed

Thanks a lot for the report.

Could you let me know if you got this message while handling replicated table or distributed table?

Most probable case is inconsistency in replicated table's replica.   Before looking into it, could you check if there's no outstanding 2PC transactions in your datanodes.   Even though you don't issue 2PC commands, XC may issue it internally when more than one node are involved in updating transactions and you may have such outstanding 2PC transactions especially when some node fails (did you have such failure?).

You can check this by selecting pg_prepared_statements catalog.   This catalog is local to each coordinator and datanode so you should issue select statement using EXECUTE DIRECT statement.   If you find such outstanding 2PC transactions, you can rollback them as follows:

1. Connect psql directly to a target datanode/coordinator as a superuser.
2. Issue "SET xc_maintenance_mode to ON" to enable local manual repair.
3. Issue ROLLBACK PREPARED statement to rollback such outstanding 2PCs.
4. Repeat 1 to 3 for all the coordinator/datanodes which contains such outstanding 2PCs.

Postgres-XC provides pgxc_clean utility to do the above step.   This will be found in contrib directory of the source tarball.

If you don't have such outstanding 2PC transactions or you still have the same error, please try the following.

If you are handling replicated tables, we have a chance that there's some inconsistencies in each replica.   To check this, use "EXECUTE DIRECT" statement to visit each datanode and check if all the replica are identical.   If not, you can recover damaged tuples as follows:

1. Connect psql directly to a target datanode as a superuser.
2. Issue "SET xc_maintenance_mode to ON" to enable local manual repair.
3. Issue SQL statement to repair damaged tuples.
4. Repeat 1 to 3 for all the damaged datanode.

If you've got the error while handling distributed tables, there could be another error cause.   I need to analylze the case more in detail.   Please let me know if you are handling distributed tables.

Best Regards;

---
Koichi Suzuki

2013/11/6 Alex Hudson <ale...@au...<mailto:ale...@au...>>
Hi guys,

I’m working in an application using php and Postgrexc clustered.
In the event of save some information, the app shows me the following lines:

Application raised an exception class <b>ADODB_Exception</b> with message <b>'postgres7 error: [-1: ERROR:  consistency check on SPI tuple count failed

Why produces this error? How can i fix it? May be because I’m using  encoding LATIN1?

Thanks and regards,

Alex Hudson S.


------------------------------------------------------------------------------
November Webinars for C, C++, Fortran Developers
Accelerate application performance with scalable programming models. Explore
techniques for threading, error checking, porting, and tuning. Get the most
from the latest Intel processors and coprocessors. See abstracts and register
http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
_______________________________________________
Postgres-xc-general mailing list
Pos...@li...<mailto:Pos...@li...>
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

------------------------------------------------------------------------------
November Webinars for C, C++, Fortran Developers
Accelerate application performance with scalable programming models. Explore
techniques for threading, error checking, porting, and tuning. Get the most
from the latest Intel processors and coprocessors. See abstracts and register
http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk_______________________________________________
Postgres-xc-general mailing list
Pos...@li...
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

Thanks a lot Koichi!! 

 

We are working with replicated tables in a 3 nodes cluster. We'll use
"EXECUTE DIRECT" statement to be sure if all the replica are identical. 

 

Thanks and Regards!

 

Alex Hudson S.

 

De: Koichi Suzuki [mailto:koi...@gm...] 
Enviado el: martes, 05 de noviembre de 2013 22:48
Para: Alex Hudson
CC: Postgres-XC mailing list
Asunto: Re: [Postgres-xc-general] consistency check on SPI tuple count
failed

 

Thanks a lot for the report.

 

Could you let me know if you got this message while handling replicated
table or distributed table?

 

Most probable case is inconsistency in replicated table's replica.   Before
looking into it, could you check if there's no outstanding 2PC transactions
in your datanodes.   Even though you don't issue 2PC commands, XC may issue
it internally when more than one node are involved in updating transactions
and you may have such outstanding 2PC transactions especially when some node
fails (did you have such failure?).

 

You can check this by selecting pg_prepared_statements catalog.   This
catalog is local to each coordinator and datanode so you should issue select
statement using EXECUTE DIRECT statement.   If you find such outstanding 2PC
transactions, you can rollback them as follows:

 

1. Connect psql directly to a target datanode/coordinator as a superuser.

2. Issue "SET xc_maintenance_mode to ON" to enable local manual repair.

3. Issue ROLLBACK PREPARED statement to rollback such outstanding 2PCs.

4. Repeat 1 to 3 for all the coordinator/datanodes which contains such
outstanding 2PCs.

 

Postgres-XC provides pgxc_clean utility to do the above step.   This will be
found in contrib directory of the source tarball.

 

If you don't have such outstanding 2PC transactions or you still have the
same error, please try the following.

 

If you are handling replicated tables, we have a chance that there's some
inconsistencies in each replica.   To check this, use "EXECUTE DIRECT"
statement to visit each datanode and check if all the replica are identical.
If not, you can recover damaged tuples as follows:

 

1. Connect psql directly to a target datanode as a superuser.

2. Issue "SET xc_maintenance_mode to ON" to enable local manual repair.

3. Issue SQL statement to repair damaged tuples.

4. Repeat 1 to 3 for all the damaged datanode.

 

If you've got the error while handling distributed tables, there could be
another error cause.   I need to analylze the case more in detail.   Please
let me know if you are handling distributed tables.

 

Best Regards;




---

Koichi Suzuki

 

2013/11/6 Alex Hudson <ale...@au...>

Hi guys,

 

I'm working in an application using php and Postgrexc clustered.

In the event of save some information, the app shows me the following lines:

 

Application raised an exception class <b>ADODB_Exception</b> with message
<b>'postgres7 error: [-1: ERROR:  consistency check on SPI tuple count
failed

 

Why produces this error? How can i fix it? May be because I'm using
encoding LATIN1?

 

Thanks and regards,

 

Alex Hudson S.

 


----------------------------------------------------------------------------
--
November Webinars for C, C++, Fortran Developers
Accelerate application performance with scalable programming models. Explore
techniques for threading, error checking, porting, and tuning. Get the most
from the latest Intel processors and coprocessors. See abstracts and
register
http://pubads.g.doubleclick.net/gampad/clk?id=60136231
<http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
> &iu=/4140/ostg.clktrk
_______________________________________________
Postgres-xc-general mailing list
Pos...@li...
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general