postgres-xc-general Mailing List for Postgres-XC

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

What I think is happening here is that the COPY process is trying to send
data to the datanodes, but they are not consuming it at the same rate.
Hence the send_some call sends only a portion of the data and the rest of
the data has to remain buffered. This builds up eventually to more than
half GB and the subsequent repalloc call to double the buffer fails because
1GB is the max limit. This is evident from the below COPY error message as
well.

What we should do is maybe if the data outbuffer size crosses a pre-defined
limit like 16MB or so, then we should use pgxc_node_flush to ensure that
all of that has reached the datanode. After some point even the socket will
not be handle direct large send_some calls and we will increase the buffer
size unnecessarily.

Regards,
Nikhils

On Fri, Aug 16, 2013 at 8:37 AM, Nikhil Sontakke <ni...@st...>wrote:

> Thanks Eiji san,
>
>
>
>>
>> At the request of Hemmi-san,
>> By applying the patch to the Postgres-XC v1.1, I have conducted a DBT-1.
>> However, the results did not change unfortunately.
>>
>> [results]
>> $ psql -p 5422 -c "copy address from '/tmp/address.data' delimiter '>';"
>> ERROR:  invalid memory alloc request size 1073741824
>> CONTEXT:  COPY address, line 30305382:
>> "30305382>TXnEREEwqOLV11zEFGDXV4AXEDEeqLn9JDDwGxHw>TCyFEgXDNVOCihBdmdExEBAAvnXdvEEBDEE>TGEgxF5RdiuAEz..."
>>
>>
> Yeah, as I slept over it I realized that what I am trying to solve is a
> "COPY TO" issue and what you guys must be facing is a "COPY FROM" error. I
> will try to look at this after when I can make more progress on the "COPY
> TO" issue.
>
> Regards,
> Nikhils
>
>> Regards,
>> uwaizumi
>>
>>
>> (2013/08/15 18:06), Hitoshi HEMMI wrote:
>> > Hi Nikhil,
>> >
>> > Thank you for the patch.
>> > I will be off from tomorrow, but I ask our team to test it
>> > and post the result.
>> >
>> > Best,
>> > -hemmi
>> >
>> >
>> > Nikhil Sontakke さんは書きました:
>> >> Hi Hitoshi-san,
>> >>
>> >> PFA, patch which should help to use less memory while running COPY
>> >> commands. This is against REL 1.1. Please do let me know if this helps
>> >> in your case.
>> >>
>> >> Regards,
>> >> Nikhils
>> >>
>> >>
>> >> On Wed, Aug 14, 2013 at 3:38 PM, Hitoshi HEMMI
>> >> <hem...@la... <mailto:hem...@la...>>
>> wrote:
>> >>
>> >>      Thank you, Nikhil.
>> >>      We will wait until a patch is available.
>> >>
>> >>      Best,
>> >>      -hemmi
>> >>
>> >>
>> >>      Nikhil Sontakke さんは書きました:
>> >>      > Hi Hitoshi-san,
>> >>      >
>> >>      > I am working on a similar issue coincidentally. Basically what's
>> >>      > happening is that the COPY command is creating tuples which
>> will get
>> >>      > freed up only at the end when the command will finish, so the
>> RAM
>> >>      > memory ends up being consumed and can lead to the error that you
>> >>      have
>> >>      > provided.
>> >>      >
>> >>      > See the discussion around RemoteQueryNext on another thread.
>> >>      >
>> >>      > Regards,
>> >>      > Nikhils
>> >>      >
>> >>      > On Wed, Aug 14, 2013 at 1:56 PM, Hitoshi HEMMI
>> >>      > <hem...@la...
>> >>      <mailto:hem...@la...>
>> >>      <mailto:hem...@la...
>> >>      <mailto:hem...@la...>>> wrote:
>> >>      >
>> >>      > Hi,
>> >>      >
>> >>      > We encountered "invalid memory alloc request size ..." error of
>> >>      > Postgres-XC, when preparing DBT-1 benchmark, and need
>> >>      > someones help.
>> >>      > (First of all, what does the error message mean?)
>> >>      >
>> >>      > ------------------
>> >>      > Version of XC: v1.1.beta
>> >>      >
>> >>      > [process to cause an error]
>> >>      > - Data generation
>> >>      > cd <DBT-1 dir>/datagen/
>> >>      > ./datagen -i 10000 -u 15000 -p <data dir>
>> >>      >
>> >>      > - Prepare XC cluster
>> >>      > We are using following HA configuration.
>> >>      > We are not sure this configuration is essential for the error,
>> >>      > but we have never encountered the error without XC slave nodes.
>> >>      >
>> >>      > ServerA: GTM (master)
>> >>      > ServerB: GTM (slave)
>> >>      > ServerC: GTM-Proxy, Coordinator1 (master), Datanode1 (master),
>> >>      > Coordinator2 (slave), Datanode2 (slave)
>> >>      > ServerD: GTM-Proxy, Coordinator2 (master), Datanode2 (master),
>> >>      > Coordinator1 (slave), Datanode1 (slave)
>> >>      >
>> >>      > - Data loading
>> >>      > $ createdb -E SQL_ASCII -T template0 -p 5422 DBT1
>> >>      > $ psql -a -p 5422 -f <DBT-1 dir>/scripts/pgsql/create_tables.sql
>> >>      > $ psql -p 5422 -c "copy address from '/tmp/address.data'
>> delimiter
>> >>      > '>';"
>> >>      >
>> >>      > [the error]
>> >>      > ERROR: invalid memory alloc request size 1073741824
>> >>      > CONTEXT: COPY address, line 33191219:
>> >>      > "33191219>EDpcFnEEAe954E5Tg6XbkiBVGLAcuHBhFAEElEE7>zxPzRj
>> >>      >
>> >>      > [Other information]
>> >>      > It seems small sized data can be copied:
>> >>      >
>> >>      > - DBT-1 tables with size (item 10000 ,user 15000)
>> >>      > address 4.7G ERROR
>> >>      > author 894K OK
>> >>      > cc_xacts 4.4G ERROR
>> >>      > customer 20G ERROR
>> >>      > item 5.0M OK
>> >>      > order_line 11G ERROR
>> >>      > orders 4.1G ERROR
>> >>      > stock 78K OK
>> >>      > --------------------
>> >>      >
>> >>      > Best regards,
>> >>      >
>> >>      > -hemmi
>> >>      >
>> >>      > --
>> >>      > Hitoshi HEMMI
>> >>      > NTT Open Source Software Center
>> >>      > hem...@la... <mailto:hem...@la...
>> >
>> >>      <mailto:hem...@la...
>> >>      <mailto:hem...@la...>>
>> >>      > Tel:(03)5860-5115
>> >>      > Fax:(03)5463-5490
>> >>      >
>> >>      >
>> >>      >
>> >>
>>  ------------------------------------------------------------------------------
>> >>      > Get 100% visibility into Java/.NET code with AppDynamics Lite!
>> >>      > It's a free troubleshooting tool designed for production.
>> >>      > Get down to code-level detail for bottlenecks, with <2%
>> overhead.
>> >>      > Download for free and get started troubleshooting in minutes.
>> >>      >
>> >>
>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>> >>      <
>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>> >
>> >>      >
>> >>      <
>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>> >>      <
>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>> >>
>> >>      > _______________________________________________
>> >>      > Postgres-xc-general mailing list
>> >>      > Pos...@li...
>> >>      <mailto:Pos...@li...>
>> >>      > <mailto:Pos...@li...
>> >>      <mailto:Pos...@li...>>
>> >>      >
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>> >>      >
>> >>      >
>> >>      >
>> >>      >
>> >>      > --
>> >>      > StormDB - http://www.stormdb.com
>> >>      > The Database Cloud
>> >>
>> >>
>> >>      --
>> >>      Hitoshi HEMMI
>> >>      NTT Open Source Software Center
>> >>      hem...@la... <mailto:hem...@la...>
>> >>      Tel:(03)5860-5115
>> >>      Fax:(03)5463-5490
>> >>
>> >>
>> >>
>>  ------------------------------------------------------------------------------
>> >>      Get 100% visibility into Java/.NET code with AppDynamics Lite!
>> >>      It's a free troubleshooting tool designed for production.
>> >>      Get down to code-level detail for bottlenecks, with <2% overhead.
>> >>      Download for free and get started troubleshooting in minutes.
>> >>
>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>> >>      <
>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>> >
>> >>      _______________________________________________
>> >>      Postgres-xc-general mailing list
>> >>      Pos...@li...
>> >>      <mailto:Pos...@li...>
>> >>      https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> StormDB - http://www.stormdb.com
>> >> The Database Cloud
>> >
>> >
>>
>>
>> --
>> -----------------------------------
>>   上泉 英二
>>   uwa...@as...
>> -----------------------------------
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Get 100% visibility into Java/.NET code with AppDynamics Lite!
>> It's a free troubleshooting tool designed for production.
>> Get down to code-level detail for bottlenecks, with <2% overhead.
>> Download for free and get started troubleshooting in minutes.
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Postgres-xc-general mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>
>
>
>
> --
> StormDB - http://www.stormdb.com
> The Database Cloud
>

-- 
StormDB - http://www.stormdb.com
The Database Cloud

2010	Jan	Feb	Mar	Apr	May (2)	Jun	Jul	Aug (6)	Sep	Oct (19)	Nov (1)	Dec
2011	Jan (12)	Feb (1)	Mar (4)	Apr (4)	May (32)	Jun (12)	Jul (11)	Aug (1)	Sep (6)	Oct (3)	Nov	Dec (10)
2012	Jan (11)	Feb (1)	Mar (3)	Apr (25)	May (53)	Jun (38)	Jul (103)	Aug (54)	Sep (31)	Oct (66)	Nov (77)	Dec (20)
2013	Jan (91)	Feb (86)	Mar (103)	Apr (107)	May (25)	Jun (37)	Jul (17)	Aug (59)	Sep (38)	Oct (78)	Nov (29)	Dec (15)
2014	Jan (23)	Feb (82)	Mar (118)	Apr (101)	May (103)	Jun (45)	Jul (6)	Aug (10)	Sep	Oct (32)	Nov	Dec (9)
2015	Jan (3)	Feb (5)	Mar	Apr (1)	May	Jun	Jul (9)	Aug (4)	Sep (3)	Oct	Nov	Dec
2016	Jan (3)	Feb	Mar	Apr	May	Jun	Jul	Aug	Sep	Oct	Nov	Dec
2017	Jan	Feb	Mar	Apr	May	Jun (3)	Jul	Aug	Sep	Oct	Nov	Dec
2018	Jan	Feb	Mar	Apr	May (4)	Jun	Jul	Aug	Sep	Oct	Nov	Dec

S	M	T	W	T	F	S
				1	2 (2)	3 (2)
4	5 (6)	6 (8)	7 (10)	8 (5)	9 (1)	10
11	12 (2)	13	14 (3)	15 (2)	16 (3)	17
18	19 (1)	20 (2)	21 (3)	22 (5)	23 (2)	24
25 (2)	26	27	28	29	30	31

postgres-xc-general Mailing List for Postgres-XC

postgres-xc-general — General info and messages