Re: [Postgres-xc-general] "invalid memory alloc request size ..." error

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

I think I'll try to output a detailed log of this matter.
I will output a detailed log with the following settings.

Postgres-XC
log_min_messages = debug5

Server
iostat and vmstat

Some time it may take, but I will report the results.

Regards,
uwaizumi

(2013/08/16 14:28), Nikhil Sontakke wrote:
> What I think is happening here is that the COPY process is trying to send
> data to the datanodes, but they are not consuming it at the same rate.
> Hence the send_some call sends only a portion of the data and the rest of
> the data has to remain buffered. This builds up eventually to more than
> half GB and the subsequent repalloc call to double the buffer fails because
> 1GB is the max limit. This is evident from the below COPY error message as
> well.
> 
> What we should do is maybe if the data outbuffer size crosses a pre-defined
> limit like 16MB or so, then we should use pgxc_node_flush to ensure that
> all of that has reached the datanode. After some point even the socket will
> not be handle direct large send_some calls and we will increase the buffer
> size unnecessarily.
> 
> Regards,
> Nikhils
> 
> 
> On Fri, Aug 16, 2013 at 8:37 AM, Nikhil Sontakke <ni...@st...>wrote:
> 
>> Thanks Eiji san,
>>
>>
>>
>>>
>>> At the request of Hemmi-san,
>>> By applying the patch to the Postgres-XC v1.1, I have conducted a DBT-1.
>>> However, the results did not change unfortunately.
>>>
>>> [results]
>>> $ psql -p 5422 -c "copy address from '/tmp/address.data' delimiter '>';"
>>> ERROR:  invalid memory alloc request size 1073741824
>>> CONTEXT:  COPY address, line 30305382:
>>> "30305382>TXnEREEwqOLV11zEFGDXV4AXEDEeqLn9JDDwGxHw>TCyFEgXDNVOCihBdmdExEBAAvnXdvEEBDEE>TGEgxF5RdiuAEz..."
>>>
>>>
>> Yeah, as I slept over it I realized that what I am trying to solve is a
>> "COPY TO" issue and what you guys must be facing is a "COPY FROM" error. I
>> will try to look at this after when I can make more progress on the "COPY
>> TO" issue.
>>
>> Regards,
>> Nikhils
>>
>>> Regards,
>>> uwaizumi
>>>
>>>
>>> (2013/08/15 18:06), Hitoshi HEMMI wrote:
>>>> Hi Nikhil,
>>>>
>>>> Thank you for the patch.
>>>> I will be off from tomorrow, but I ask our team to test it
>>>> and post the result.
>>>>
>>>> Best,
>>>> -hemmi
>>>>
>>>>
>>>> Nikhil Sontakke さんは書きました:
>>>>> Hi Hitoshi-san,
>>>>>
>>>>> PFA, patch which should help to use less memory while running COPY
>>>>> commands. This is against REL 1.1. Please do let me know if this helps
>>>>> in your case.
>>>>>
>>>>> Regards,
>>>>> Nikhils
>>>>>
>>>>>
>>>>> On Wed, Aug 14, 2013 at 3:38 PM, Hitoshi HEMMI
>>>>> <hem...@la... <mailto:hem...@la...>>
>>> wrote:
>>>>>
>>>>>       Thank you, Nikhil.
>>>>>       We will wait until a patch is available.
>>>>>
>>>>>       Best,
>>>>>       -hemmi
>>>>>
>>>>>
>>>>>       Nikhil Sontakke さんは書きました:
>>>>>       > Hi Hitoshi-san,
>>>>>       >
>>>>>       > I am working on a similar issue coincidentally. Basically what's
>>>>>       > happening is that the COPY command is creating tuples which
>>> will get
>>>>>       > freed up only at the end when the command will finish, so the
>>> RAM
>>>>>       > memory ends up being consumed and can lead to the error that you
>>>>>       have
>>>>>       > provided.
>>>>>       >
>>>>>       > See the discussion around RemoteQueryNext on another thread.
>>>>>       >
>>>>>       > Regards,
>>>>>       > Nikhils
>>>>>       >
>>>>>       > On Wed, Aug 14, 2013 at 1:56 PM, Hitoshi HEMMI
>>>>>       > <hem...@la...
>>>>>       <mailto:hem...@la...>
>>>>>       <mailto:hem...@la...
>>>>>       <mailto:hem...@la...>>> wrote:
>>>>>       >
>>>>>       > Hi,
>>>>>       >
>>>>>       > We encountered "invalid memory alloc request size ..." error of
>>>>>       > Postgres-XC, when preparing DBT-1 benchmark, and need
>>>>>       > someones help.
>>>>>       > (First of all, what does the error message mean?)
>>>>>       >
>>>>>       > ------------------
>>>>>       > Version of XC: v1.1.beta
>>>>>       >
>>>>>       > [process to cause an error]
>>>>>       > - Data generation
>>>>>       > cd <DBT-1 dir>/datagen/
>>>>>       > ./datagen -i 10000 -u 15000 -p <data dir>
>>>>>       >
>>>>>       > - Prepare XC cluster
>>>>>       > We are using following HA configuration.
>>>>>       > We are not sure this configuration is essential for the error,
>>>>>       > but we have never encountered the error without XC slave nodes.
>>>>>       >
>>>>>       > ServerA: GTM (master)
>>>>>       > ServerB: GTM (slave)
>>>>>       > ServerC: GTM-Proxy, Coordinator1 (master), Datanode1 (master),
>>>>>       > Coordinator2 (slave), Datanode2 (slave)
>>>>>       > ServerD: GTM-Proxy, Coordinator2 (master), Datanode2 (master),
>>>>>       > Coordinator1 (slave), Datanode1 (slave)
>>>>>       >
>>>>>       > - Data loading
>>>>>       > $ createdb -E SQL_ASCII -T template0 -p 5422 DBT1
>>>>>       > $ psql -a -p 5422 -f <DBT-1 dir>/scripts/pgsql/create_tables.sql
>>>>>       > $ psql -p 5422 -c "copy address from '/tmp/address.data'
>>> delimiter
>>>>>       > '>';"
>>>>>       >
>>>>>       > [the error]
>>>>>       > ERROR: invalid memory alloc request size 1073741824
>>>>>       > CONTEXT: COPY address, line 33191219:
>>>>>       > "33191219>EDpcFnEEAe954E5Tg6XbkiBVGLAcuHBhFAEElEE7>zxPzRj
>>>>>       >
>>>>>       > [Other information]
>>>>>       > It seems small sized data can be copied:
>>>>>       >
>>>>>       > - DBT-1 tables with size (item 10000 ,user 15000)
>>>>>       > address 4.7G ERROR
>>>>>       > author 894K OK
>>>>>       > cc_xacts 4.4G ERROR
>>>>>       > customer 20G ERROR
>>>>>       > item 5.0M OK
>>>>>       > order_line 11G ERROR
>>>>>       > orders 4.1G ERROR
>>>>>       > stock 78K OK
>>>>>       > --------------------
>>>>>       >
>>>>>       > Best regards,
>>>>>       >
>>>>>       > -hemmi
>>>>>       >
>>>>>       > --
>>>>>       > Hitoshi HEMMI
>>>>>       > NTT Open Source Software Center
>>>>>       > hem...@la... <mailto:hem...@la...
>>>>
>>>>>       <mailto:hem...@la...
>>>>>       <mailto:hem...@la...>>
>>>>>       > Tel:(03)5860-5115
>>>>>       > Fax:(03)5463-5490
>>>>>       >
>>>>>       >
>>>>>       >
>>>>>
>>>   ------------------------------------------------------------------------------
>>>>>       > Get 100% visibility into Java/.NET code with AppDynamics Lite!
>>>>>       > It's a free troubleshooting tool designed for production.
>>>>>       > Get down to code-level detail for bottlenecks, with <2%
>>> overhead.
>>>>>       > Download for free and get started troubleshooting in minutes.
>>>>>       >
>>>>>
>>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>>>>>       <
>>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>>>>
>>>>>       >
>>>>>       <
>>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>>>>>       <
>>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>>>>>
>>>>>       > _______________________________________________
>>>>>       > Postgres-xc-general mailing list
>>>>>       > Pos...@li...
>>>>>       <mailto:Pos...@li...>
>>>>>       > <mailto:Pos...@li...
>>>>>       <mailto:Pos...@li...>>
>>>>>       >
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>>>       >
>>>>>       >
>>>>>       >
>>>>>       >
>>>>>       > --
>>>>>       > StormDB - http://www.stormdb.com
>>>>>       > The Database Cloud
>>>>>
>>>>>
>>>>>       --
>>>>>       Hitoshi HEMMI
>>>>>       NTT Open Source Software Center
>>>>>       hem...@la... <mailto:hem...@la...>
>>>>>       Tel:(03)5860-5115
>>>>>       Fax:(03)5463-5490
>>>>>
>>>>>
>>>>>
>>>   ------------------------------------------------------------------------------
>>>>>       Get 100% visibility into Java/.NET code with AppDynamics Lite!
>>>>>       It's a free troubleshooting tool designed for production.
>>>>>       Get down to code-level detail for bottlenecks, with <2% overhead.
>>>>>       Download for free and get started troubleshooting in minutes.
>>>>>
>>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>>>>>       <
>>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>>>>
>>>>>       _______________________________________________
>>>>>       Postgres-xc-general mailing list
>>>>>       Pos...@li...
>>>>>       <mailto:Pos...@li...>
>>>>>       https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> StormDB - http://www.stormdb.com
>>>>> The Database Cloud
>>>>
>>>>
>>>
>>>
>>> --
>>> -----------------------------------
>>>    上泉 英二
>>>    uwa...@as...
>>> -----------------------------------
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Get 100% visibility into Java/.NET code with AppDynamics Lite!
>>> It's a free troubleshooting tool designed for production.
>>> Get down to code-level detail for bottlenecks, with <2% overhead.
>>> Download for free and get started troubleshooting in minutes.
>>>
>>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>>> _______________________________________________
>>> Postgres-xc-general mailing list
>>> Pos...@li...
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>
>>
>>
>>
>> --
>> StormDB - http://www.stormdb.com
>> The Database Cloud
>>
> 
> 
> 