postgres-xc-general Mailing List for Postgres-XC

Hi Nikhil,

Thank you for the patch.
I will be off from tomorrow, but I ask our team to test it
and post the result.

Best,
-hemmi


Nikhil Sontakke さんは書きました:
> Hi Hitoshi-san,
>
> PFA, patch which should help to use less memory while running COPY
> commands. This is against REL 1.1. Please do let me know if this helps
> in your case.
>
> Regards,
> Nikhils
>
>
> On Wed, Aug 14, 2013 at 3:38 PM, Hitoshi HEMMI
> <hem...@la... <mailto:hem...@la...>> wrote:
>
>     Thank you, Nikhil.
>     We will wait until a patch is available.
>
>     Best,
>     -hemmi
>
>
>     Nikhil Sontakke さんは書きました:
>     > Hi Hitoshi-san,
>     >
>     > I am working on a similar issue coincidentally. Basically what's
>     > happening is that the COPY command is creating tuples which will get
>     > freed up only at the end when the command will finish, so the RAM
>     > memory ends up being consumed and can lead to the error that you
>     have
>     > provided.
>     >
>     > See the discussion around RemoteQueryNext on another thread.
>     >
>     > Regards,
>     > Nikhils
>     >
>     > On Wed, Aug 14, 2013 at 1:56 PM, Hitoshi HEMMI
>     > <hem...@la...
>     <mailto:hem...@la...>
>     <mailto:hem...@la...
>     <mailto:hem...@la...>>> wrote:
>     >
>     > Hi,
>     >
>     > We encountered "invalid memory alloc request size ..." error of
>     > Postgres-XC, when preparing DBT-1 benchmark, and need
>     > someones help.
>     > (First of all, what does the error message mean?)
>     >
>     > ------------------
>     > Version of XC: v1.1.beta
>     >
>     > [process to cause an error]
>     > - Data generation
>     > cd <DBT-1 dir>/datagen/
>     > ./datagen -i 10000 -u 15000 -p <data dir>
>     >
>     > - Prepare XC cluster
>     > We are using following HA configuration.
>     > We are not sure this configuration is essential for the error,
>     > but we have never encountered the error without XC slave nodes.
>     >
>     > ServerA: GTM (master)
>     > ServerB: GTM (slave)
>     > ServerC: GTM-Proxy, Coordinator1 (master), Datanode1 (master),
>     > Coordinator2 (slave), Datanode2 (slave)
>     > ServerD: GTM-Proxy, Coordinator2 (master), Datanode2 (master),
>     > Coordinator1 (slave), Datanode1 (slave)
>     >
>     > - Data loading
>     > $ createdb -E SQL_ASCII -T template0 -p 5422 DBT1
>     > $ psql -a -p 5422 -f <DBT-1 dir>/scripts/pgsql/create_tables.sql
>     > $ psql -p 5422 -c "copy address from '/tmp/address.data' delimiter
>     > '>';"
>     >
>     > [the error]
>     > ERROR: invalid memory alloc request size 1073741824
>     > CONTEXT: COPY address, line 33191219:
>     > "33191219>EDpcFnEEAe954E5Tg6XbkiBVGLAcuHBhFAEElEE7>zxPzRj
>     >
>     > [Other information]
>     > It seems small sized data can be copied:
>     >
>     > - DBT-1 tables with size (item 10000 ,user 15000)
>     > address 4.7G ERROR
>     > author 894K OK
>     > cc_xacts 4.4G ERROR
>     > customer 20G ERROR
>     > item 5.0M OK
>     > order_line 11G ERROR
>     > orders 4.1G ERROR
>     > stock 78K OK
>     > --------------------
>     >
>     > Best regards,
>     >
>     > -hemmi
>     >
>     > --
>     > Hitoshi HEMMI
>     > NTT Open Source Software Center
>     > hem...@la... <mailto:hem...@la...>
>     <mailto:hem...@la...
>     <mailto:hem...@la...>>
>     > Tel:(03)5860-5115
>     > Fax:(03)5463-5490
>     >
>     >
>     >
>     ------------------------------------------------------------------------------
>     > Get 100% visibility into Java/.NET code with AppDynamics Lite!
>     > It's a free troubleshooting tool designed for production.
>     > Get down to code-level detail for bottlenecks, with <2% overhead.
>     > Download for free and get started troubleshooting in minutes.
>     >
>     http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>     <http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk>
>     >
>     <http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>     <http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk>>
>     > _______________________________________________
>     > Postgres-xc-general mailing list
>     > Pos...@li...
>     <mailto:Pos...@li...>
>     > <mailto:Pos...@li...
>     <mailto:Pos...@li...>>
>     > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>     >
>     >
>     >
>     >
>     > --
>     > StormDB - http://www.stormdb.com
>     > The Database Cloud
>
>
>     --
>     Hitoshi HEMMI
>     NTT Open Source Software Center
>     hem...@la... <mailto:hem...@la...>
>     Tel:(03)5860-5115
>     Fax:(03)5463-5490
>
>
>     ------------------------------------------------------------------------------
>     Get 100% visibility into Java/.NET code with AppDynamics Lite!
>     It's a free troubleshooting tool designed for production.
>     Get down to code-level detail for bottlenecks, with <2% overhead.
>     Download for free and get started troubleshooting in minutes.
>     http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>     <http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk>
>     _______________________________________________
>     Postgres-xc-general mailing list
>     Pos...@li...
>     <mailto:Pos...@li...>
>     https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
>
>
>
> -- 
> StormDB - http://www.stormdb.com
> The Database Cloud 


-- 
Hitoshi HEMMI
NTT Open Source Software Center
hem...@la...
Tel:(03)5860-5115
Fax:(03)5463-5490