Re: [Postgres-xc-general] tracing cause of datanode crash

Could you let me know what "line 682 -- 637" means?

Because any communication between coordinator and datanode is done
using libpq, I don't think XC populates information using raw
stringinfo.   In fact, from datanodes, coordinator is just a client
except for supplying XID and snapshot.   However,  COPY TO uses
send()/receive(), not through libpq, to send data to datanode.  In
this case, if incoming data file contains NULL in the data, such data
may be sent via send() to datanodes.   COPY TO assumes incoming data
in a text format.   Do you think you have such situations?

Analysis may take a bit but let me try.

Regards;
---
Koichi Suzuki


2013/11/15 Sandeep Gupta <gup...@gm...>:
> Hi Koichi, Masataka,
>
>  Thanks for taking a look. I double checked both memory usage and the dmesg
> log. The kill is not happening due to the out of memory. The kernel doesn't
> show any processes being killed.
>
> My current suspicion is that something amiss is happening over the network.
> This happens consistently when performing copy over large datasets. Few of
> datanodes end up with "invalid string in message".
> Going over the code in pqformat.c this happens because the invariant
> "StringInfo is guaranteed to have a trailing null byte" is not valid (line
> 682 -- 637). I am not sure why this happens.
>
>
> After this the datanodes do something illegal (probably buffer overflow on
> the port..just a guess) that the OS decides to issue a sigkill.  I am also
> not sure why the OS is not logging this killed operation.
>
>
> Please let me know of  suggestions you may have regarding the "invalid
> string in message". In particular, where in the code is StringInfo msg is
> getting populated i.e. which code reads the message over the network and
> keeps it ready for the copy operation on the datanode. Any other suggestion
> would also be helpful to trace this bug.
>
> Best,
> Sandeep
>
>
>
>
>
>
> On Thu, Nov 14, 2013 at 10:25 AM, Masataka Saito <pg...@gm...> wrote:
>>
>> Hi,
>>
>> I think I may be onto something.
>>
>> When the memory is exhausted, the Linux kernel kills process randomly
>> by SIGKILL.
>> The mechanism is called OOM(Out Of Memory) killer.
>> OOM killer logs its activity to kernel mesage buffer.
>>
>> Could you check display message (dmesg) and memory resource?
>>
>> Regards.
>>
>> On Tue, Nov 12, 2013 at 3:55 AM, Sandeep Gupta <gup...@gm...>
>> wrote:
>> > Hi Koichi,
>> >
>> >    It is a bit of mystery  because it does not happen consistently.
>> > Thanks
>> > for the clarification though. It is indeed helpful.
>> >
>> > -Sandeep
>> >
>> >
>> >
>> >
>> > On Mon, Nov 11, 2013 at 1:58 AM, 鈴木 幸市 <ko...@in...> wrote:
>> >>
>> >> Someone is sending SIGKILL to coordinator/datanode backend processes.
>> >> Although XC (originally PG) code has a handler for SIGKILL, I didn’t
>> >> found
>> >> any code in XC sending SIGKILL to other processes.   I’m afraid there
>> >> could
>> >> be another process sending SIGKILL to them.
>> >>
>> >> Best Regards;
>> >> ---
>> >> Koichi Suzuki
>> >>
>> >> 2013/11/09 2:54、Sandeep Gupta <gup...@gm...> のメール：
>> >>
>> >> > I am running single instance of postgres-xc with  several datanode
>> >> > Each data node runs on its own machine.
>> >> > After instantiating the cluster the database sits idle. I do not
>> >> > perform
>> >> > any create table or insert operations. However, after some time all
>> >> > the
>> >> > datanode  automatically shuts down (log messages attached).
>> >> >
>> >> > Any pointers as to why this maybe happening would be very useful.
>> >> >
>> >> > -Sandeep
>> >> >
>> >> > The log files across all datanode contain various error messages.
>> >> > These
>> >> > include:
>> >> >
>> >> >
>> >> > LOG:  statistics collector process (PID 23073) was terminated by
>> >> > signal
>> >> > 9: Killed
>> >> > LOG:  WAL writer process (PID 23071) was terminated by signal 9:
>> >> > Killed
>> >> > LOG:  terminating any other active server processes
>> >> > WARNING:  terminating connection because of crash of another server
>> >> > process
>> >> > DETAIL:  The postmaster has commanded this server process to roll
>> >> > back
>> >> > the current transaction and exit, because another server process
>> >> > exited
>> >> > abnormally and possibly corrupted shared memory.
>> >> > HINT:  In a moment you should be able to reconnect to the database
>> >> > and
>> >> > repeat your command.
>> >> > LOG:  all server processes terminated; reinitializing
>> >> >
>> >> >
>> >> > LOG:  checkpointer process (PID 21406) was terminated by signal 9:
>> >> > Killed
>> >> > LOG:  terminating any other active server processes
>> >> > WARNING:  terminating connection because of crash of another server
>> >> > process
>> >> > DETAIL:  The postmaster has commanded this server process to roll
>> >> > back
>> >> > the current transaction and exit, because another server process
>> >> > exited
>> >> > abnormally and possibly corrupted shared memory.
>> >> > LOG:  statistics collector process (PID 28881) was terminated by
>> >> > signal
>> >> > 9: Killed
>> >> > LOG:  autovacuum launcher process (PID 28880) was terminated by
>> >> > signal
>> >> > 9: Killed
>> >> > LOG:  terminating any other active server processes
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > ------------------------------------------------------------------------------
>> >> > November Webinars for C, C++, Fortran Developers
>> >> > Accelerate application performance with scalable programming models.
>> >> > Explore
>> >> > techniques for threading, error checking, porting, and tuning. Get
>> >> > the
>> >> > most
>> >> > from the latest Intel processors and coprocessors. See abstracts and
>> >> > register
>> >> >
>> >> >
>> >> > http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk_______________________________________________
>> >> > Postgres-xc-general mailing list
>> >> > Pos...@li...
>> >> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>> >>
>> >
>> >
>> >
>> > ------------------------------------------------------------------------------
>> > November Webinars for C, C++, Fortran Developers
>> > Accelerate application performance with scalable programming models.
>> > Explore
>> > techniques for threading, error checking, porting, and tuning. Get the
>> > most
>> > from the latest Intel processors and coprocessors. See abstracts and
>> > register
>> >
>> > http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
>> > _______________________________________________
>> > Postgres-xc-general mailing list
>> > Pos...@li...
>> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>> >
>
>
>
> ------------------------------------------------------------------------------
> DreamFactory - Open Source REST & JSON Services for HTML5 & Native Apps
> OAuth, Users, Roles, SQL, NoSQL, BLOB Storage and External API Access
> Free app hosting. Or install the open source package on any LAMP server.
> Sign up and see examples for AngularJS, jQuery, Sencha Touch and Native!
> http://pubads.g.doubleclick.net/gampad/clk?id=63469471&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>