postgres-xc-general Mailing List for Postgres-XC

Sorry for the late reply

here is the first 2 lines of gtm.control file

28875757
opensips.public.aliases_id_seq\00       34160   1       1       1
9223372036854775807     f       t       1




On Fri, May 2, 2014 at 1:12 PM, 鈴木 幸市 <ko...@in...> wrote:

>  The size of the directory is not the matter.  The file 000B cannot be
> extended in size.   I hope you have plenty of space available in the file
> system.   You can get this by df command.  If there are sufficient space
> available, it must not be an issue.
>
>  I’m curious why the transaction is writing to quite old clog 000B.
>  I’m afraid, as suggested, gtm.control is corrupted.   Could you share the
> first line of gtm.control which suggests restarting GXID value?
>
>  Regards;
> ---
> Koichi Suzuki
>
>  2014/05/02 15:07、Juned Khan <jkh...@gm...> のメール：
>
>  astpp=# select count(*) from accounts;
> ERROR:  could not access status of transaction 0
> DETAIL:  Could not write to file "pg_clog/000B" at offset 40960: No space
> left on device.
>
> postgres@db02:~/pgxc/nodes/dn_master/pg_clog$ ls
> 0000  0001  0002  0003  0004  0005  0006  0007  0008  0009  000A  000B
> 000C  000D  000E  000F  0010  0011  0012  0013  0014  0015  0016  0017
> 0018  0019  001A
>
>  can i remove this file, although size of this directory not to big its
> just 6.8M
>
>
> On Fri, May 2, 2014 at 10:36 AM, Juned Khan <jkh...@gm...> wrote:
>
>>  Actually i haven setup anything related to checkpoint_segments manually.
>>
>>
>>
>> On Fri, May 2, 2014 at 10:31 AM, Michael Paquier <
>> mic...@gm...> wrote:
>>
>>> On Fri, May 2, 2014 at 11:46 AM, Masataka Saito <pg...@gm...>
>>> wrote:
>>> > Michael suggested followings.
>>> >  * The partition for XC is full.
>>> >  * df command will help you. (df is basic command enough to assume all
>>> > of Linux engineers knows it)
>>>  And reducing checkpoint_segments would help. I am fairly guessing that
>>> Juned set it up to an utterly high value to avoid any checkpoints to
>>> be kicked by server during his benchmark tests to avoid I/O spikes
>>> caused by dirty page flushes.
>>> --
>>> Michael
>>>
>>
>>
>>
>>  --
>> Thanks,
>> Juned Khan
>> iNextrix Technologies Pvt Ltd.
>> www.inextrix.com
>>
>
>
>
> --
> Thanks,
> Juned Khan
> iNextrix Technologies Pvt Ltd.
> www.inextrix.com
> ------------------------------------------------------------------------------
> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> Instantly run your Selenium tests across 300+ browser/OS combos.  Get
> unparalleled scalability from the best Selenium testing platform available.
> Simple to use. Nothing to install. Get started now for free."
>
> http://p.sf.net/sfu/SauceLabs_______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
>
>


-- 
Thanks,
Juned Khan
iNextrix Technologies Pvt Ltd.
www.inextrix.com

Hello;

Because there are many discussions here on Postgres-XC configuration
and operation, I drafted a paper "Pgcx_ctl primer" to provide how to
configure Postgres-XC cluster and how to operate it.

https://sourceforge.net/projects/postgres-xc/files/Pgxc_ctl_primer/?
contains the document.

Hope to have feedback on it.

Regards;
---
Koichi Suzuki

The size of the directory is not the matter.  The file 000B cannot be extended in size.   I hope you have plenty of space available in the file system.   You can get this by df command.  If there are sufficient space available, it must not be an issue.

I’m curious why the transaction is writing to quite old clog 000B.    I’m afraid, as suggested, gtm.control is corrupted.   Could you share the first line of gtm.control which suggests restarting GXID value?

Regards;
---
Koichi Suzuki

2014/05/02 15:07、Juned Khan <jkh...@gm...<mailto:jkh...@gm...>> のメール：

astpp=# select count(*) from accounts;
ERROR:  could not access status of transaction 0
DETAIL:  Could not write to file "pg_clog/000B" at offset 40960: No space left on device.

postgres@db02:~/pgxc/nodes/dn_master/pg_clog$ ls
0000  0001  0002  0003  0004  0005  0006  0007  0008  0009  000A  000B  000C  000D  000E  000F  0010  0011  0012  0013  0014  0015  0016  0017  0018  0019  001A

can i remove this file, although size of this directory not to big its just 6.8M


On Fri, May 2, 2014 at 10:36 AM, Juned Khan <jkh...@gm...<mailto:jkh...@gm...>> wrote:
Actually i haven setup anything related to checkpoint_segments manually.



On Fri, May 2, 2014 at 10:31 AM, Michael Paquier <mic...@gm...<mailto:mic...@gm...>> wrote:
On Fri, May 2, 2014 at 11:46 AM, Masataka Saito <pg...@gm...<mailto:pg...@gm...>> wrote:
> Michael suggested followings.
>  * The partition for XC is full.
>  * df command will help you. (df is basic command enough to assume all
> of Linux engineers knows it)
And reducing checkpoint_segments would help. I am fairly guessing that
Juned set it up to an utterly high value to avoid any checkpoints to
be kicked by server during his benchmark tests to avoid I/O spikes
caused by dirty page flushes.
--
Michael



--
Thanks,
Juned Khan
iNextrix Technologies Pvt Ltd.
www.inextrix.com<http://www.inextrix.com/>



--
Thanks,
Juned Khan
iNextrix Technologies Pvt Ltd.
www.inextrix.com<http://www.inextrix.com/>
------------------------------------------------------------------------------
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.  Get
unparalleled scalability from the best Selenium testing platform available.
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs_______________________________________________
Postgres-xc-general mailing list
Pos...@li...
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

astpp=# select count(*) from accounts;
ERROR:  could not access status of transaction 0
DETAIL:  Could not write to file "pg_clog/000B" at offset 40960: No space
left on device.

postgres@db02:~/pgxc/nodes/dn_master/pg_clog$ ls
0000  0001  0002  0003  0004  0005  0006  0007  0008  0009  000A  000B
000C  000D  000E  000F  0010  0011  0012  0013  0014  0015  0016  0017
0018  0019  001A

can i remove this file, although size of this directory not to big its just
6.8M


On Fri, May 2, 2014 at 10:36 AM, Juned Khan <jkh...@gm...> wrote:

> Actually i haven setup anything related to checkpoint_segments manually.
>
>
>
> On Fri, May 2, 2014 at 10:31 AM, Michael Paquier <
> mic...@gm...> wrote:
>
>> On Fri, May 2, 2014 at 11:46 AM, Masataka Saito <pg...@gm...> wrote:
>> > Michael suggested followings.
>> >  * The partition for XC is full.
>> >  * df command will help you. (df is basic command enough to assume all
>> > of Linux engineers knows it)
>> And reducing checkpoint_segments would help. I am fairly guessing that
>> Juned set it up to an utterly high value to avoid any checkpoints to
>> be kicked by server during his benchmark tests to avoid I/O spikes
>> caused by dirty page flushes.
>> --
>> Michael
>>
>
>
>
> --
> Thanks,
> Juned Khan
> iNextrix Technologies Pvt Ltd.
> www.inextrix.com
>



-- 
Thanks,
Juned Khan
iNextrix Technologies Pvt Ltd.
www.inextrix.com

Actually i haven setup anything related to checkpoint_segments manually.



On Fri, May 2, 2014 at 10:31 AM, Michael Paquier
<mic...@gm...>wrote:

> On Fri, May 2, 2014 at 11:46 AM, Masataka Saito <pg...@gm...> wrote:
> > Michael suggested followings.
> >  * The partition for XC is full.
> >  * df command will help you. (df is basic command enough to assume all
> > of Linux engineers knows it)
> And reducing checkpoint_segments would help. I am fairly guessing that
> Juned set it up to an utterly high value to avoid any checkpoints to
> be kicked by server during his benchmark tests to avoid I/O spikes
> caused by dirty page flushes.
> --
> Michael
>



-- 
Thanks,
Juned Khan
iNextrix Technologies Pvt Ltd.
www.inextrix.com

I know df command but i am not sure about what to remove from which
directory of postgre-xc
as i mentioned in earlier reply , the size of pg_xlog directory is 33G and
yeah still 123G space is available
on DB server.


On Fri, May 2, 2014 at 8:16 AM, Masataka Saito <pg...@gm...> wrote:

> I'm almost sure this issue caused by shot of the storage.
>
> Michael suggested followings.
>  * The partition for XC is full.
>  * df command will help you. (df is basic command enough to assume all
> of Linux engineers knows it)
>
> You could do many things by that suggestion.
> Did you run df command at every server? checked your free area?
> checked quota settings? determine if you can write some data into the
> directory?
>
> And usage of GTM proxy is unusual. I guess that the proxy produced
> enormous error logs about communication with the GTM. As GTM is not
> working well, the log may be no use and you can remove them.
>
> After that you need to recover gtm.control from GTM slave. If slave's
> one is also broken, I think it is difficult to recover the cluster.
>
> Regards.
>
>
> On 1 May 2014 19:02, Juned Khan <jkh...@gm...> wrote:
> > I am not sure but may be datanode_archlog directory is not exists in my
> > case. may be its have different name.
> > however size of /home/postgres/pgxc/nodes/dn_master directory is 42G and
> > /home/postgres/pgxc/nodes/gtm_pxy is of 31G.
> >
> > Please suggest.
> >
> >
> > On Thu, May 1, 2014 at 12:50 PM, Michael Paquier <
> mic...@gm...>
> > wrote:
> >>
> >> On Mon, Apr 28, 2014 at 8:31 PM, Juned Khan <jkh...@gm...>
> wrote:
> >> > "/home/postgres/pgxc/nodes/datanode_archlog/000000010000001C0000006A":
> >> > No
> >> > space left on device (28)
> >> This means that the partition containing folder
> >> /home/postgres/pgxc/nodes/datanode_archlog/ is full. Try to do some df
> >> and monitor the size of this partition/disk that you are using for
> >> your archive files.
> >> --
> >> Michael
> >
> >
> >
> >
> > --
> > Thanks,
> > Juned Khan
> > iNextrix Technologies Pvt Ltd.
> > www.inextrix.com
> >
> >
> ------------------------------------------------------------------------------
> > "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> > Instantly run your Selenium tests across 300+ browser/OS combos.  Get
> > unparalleled scalability from the best Selenium testing platform
> available.
> > Simple to use. Nothing to install. Get started now for free."
> > http://p.sf.net/sfu/SauceLabs
> > _______________________________________________
> > Postgres-xc-general mailing list
> > Pos...@li...
> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> >
>



-- 
Thanks,
Juned Khan
iNextrix Technologies Pvt Ltd.
www.inextrix.com

On Fri, May 2, 2014 at 11:46 AM, Masataka Saito <pg...@gm...> wrote:
> Michael suggested followings.
>  * The partition for XC is full.
>  * df command will help you. (df is basic command enough to assume all
> of Linux engineers knows it)
And reducing checkpoint_segments would help. I am fairly guessing that
Juned set it up to an utterly high value to avoid any checkpoints to
be kicked by server during his benchmark tests to avoid I/O spikes
caused by dirty page flushes.
-- 
Michael

I'm almost sure this issue caused by shot of the storage.

Michael suggested followings.
 * The partition for XC is full.
 * df command will help you. (df is basic command enough to assume all
of Linux engineers knows it)

You could do many things by that suggestion.
Did you run df command at every server? checked your free area?
checked quota settings? determine if you can write some data into the
directory?

And usage of GTM proxy is unusual. I guess that the proxy produced
enormous error logs about communication with the GTM. As GTM is not
working well, the log may be no use and you can remove them.

After that you need to recover gtm.control from GTM slave. If slave's
one is also broken, I think it is difficult to recover the cluster.

Regards.


On 1 May 2014 19:02, Juned Khan <jkh...@gm...> wrote:
> I am not sure but may be datanode_archlog directory is not exists in my
> case. may be its have different name.
> however size of /home/postgres/pgxc/nodes/dn_master directory is 42G and
> /home/postgres/pgxc/nodes/gtm_pxy is of 31G.
>
> Please suggest.
>
>
> On Thu, May 1, 2014 at 12:50 PM, Michael Paquier <mic...@gm...>
> wrote:
>>
>> On Mon, Apr 28, 2014 at 8:31 PM, Juned Khan <jkh...@gm...> wrote:
>> > "/home/postgres/pgxc/nodes/datanode_archlog/000000010000001C0000006A":
>> > No
>> > space left on device (28)
>> This means that the partition containing folder
>> /home/postgres/pgxc/nodes/datanode_archlog/ is full. Try to do some df
>> and monitor the size of this partition/disk that you are using for
>> your archive files.
>> --
>> Michael
>
>
>
>
> --
> Thanks,
> Juned Khan
> iNextrix Technologies Pvt Ltd.
> www.inextrix.com
>
> ------------------------------------------------------------------------------
> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> Instantly run your Selenium tests across 300+ browser/OS combos.  Get
> unparalleled scalability from the best Selenium testing platform available.
> Simple to use. Nothing to install. Get started now for free."
> http://p.sf.net/sfu/SauceLabs
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>

On Fri, May 2, 2014 at 10:01 AM, 鈴木 幸市 <ko...@in...> wrote:
> If you’re quite sure that the node does not fail until removed xlog is not used at the recovery.
Juned meant on master AFAIK.

Per se:
On Thu, May 1, 2014 at 7:45 PM, Juned Khan <jkh...@gm...> wrote:
> what will be the impact if i remove files under pg_xlog directory ?
>
> root@db02:/home/postgres/pgxc/
nodes/dn_master/pg_xlog# du -sh
> 33G     .
-- 
Michael

If you’re quite sure that the node does not fail until removed xlog is not used at the recovery.
---
Koichi Suzuki

2014/05/01 20:23、Michael Paquier <mic...@gm...> のメール：

> On Thu, May 1, 2014 at 7:49 PM, 鈴木 幸市 <ko...@in...> wrote:
>> Do you mean pg_xlog at the slave?   Because pg_xlog contains all the redo
>> logs needed to recovery, it is quite harmful to do so.   If you are removing
>> pg_xlog at the slave, it is also harmful too.   Pgxc_ctl configures the
>> slave to remove unnecessary files under pg_xlog automatically.
> Not harmful. Catastrophic enough to be haunted for 3 generations by
> your users if you do it.
> -- 
> Michael
>