postgres-xc-general Mailing List for Postgres-XC

It's quite possible I'm missing something obvious, but here is how I've modified procarray.c - the idea was to capture the values that were failing to understand why it was failing.

void
ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
{
        ProcArrayStruct *arrayP = procArray;
        int                     index;
        int    _xNumProcs;
        int    _xIndex;
...
        for (index = 0; (_xIndex = index) < (_xNumProcs = arrayP->numProcs); index++)
        {
                if (arrayP->pgprocnos[index] == proc->pgprocno)
                {
                        /* Keep the PGPROC array sorted. See notes above */
                        memmove(&arrayP->pgprocnos[index], &arrayP->pgprocnos[index + 1],
                                        (arrayP->numProcs - index - 1) * sizeof(int));
                        arrayP->pgprocnos[arrayP->numProcs - 1] = -1;           /* for debugging */
                        arrayP->numProcs--;
                        LWLockRelease(ProcArrayLock);
                        return;
                }
        }

        /* Ooops */
        LWLockRelease(ProcArrayLock);
        elog(LOG, "ProcArrayRemove(post-test): %d | %d | %d | %d", _xIndex, _xNumProcs, arrayP->numProcs, _xIndex < _xNumProcs);

        elog(LOG, "failed to find proc %p in ProcArray", proc);
}

With CFLAGS="" this works as expected.  Once I set CFLAGS="-O2" (or anything else similar) it falls apart.  For example, the fall through case triggered and it showed the following ...

ProcArrayRemove(post-test): 1 | 9 | 9 | 1

Which means the loop test should have succeeded.  I could take this one step further and cache the result of the for loop, however, I can tell you from prior experience, _xIndex < _xNumProcs evaluated as FALSE.  Really not sure what the compiler is doing to draw that conclusion from 1 < 9.

Aaron

________________________________
From: Aaron Jackson [aja...@re...]
Sent: Friday, April 25, 2014 3:05 PM
To: 鈴木 幸市
Cc: pos...@li...
Subject: Re: [Postgres-xc-general] failed to find proc - increasing numProcs

CFLAGS="-O2"
gcc version 4.8.1 (Ubuntu/Linaro 4.8.1-10ubuntu9)

The failed evaluation occurs on line 421 of backend/storage/ipc/procarray.c

The test portion of the clause fails.  I'm not entirely sure why gcc specifically fails, but if I were taking an educated guess, it would be that arrayP->numProcs was volatile and the resultant value of the test was optimized and cached.  I've used several techniques (none of which I like) to fool gcc into believing the value is volatile and discarding the value of arrayP->numProcs.  It concerns me more because the ProcArrayLock should be locked during this sequence.

Aaron

________________________________
From: 鈴木 幸市 [ko...@in...]
Sent: Sunday, April 13, 2014 7:55 PM
To: Aaron Jackson
Cc: pos...@li...
Subject: Re: [Postgres-xc-general] failed to find proc - increasing numProcs

Thank you Aaron for the detailed analysis.   As long as the issue is just for XC, we need a fix for it to work correctly regardless the compiler optimization.

Did to locate where such wrong estimation takes place?   And what compilation option did you use?

They are very helpful.

Best;
---
Koichi Suzuki

2014/04/12 11:40、Aaron Jackson <aja...@re...<mailto:aja...@re...>> のメール：

It appears that problem is a compiler optimization issue.  I narrowed the issue down to the loop at the end of the ProcArrayRemove method.  I'm not entirely sure why, but the compiler generated code that evaluates the test block of the loop improperly.  Since changing the compiler options, the problem has been resolved.

Aaron

________________________________
From: Aaron Jackson [aja...@re...<mailto:aja...@re...>]
Sent: Friday, April 11, 2014 1:07 AM
To: pos...@li...<mailto:pos...@li...>
Subject: Re: [Postgres-xc-general] failed to find proc - increasing numProcs

I forgot to mention that if I injected a context switch (sleep(0) did the trick as did an elog statement) during the test in the ProcArrayRemove, that it no longer failed.  Hopefully that will help in understanding the reasons why that may have triggered the ProcArrayRemove to succeed.
------------------------------------------------------------------------------
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration
Continuously Automate Build, Test & Deployment
Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees_______________________________________________
Postgres-xc-general mailing list
Pos...@li...
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

CFLAGS="-O2"
gcc version 4.8.1 (Ubuntu/Linaro 4.8.1-10ubuntu9)

The failed evaluation occurs on line 421 of backend/storage/ipc/procarray.c

The test portion of the clause fails.  I'm not entirely sure why gcc specifically fails, but if I were taking an educated guess, it would be that arrayP->numProcs was volatile and the resultant value of the test was optimized and cached.  I've used several techniques (none of which I like) to fool gcc into believing the value is volatile and discarding the value of arrayP->numProcs.  It concerns me more because the ProcArrayLock should be locked during this sequence.

Aaron

________________________________
From: 鈴木 幸市 [ko...@in...]
Sent: Sunday, April 13, 2014 7:55 PM
To: Aaron Jackson
Cc: pos...@li...
Subject: Re: [Postgres-xc-general] failed to find proc - increasing numProcs

Thank you Aaron for the detailed analysis.   As long as the issue is just for XC, we need a fix for it to work correctly regardless the compiler optimization.

Did to locate where such wrong estimation takes place?   And what compilation option did you use?

They are very helpful.

Best;
---
Koichi Suzuki

2014/04/12 11:40、Aaron Jackson <aja...@re...<mailto:aja...@re...>> のメール：

It appears that problem is a compiler optimization issue.  I narrowed the issue down to the loop at the end of the ProcArrayRemove method.  I'm not entirely sure why, but the compiler generated code that evaluates the test block of the loop improperly.  Since changing the compiler options, the problem has been resolved.

Aaron

________________________________
From: Aaron Jackson [aja...@re...<mailto:aja...@re...>]
Sent: Friday, April 11, 2014 1:07 AM
To: pos...@li...<mailto:pos...@li...>
Subject: Re: [Postgres-xc-general] failed to find proc - increasing numProcs

I forgot to mention that if I injected a context switch (sleep(0) did the trick as did an elog statement) during the test in the ProcArrayRemove, that it no longer failed.  Hopefully that will help in understanding the reasons why that may have triggered the ProcArrayRemove to succeed.
------------------------------------------------------------------------------
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration
Continuously Automate Build, Test & Deployment
Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees_______________________________________________
Postgres-xc-general mailing list
Pos...@li...
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general

I installed pgxc 1.2.1 and I wanted to start gtm and gtm_proxy on the same sever, but gtm_proxy didn't work.
I did it as such:sudo sumkdir /usr/local/pgsql/data_gtmmkdir /usr/local/pgsql/data_gtm_proxychown l /usr/local/pgsql/data_gtmchown l /usr/local/pgsql/data_gtm_proxysu linitgtm -Z gtm -D /usr/local/pgsql/data_gtminitgtm -Z gtm_proxy -D /usr/local/pgsql/data_gtm_proxygtm -D /usr/local/pgsql/data_gtm &gtm_proxy -D /usr/local/pgsql/data_gtm_proxy
On the last step,it returns CST -FATAL:  can not connect to GTMLOCATION:  ConnectGTM, proxy_main.c:3344.I followed it in the gtm_proxy,I found errorno 111 which means Connection refused infunction connectFailureMessagein GTMPQconnectPollin connectGTMStartin PQconnectGTMStartin PQconnectGTMin ConnectGTMin RegisterProxyin BaseInit in main.
My os is ubuntu 12.04 amd64 and I also tested it on centos 6.I also installed pgxc 1.2.1 on both of them. But they all get the same error.I found a mail "http://sourceforge.net/p/postgres-xc/mailman/message/30755193/", we are exactly the same.
I followed and tried each every pages on net, but still can't solve it. Can you please tell me what I can do? Any help here would be really appreciated.
gtm.conf and gtm_proxy.conf come as follow:



gtm.conf:# ----------------------# GTM configuration file# ----------------------## This file must be placed on gtm working directory # specified by -D command line option of gtm or gtm_ctl.  The# configuration file name must be "gtm.conf"### This file consists of lines of the form##  name = value## (The "=" is optional.) Whitespace may be used.   Comments are# introduced with "#" anywhere on a line.  The complete list of# parameter names and allowed values can be found in the# Postgres-XC documentation.## The commented-out settings shown in this file represent the default# values.## Re-commenting a setting is NOT sufficient to revert it to the default# value.## You need to restart the server.
#------------------------------------------------------------------------------# GENERAL PARAMETERS#------------------------------------------------------------------------------nodename = 'one'				# Specifies the node name.					# (changes requires restart)#listen_addresses = '*'			# Listen addresses of this GTM.					# (changes requires restart)port = 6666				# Port number of this GTM.					# (changes requires restart)
#startup = ACT				# Start mode. ACT/STANDBY.
#------------------------------------------------------------------------------# GTM STANDBY PARAMETERS#------------------------------------------------------------------------------#Those parameters are effective when GTM is activated as a standby server#active_host = ''			# Listen address of active GTM.					# (changes requires restart)#active_port =				# Port number of active GTM.					# (changes requires restart)
#---------------------------------------# OTHER OPTIONS#---------------------------------------#keepalives_idle = 0			# Keepalives_idle parameter.#keepalives_interval = 0		# Keepalives_interval parameter.#keepalives_count = 0			# Keepalives_count internal parameter.#log_file = 'gtm.log'			# Log file name#log_min_messages = WARNING		# log_min_messages.  Default WARNING.				  	# Valid value: DEBUG, DEBUG5, DEBUG4, DEBUG3,					# DEBUG2, DEBUG1, INFO, NOTICE, WARNING,					# ERROR, LOG, FATAL, PANIC#synchronous_backup = off	# If backup to standby is synchronous






gtm_proxy.conf:
#-----------------------------# GTM Proxy configuration file#-----------------------------## This file must be placed on gtm working directory # specified by -D command line option of gtm_proxy or gtm_ctl.# The configuration file name must be "gtm_proxy.conf"### This file consists of lines of the form##  name = value## (The "=" is optional.) Whitespace may be used.   Comments are# introduced with "#" anywhere on a line.  The complete list of# parameter names and allowed values can be found in the# Postgres-XC documentation.## The commented-out settings shown in this file represent the default# values.## Re-commenting a setting is NOT sufficient to revert it to the default# value.## You need to restart the server.
#------------------------------------------------------------------------------# GENERAL PARAMETERS#------------------------------------------------------------------------------nodename = 'one'					# Specifies the node name.								# (changes requires restart)#listen_addresses = '*'			# Listen addresses of this GTM.								# (changes requires restart)port = 6666					# Port number of this GTM.								# (changes requires restart)
#------------------------------------------------------------------------------# GTM PROXY PARAMETERS#------------------------------------------------------------------------------#worker_threads = 1				# Number of the worker thread of this								# GTM proxy								# (changes requires restart)
#------------------------------------------------------------------------------# GTM CONNECTION PARAMETERS#------------------------------------------------------------------------------# Those parameters are used to connect to a GTM servergtm_host = 'localhost' 					# Listen address of the active GTM.								# (changes requires restart)gtm_port = 6668 					# Port number of the active GTM.								# (changes requires restart)
#------------------------------------------------------------------------------# Behavior at GTM communication error#------------------------------------------------------------------------------#gtm_connect_retry_interval = 0	# How long (in secs) to wait until the next								# retry to connect to GTM.###------------------------------------------------------------------------------# Other options#------------------------------------------------------------------------------#keepalives_idle = 0			# Keepalives_idle parameter.#keepalives_interval = 0		# Keepalives_interval parameter.#keepalives_count = 0			# Keepalives_count internal parameter.#log_file = 'gtm_proxy.log'		# Log file name#log_min_messages = WARNING		# log_min_messages.  Default WARNING.							  	# Valid value: DEBUG, DEBUG5, DEBUG4, DEBUG3,								# DEBUG2, DEBUG1, INFO, NOTICE, WARNING,								# ERROR, LOG, FATAL, PANIC.


--Ronian

I installed pgxc 1.2.1 and I wanted to start gtm and gtm_proxy on the same
sever, but gtm_proxy didn't work.

I did it as such:
sudo su
mkdir /usr/local/pgsql/data_gtm
mkdir /usr/local/pgsql/data_gtm_proxy
chown l /usr/local/pgsql/data_gtm
chown l /usr/local/pgsql/data_gtm_proxy
su l
initgtm -Z gtm -D /usr/local/pgsql/data_gtm
initgtm -Z gtm_proxy -D /usr/local/pgsql/data_gtm_proxy
gtm -D /usr/local/pgsql/data_gtm &
gtm_proxy -D /usr/local/pgsql/data_gtm_proxy

On the last step,it returns CST -FATAL:  can not connect to GTM
LOCATION:  ConnectGTM, proxy_main.c:3344.
I followed it in the gtm_proxy,I found errorno 111 which means Connection
refused in
function connectFailureMessage
in GTMPQconnectPoll
in connectGTMStart
in PQconnectGTMStart
in PQconnectGTM
in ConnectGTM
in RegisterProxy
in BaseInit
in main.

My os is ubuntu 12.04 amd64 and I also tested it on centos 6.I also
installed pgxc 1.2.1 on both of them. But they all get the same error.I
found a mail "http://sourceforge.net/p/postgres-xc/mailman/message/30755193/",
we are exactly the same.

I followed and tried each every pages on net, but still can't solve it. Can
you please tell me what I can do? Any help here would be really appreciated.

gtm.conf and gtm_proxy.conf come as follow:




gtm.conf:
# ----------------------
# GTM configuration file
# ----------------------
#
# This file must be placed on gtm working directory
# specified by -D command line option of gtm or gtm_ctl.  The
# configuration file name must be "gtm.conf"
#
#
# This file consists of lines of the form
#
#  name = value
#
# (The "=" is optional.) Whitespace may be used.   Comments are
# introduced with "#" anywhere on a line.  The complete list of
# parameter names and allowed values can be found in the
# Postgres-XC documentation.
#
# The commented-out settings shown in this file represent the default
# values.
#
# Re-commenting a setting is NOT sufficient to revert it to the default
# value.
#
# You need to restart the server.

#------------------------------------------------------------------------------
# GENERAL PARAMETERS
#------------------------------------------------------------------------------
nodename = 'one' # Specifies the node name.
# (changes requires restart)
#listen_addresses = '*' # Listen addresses of this GTM.
# (changes requires restart)
port = 6666 # Port number of this GTM.
# (changes requires restart)

#startup = ACT # Start mode. ACT/STANDBY.

#------------------------------------------------------------------------------
# GTM STANDBY PARAMETERS
#------------------------------------------------------------------------------
#Those parameters are effective when GTM is activated as a standby server
#active_host = '' # Listen address of active GTM.
# (changes requires restart)
#active_port = # Port number of active GTM.
# (changes requires restart)

#---------------------------------------
# OTHER OPTIONS
#---------------------------------------
#keepalives_idle = 0 # Keepalives_idle parameter.
#keepalives_interval = 0 # Keepalives_interval parameter.
#keepalives_count = 0 # Keepalives_count internal parameter.
#log_file = 'gtm.log' # Log file name
#log_min_messages = WARNING # log_min_messages.  Default WARNING.
   # Valid value: DEBUG, DEBUG5, DEBUG4, DEBUG3,
# DEBUG2, DEBUG1, INFO, NOTICE, WARNING,
# ERROR, LOG, FATAL, PANIC
#synchronous_backup = off # If backup to standby is synchronous







gtm_proxy.conf:

#-----------------------------
# GTM Proxy configuration file
#-----------------------------
#
# This file must be placed on gtm working directory
# specified by -D command line option of gtm_proxy or gtm_ctl.
# The configuration file name must be "gtm_proxy.conf"
#
#
# This file consists of lines of the form
#
#  name = value
#
# (The "=" is optional.) Whitespace may be used.   Comments are
# introduced with "#" anywhere on a line.  The complete list of
# parameter names and allowed values can be found in the
# Postgres-XC documentation.
#
# The commented-out settings shown in this file represent the default
# values.
#
# Re-commenting a setting is NOT sufficient to revert it to the default
# value.
#
# You need to restart the server.

#------------------------------------------------------------------------------
# GENERAL PARAMETERS
#------------------------------------------------------------------------------
nodename = 'one' # Specifies the node name.
# (changes requires restart)
#listen_addresses = '*' # Listen addresses of this GTM.
# (changes requires restart)
port = 6666 # Port number of this GTM.
# (changes requires restart)

#------------------------------------------------------------------------------
# GTM PROXY PARAMETERS
#------------------------------------------------------------------------------
#worker_threads = 1 # Number of the worker thread of this
# GTM proxy
# (changes requires restart)

#------------------------------------------------------------------------------
# GTM CONNECTION PARAMETERS
#------------------------------------------------------------------------------
# Those parameters are used to connect to a GTM server
gtm_host = 'localhost' # Listen address of the active GTM.
# (changes requires restart)
gtm_port = 6668 # Port number of the active GTM.
# (changes requires restart)

#------------------------------------------------------------------------------
# Behavior at GTM communication error
#------------------------------------------------------------------------------
#gtm_connect_retry_interval = 0 # How long (in secs) to wait until the next
# retry to connect to GTM.
#
#
#------------------------------------------------------------------------------
# Other options
#------------------------------------------------------------------------------
#keepalives_idle = 0 # Keepalives_idle parameter.
#keepalives_interval = 0 # Keepalives_interval parameter.
#keepalives_count = 0 # Keepalives_count internal parameter.
#log_file = 'gtm_proxy.log' # Log file name
#log_min_messages = WARNING # log_min_messages.  Default WARNING.
   # Valid value: DEBUG, DEBUG5, DEBUG4, DEBUG3,
# DEBUG2, DEBUG1, INFO, NOTICE, WARNING,
# ERROR, LOG, FATAL, PANIC.



--Ronian

Did you backup the file you cleared?

if not, it was not a good idea to clean things because GTM writes its restart point at its working directory.   If you did, then you can restore the file gtm.control back.   It is in fact a text file so if you would like, you can edit it for repair.  First like is the next GXID value.    Following lines define each sequence.   This could be a bit harmful but should work.   If you ran gtm slave, it will also have gtm.control which can be used as well.

Good luck.
---
Koichi Suzuki

2014/04/25 2:20、Aaron Jackson <aja...@re...<mailto:aja...@re...>> のメール：

So I noticed a problem creating a sequence on my cluster - the logs indicated that there was a gtm failure.  The GTM indicated that the sequence existed.  So I stopped the GTM but left the two coordinators (and proxies) running.  I then wiped out everything exception the configuration file and restarted.  Soon after that, I noticed that the schemas I had created were no longer available through the coordinator - through attempts to create them said they existed.  The best I can guess is that my reset caused the transaction #s to get munged and the schemas were not visible at the mvcc level I was currently querying.  Long story short, I did this one to myself - however, in a real scenario, how would you go about resetting the environment if the GTM suffered a failure and needed to be rebuilt?
------------------------------------------------------------------------------
Start Your Social Network Today - Download eXo Platform
Build your Enterprise Intranet with eXo Platform Software
Java Based Open Source Intranet - Social, Extensible, Cloud Ready
Get Started Now And Turn Your Intranet Into A Collaboration Platform
http://p.sf.net/sfu/ExoPlatform_______________________________________________
Postgres-xc-general mailing list
Pos...@li...
https://lists.sourceforge.net/lists/listinfo/postgres-xc-general