Menu
▾
▴
#7 Postgres-XC crashes with failed to find proc 0x7fb0cb34f7c0 in ProcArray and sorry, too many clients already
Hello,
I am really devastated. For the last week, week and half, I was trying to ran successfully almost all postgres-xc versions 1.1, 1.1.1, 1.2, 1.2.1...etc and the only one which runs without any problems was 1.0.4.
My setup is based on 4 AWS instances with Ubuntu 14.04 LTS on them.
2xData-Nodes 1xCordinator 1xGTM server. While I was strugling with it I tried many approaches like DL the code and compile it, install ubuntu package and so on.
Here is my problem and steps I do.
1. DL the code from sourceforge.net
2. unpak and do ./configure, make and make install
3. Initdb on all related hosts:
Cordinator
/usr/local/pgsql/bin/initdb -D /var/lib/postgres-xc/data --nodename cn1
DataNode1
/usr/local/pgsql/bin/initdb -D /var/lib/postgres-xc/data --nodename dn1
DataNode2
/usr/local/pgsql/bin/initdb -D /var/lib/postgres-xc/data --nodename dn2
GTM server
/usr/local/pgsql/bin/initgtm -Z gtm -D /var/lib/postgres-xc/data
-
Setting up the IP address of the GTM server on all nodes and start them with:
/usr/local/pgsql/bin/pg_ctl start -D /var/lib/postgres-xc/data -Z datanode -l /var/lib/postgres-xc/data/logfile
/usr/local/pgsql/bin/pg_ctl start -D /var/lib/postgres-xc/data -Z coordinator -l /var/lib/postgres-xc/data/logfile
/usr/local/pgsql/bin/gtm_ctl -Z gtm -D /var/lib/postgres-xc/data start -
Adding nodes into cordinator:
postgres=# CREATE NODE dn1 WITH (HOST = '172.30.0.14', type = 'datanode', PORT = 5432);
postgres=# CREATE NODE dn2 WITH (HOST = '172.30.0.15', type = 'datanode', PORT = 5432);
postgres=# SELECT pgxc_pool_reload();
When I issue "createdb -h 172.30.0.11(cordinatorIP) -Upostgres-xc pgbench" from 5th node, it creates the DB on all nodes. Then I populate the DB with "pgbench -h 172.30.0.11 -i -s 10 -Upostgres-xc pgbench" and when i try to run any tests the whole setup crashes with:
starting vacuum...end.
starting vacuum pgbench_accounts...end.
Client 0 aborted in state 13: ERROR: failed to find GlobalTransaction for xid 10135
WARNING: unexpected EOF on datanode connection
Client 1 aborted in state 13: ERROR: sorry, too many clients already
transaction type: TPC-B (sort of)
scaling factor: 10
query mode: simple
number of clients: 2
number of threads: 2
duration: 3600 s
number of transactions actually processed: 126
tps = 42.505160 (including connections establishing)
tps = 42.612909 (excluding connections establishing)
On the coordinator node I am observing the folloging errors into log file:
LOG: failed to find proc 0x7fb0cb3f2480 in ProcArray
STATEMENT: drop table if exists pgbench_history
LOG: failed to find proc 0x7fb0cb3f2480 in ProcArray
STATEMENT: create table pgbench_history(tid int,bid int,aid int,delta int,mtime timestamp,filler char(22))
LOG: failed to find proc 0x7fb0cb3f2480 in ProcArray
STATEMENT: drop table if exists pgbench_tellers
LOG: failed to find proc 0x7fb0cb3f2480 in ProcArray
STATEMENT: create table pgbench_tellers(tid int not null,bid int,tbalance int,filler char(84)) with (fillfactor=100)
LOG: failed to find proc 0x7fb0cb3f2480 in ProcArray
STATEMENT: drop table if exists pgbench_accounts
LOG: failed to find proc 0x7fb0cb3f2480 in ProcArray
STATEMENT: create table pgbench_accounts(aid int not null,bid int,abalance int,filler char(84)) with (fillfactor=100)
LOG: failed to find proc 0x7fb0cb3f2480 in ProcArray
STATEMENT: drop table if exists pgbench_branches
LOG: failed to find proc 0x7fb0cb3f2480 in ProcArray
STATEMENT: create table pgbench_branches(bid int not null,bbalance int,filler char(88)) with (fillfactor=100)
LOG: failed to find proc 0x7fb0cb3f2480 in ProcArray
STATEMENT: commit
LOG: failed to find proc 0x7fb0cb3f2480 in ProcArray
STATEMENT: alter table pgbench_branches add primary key (bid)
LOG: failed to find proc 0x7fb0cb3f2480 in ProcArray
STATEMENT: alter table pgbench_tellers add primary key (tid)
LOG: failed to find proc 0x7fb0cb3f2480 in ProcArray
STATEMENT: alter table pgbench_accounts add primary key (aid)
LOG: failed to find proc 0x7fb0cb3f2480 in ProcArray
STATEMENT: truncate pgbench_history
ERROR: failed to find GlobalTransaction for xid 11378
STATEMENT: END;
WARNING: unexpected EOF on datanode connection
ERROR: sorry, too many clients already
STATEMENT: END;
WARNING: unexpected EOF on datanode connection
LOG: Failed to ABORT at node 16384
Detail: unexpected EOF on datanode connection
LOG: Failed to ABORT an implicitly PREPARED transaction status - 7
LOG: could not send data to client: Broken pipe
FATAL: connection to client lost
LOG: failed to find proc 0x7fb0cb34f7c0 in ProcArray
LOG: failed to find proc 0x7fb0cb34f7c0 in ProcArray
LOG: failed to find proc 0x7fb0cb34f7c0 in ProcArray
LOG: failed to find proc 0x7fb0cb34f7c0 in ProcArray
LOG: failed to find proc 0x7fb0cb34f7c0 in ProcArray**
On the data-nodes:
LOG: failed to find proc 0x7f379dee4a00 in ProcArray
LOG: failed to find proc 0x7f379dee4a00 in ProcArray
LOG: failed to find proc 0x7f379dee4a00 in ProcArray
and some of the transactons are still uncommited on some of the nodes and if I want to connect to other node ot throws the error "psql: FATAL: sorry, too many clients already"
Please any ideas, clue help....etc.
Thanks
Sorry, took too long.
This is caused by a pooler bug to initialize detached connection.
This may not happen when you restart the cluster.
I know the fix but the code was applied to XL, whose license is
different from XC. This is an issue.
Thank you ;
Koichi Suzuki
2014-10-21 20:05 GMT+09:00 Mladen Stefanov mlade77@users.sf.net:
Related
Support Requests: #7
Dear Koichi,
I was hit by this bug too. Do you plan to apply the patch anytime soon?
According to wikipedia, the licensing should not be an issue here as XL uses MPL version 2.0:
http://en.wikipedia.org/wiki/Mozilla_Public_License
"In the absence of patents, MPL-licensed code can be freely used, altered, and redistributed. Versions with patented code can still be used, transferred, and even sold, but cannot be altered without special permission."
happily, it's not patented and I would take word 'use' as permission to apply it as a patch elsewhere as it is constrained to the 'component'.
But as usual with licensing it is confusing. Is there any chance to get a clear statement from XC and XL teams whether patches to already existing code in XC are moot if it comes to re-applying from MPL to BSD or there is chance for a mutual agreement to speed up development of both projects?
Cheers,
Chris.
Thanks Yazun for the info. Yes, I have good connection with XL
people. Because XL is based on XC and I'm participating XL project
as well, I hope there's a good chance to make this clear. Only one
point is if it is okay to bring their license code and use this as a
part of PostgerSQL license product.
Regards;
Koichi Suzuki
2014-11-06 1:45 GMT+09:00 yazun yazuna@users.sf.net:
Related
Support Requests: #7
On Wed, Nov 5, 2014 at 8:45 AM, yazun yazuna@users.sf.net wrote:
I am not a lawyer, but I think theoretically, if you include MPL code that
file can no longer be taken and made closed source in any other software,
it must stay open. (You can still allow other files to be closed source.)
Anyway, which Postgres-XL commit has the bug fix? If written by myself,
Pavan Deolasee, Nikhil Sontakke, or Andrei Martsinchyk, please feel free to
apply it to XC for this particular issue.
Developers on the XL team (we previously worked on XC) had been submitting
important patches to XC as well, but not so much recently. If there are
other bug fixes you need, please us me know.
But as usual with licensing it is confusing. Is there any chance to get a
Regards,
Mason
Cheers,
Related
Support Requests: #7
attachment patch should fix this bug.
Sorry for the late patch application.
Please let me find the earliest chance to commit the patch. I’m now out and have some problem in the mail.
Best;
—
Koichi Suzuki
Related
Support Requests: #7
Sorry for the late patch application.
Please let me find the earliest chance to commit the patch. I’m now out and have some problem in the mail.
Best;
—
Koichi Suzuki
Related
Support Requests: #7