postgres-xc-developers Mailing List for Postgres-XC

Hello All,
    We were doing some heavy volume testing by deleting and inserting blocks of rows (using general delete and insert statements). We noticed that the number of open sessions kept climbing, even though our client programs were ended, and eventually reached around 1020 or so. At this point, we are no longer able to issue our delete insert statements. We suspected that we reached some kind of limit in PGXC, and that it was not reporting an error but simply waiting. We have thoroughly checked our clients to make sure there are no leaky connections. In fact, these are just plain java programs that do a delete/insert of a day's data and then end.

    An strace of the application client program usually show a FUTEX_WAIT.

    An strace of a typical session shows that it is doing a " recvfrom(10, "

    We tried issuing a 'CLEAN CONNECTION TO ALL FORCE FOR DATABASE ADCHEMY", but it was hanging also. The strace showed that it was doing a " recvmsg(11 ".

    We are able to restart the cluster using pg_ctl 'immediate' option, but not the 'fast' option.

    Our configuration has the following:

    32 datanodes
               max connections  1000
               max pool size      1000
    8   coordinators
               max connections  100
               max pool size       4000

     Could one of you please tell me where we should start looking next? It is evident that the number of 'sessions' is not going over a 1000 or so. Thats where things start hanging.


________________________________________

Venky Kandaswamy

Principal Engineer, Adchemy Inc.

925-200-7124

Thanks Abbas for the wonderful work.
----------
Koichi Suzuki


2013/2/19 Abbas Butt <abb...@en...>:
> Hi,
> Attached please find a patch that locks the cluster so that dump can be
> taken to be restored on the new node to be added.
>
> To lock the cluster the patch adds a new GUC parameter called
> xc_lock_for_backup, however its status is maintained by the pooler. The
> reason is that the default behavior of XC is to release connections as soon
> as a command is done and it uses PersistentConnections GUC to control the
> behavior. We in this case however need a status that is independent of the
> setting of PersistentConnections.
>
> Assume we have two coordinator cluster, the patch provides this behavior:
>
> Case 1: set and show
> ====================
> psql test -p 5432
> set xc_lock_for_backup=yes;
> show xc_lock_for_backup;
>  xc_lock_for_backup
> --------------------
>  yes
> (1 row)
>
> Case 2: set from one client show from other
> ==================================
> psql test -p 5432
> set xc_lock_for_backup=yes;
> (From another tab)
> psql test -p 5432
> show xc_lock_for_backup;
>  xc_lock_for_backup
> --------------------
>  yes
> (1 row)
>
> Case 3: set from one, quit it, run again and show
> ======================================
> psql test -p 5432
> set xc_lock_for_backup=yes;
> \q
> psql test -p 5432
> show xc_lock_for_backup;
>  xc_lock_for_backup
> --------------------
>  yes
> (1 row)
>
> Case 4: set on one coordinator, show from other
> =====================================
> psql test -p 5432
> set xc_lock_for_backup=yes;
> (From another tab)
> psql test -p 5433
> show xc_lock_for_backup;
>  xc_lock_for_backup
> --------------------
>  yes
> (1 row)
>
> pg_dump and pg_dumpall seem to work fine after locking the cluster for
> backup but I would test these utilities in detail next.
>
> Also I have yet to look in detail that standard_ProcessUtility is the only
> place that updates the portion of catalog that is dumped. There may be some
> other places too that need to be blocked for catalog updates.
>
> The patch adds no extra warnings and regression shows no extra failure.
>
> Comments are welcome.
>
> --
> Abbas
> Architect
> EnterpriseDB Corporation
> The Enterprise PostgreSQL Company
>
> Phone: 92-334-5100153
>
> Website: www.enterprisedb.com
> EnterpriseDB Blog: http://blogs.enterprisedb.com/
> Follow us on Twitter: http://www.twitter.com/enterprisedb
>
> This e-mail message (and any attachment) is intended for the use of
> the individual or entity to whom it is addressed. This message
> contains information from EnterpriseDB Corporation that may be
> privileged, confidential, or exempt from disclosure under applicable
> law. If you are not the intended recipient or authorized to receive
> this for the intended recipient, any use, dissemination, distribution,
> retention, archiving, or copying of this communication is strictly
> prohibited. If you have received this e-mail in error, please notify
> the sender immediately by reply e-mail and delete this message.
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_feb
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>

Changing the macro into a function does not solve the issue.   It does
not eliminate the need to acquire the lock.   Adding comment to the
macro definition is reasonable compromise.

Regards;
----------
Koichi Suzuki


2013/2/22 Michael Paquier <mic...@gm...>:
> Thanks. Applied.
>
> On Fri, Feb 22, 2013 at 12:41 AM, Nikhil Sontakke <ni...@st...>
> wrote:
>>
>> Or we can even remove this macro to be on the safe side.
>
> Not sure, it looks pretty useful to do that as a macro for transaction count
> control.
> --
> Michael
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_feb
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>