[Postgres-xc-general] failed to find proc - increasing numProcs

During my initialization process, I make several calls to psql in a row.  What I notice is that if I call psql often enough, I cause the coordinator to exhaust the # of clients in can support.  It shows up in the coordinator log as "PANIC: sorry, too many clients already"

I took a deeper look into what was happening and I believe, the problem is that during one of the invocations to psql, the prepared procedures allocate themselves in the procarray improperly.  ProcArrayRemove then fails to remove the procedure and the number of processes stays one client too high artificially.  Once this happens enough, the clients in the procarray are exhausted and the panic occurs.

The initial startup of the server allocated a total of 15 slots in the procarray - with one being released leaving an initial allocation of 14 processes in the array.

Process 31098 (a coordinator) starts in response to an invocation of psql.  It's assigned proc 0x7ff2ebe96c00 and adds that to the array via InitProcessPhase2.  A prepared transaction is created and proc 0x7ff2ebe99ce0 is subsequently added to the proc array.  I watched very carefully and did not see anywhere that process 31098 did anything to clear that proc from the proc array.  ProcArrayRemove is called and does *not* find 0x7ff2ebe99ce0.  This now leaves numProcs too large by +1 - eventually, we run out of slots.  Oddly enough, right after the failure, we called MarkAsPrepared again with the same proc.  Again it failed and we are +2.  The process then terminated making me believe it corresponded to the "DROP TABLE" and "CREATE TABLE" that I had in one of my initialization files.

I'm clearly not familiar with how GlobalTransactions are associated with a Proc so I will continue to research this.  Is it possible for one of the other participants to have removed the Proc from the ProcArray?

Aaron