postgres-xc-developers Mailing List for Postgres-XC

Now that  session level locks implementation is checked in, here are some
notes on implementation of transaction level locks ... (Just archiving my
thoughts here).

Transaction level locks do not have an explicit way to unlock. They are
released once the transaction ends.

Similar to session level locks, we need to propagate the
pg_advisory_xact_lock function to other nodes to make them cluster-aware.
But should we propagate to data nodes or coordinators?

I was earlier of the opinion that we should propagate on data nodes, but
now I feel the propagation can be kept to coordinators because they are
aware of the transactions. I realized that besides datanodes, even the
coordinator keeps a local transaction active until user ends the
transaction.

Is it necessary to  propagate to data nodes also, besides coordinators? I
don't think it is necessary. Because all advisory lock commands have to go
through coordinators. And these locks do not share the same space with
implicit database object locks which are held on datanodes when a table is
referenced in a query.

The LOCK <table> command is implemented by propagating to both coordinator
and data nodes, which is necessary because the lock held on a table using
LOCK command shares the same space with implicit table locks held when a
query accesses tables. So, when a query like "select * from tab1" is
executed in a transaction on data node, it should wait for a lock held on
the table using LOCK tab1. And for that, this explicit lock should be
propagated onto data nodes, otherwise the query won't see this lock.

Will the coordinator-to-coordinator connection pooling have the same
unlocking issues like session level locks? No, because the the transaction
locks are never unlocked explicitly.

Will the coord-to-coord pooler connection keep the lock active even after
the user ends transaction ? No, because when a user transaction ends, the
pooler connection also ends its transaction although it does not end the
connection. When there are two simultaneous transactions from the same
coordinator, the coordinator creates two pooler connections to the remote
coordinator. The advisory locks on the two transactions will be held on
these two pooler connections until the transactions end.

So effectively, there does not look to be much more work for transaction
level locks except that they keep all the coord-coord locks active, they
don't unlock the remote locks unlike session-level locks. Attached patch
does this for a sample function. All synchronization issues considered for
session level locks apply here also.

Will send the complete patch after more testing.


On 30 January 2012 17:25, Amit Khandekar <ami...@en...>wrote:

> On 20 January 2012 17:50, Amit Khandekar
> <ami...@en...> wrote:
> > Before going into implementation issues that I thought of on advisory
> locks
> > , here's a PG doc snippet briefly explaining what are they:
> >
> > "There are two different types of advisory locks in PostgreSQL: session
> > level and transaction level. Once acquired, a session level advisory
> lock is
> > held until explicitly released or the session ends. Unlike standard
> locks,
> > session level advisory locks do not honor transaction semantics: a lock
> > acquired during a transaction that is later rolled back will still be
> held
> > following the rollback, and likewise an unlock is effective even if the
> > calling transaction fails later. The same session level lock can be
> acquired
> > multiple times by its owning process: for each lock request there must
> be a
> > corresponding unlock request before the lock is actually released. (If a
> > session already holds a given lock, additional requests will always
> succeed,
> > even if other sessions are awaiting the lock.) Transaction level locks on
> > the other hand behave more like regular locks; they are automatically
> > released at the end of the transaction, and can not be explicitly
> unlocked."
> >
> >
> > Implementation notes for session level locks.
> > ----------------------------
> >
> > In a nutshell this is how they work :
> >
> > Application 1 calls select pg_advisory_lock(100)
> > Application 2 calls select pg_advisory_lock(100)  and waits for app 1 to
> > call select pg_advisory_unlock(100) or to end the session.
> >
> >
> > Goal is to make sure that when a client from coordinator C1 locks on a
> key,
> > another client from coordinator C2 should wait on the lock. THe
> > advisory_locks calls from the session get stacked, that means the
> resource
> > won't get released until it is unlocked as many times as it has locked.
> So
> > pg_advisory_lock() 3 times should be followed by pg_advisory_unlock() 3
> > times for the resource to be freed for other sessions.
> >
> > One simple way is to just send a parallel 'exec direct $$select
> > pg_advisory_lock()$$ '  on all coordinators.
> >
> > Sequence of actions:
> >
> > 1. Client1 from C1 calls :
> > select pg_advisory_lock(100)
> > THis call will be propogated to all coords (C1 and C2) so there will be
> > locks held on both C1 and C2. C1 lock is native lock, and the lock on C2
> is
> > through C1's pooler connection to C2.
> >
> > 2. Next client2 from C2 calls :
> > select pg_advisory_lock(100)
> > This will again call pg_advisory_lock() on C1 and C2. Both of these calls
> > would wait because client1 has held locks. so effectively the client2
> will
> > wait on this call.
> >
> > 3a. Next client1 calls pg_advisory_unlock().
> >     This will call pg_advisory_unlock() on C1 and C2, unlocking the
> earlier
> > held locks.
> >     THis will in turn make client2 return from its pg_advisory_lock()
> call.
> > OR
> >
> > 3b. Client 1 exits without calling pg_advisory_unlock().
> > This will automatically release the lock held on the native coordinator
> C1.
> > But the one held on C2 will not be released because that lock is held on
> the
> > pooler session to C2. So here we are in trouble. This C2 lock will be
> held
> > there permanently until the pooler exists or someone explicitly calls
> > pg_advisory_unlock, and so client2 waiting on this lock will hang
> > indefintely.
> >
> > To handle this issue with pooler connection, the pg_advisory_lock()
> > definition should immediately unlock all locks except the native
> coordinator
> > lock. But still these lock-unlock calls on remote coordinator are
> necessary
> > because it should get a chance to wait for a resource in case already
> > someone else has held the resource from some other coordinator.
> >
> > pg_advisory_lock ()
> > {
> >     /*
> >      * Go on locking on each coordinator. Keep on unlocking the previous
> one
> >      * each time a new lock is held. Don't unlock the native coordinator.
> >      * After finishing all coordinators, ultimately only the native
> > coordinator
> >      * would be held, but still we will have scanned all coordinators to
> > make
> >      * sure no one else is already grabbed the lock.
> >      */
> >     for (i = 0; i < last_coordinator_index; i++)
> >     {
> >         Call pg_advisory_lock() on coordinator[i];
> >         if (i > 0 && !is_native(coordinator[i-1]) )
> >             Call pg_advisory_unlock() on coordinator[i-1];
> >     }
> >
> >
> > }
> >
> >     Note that the order of locking all coordinators must strictly be
> common
> > to all. If client1 from C1 locks C1 first and then C2, and in parallel
> > client2 from C2 locks C2 first and then C1, there would arise the typical
> > synchronisation issues. For e.g., simultaneously client1 and client2 call
> > pg_advisory_lock() on C1 and C2 respectively. They won't wait because
> they
> > are on different coordinators. NExt, client1 and client2 both would
> > simultaneously try to lock on C2 and C1 resp. , and both would wait on
> each
> > other. This deadlock would not arise if both lock with a common order,
> say
> > C1 then C2.
> >
> > And that's the reason lock function cannot be propogated to all
> coordinators
> > in parallel because that way we cannot gurantee the order of locking.
> >
> >
>
> Based on the implementation notes, attached is a patch that covers
> session level advisory locks. A common function pgxc_advisory_locks()
> is used to implement the common algorithm used by all variants of
> pg_advisory_lock function. The algorithm is based keeping in mind the
> above issue that I had discussed.
>
> Additionally, I had to tweak a bit the existing LockAcquire() function
> in locks.c. When a user from the same session calls advisory lock a
> second time, then it should not propogate the call to all
> coordinators, rather it should just increment the lock reference
> locally. So it was needed to check if the lock already exists on the
> same session, and then increment it, else just return without locking.
> This specific behaviour was not present in existing LockAcquire()
> function, so I made it support the same.
>
> Tweaked pgxc_execute_on_nodes() function that was written for object
> size functions so that it can be reused for advisory lock functions.
>
> I was planning to include both session level and transaction level
> locking functions, but looks like transaction level functions have to
> be handled differently, so I am working on it, and will send it using
> a separate patch.
>
>
> >
> > Implementation notes for transaction  level locks.
> > --------------------------------------------------------
> >
> > Yet to think on this, but for this, I think we need to propogate the
> calls
> > to datanode rather than coordinator. Because datanodes are the ones who
> > handle transactions. These locks cannot be unlocked. They only unlock at
> end
> > of transmission.
>

>
>
> BTW, I also feel execRemote.c is currently placed at a wrong place. I
> can't see why it should be inside "pool" directory.
>
>
That is because of the historical reasons. The execRemote.c was one of the
first XC modules and was closely related to pooled connections.
Later it was integrated with the pooler and should be better placed in
executor. Even better if its content is split on multiple modules: have
functions for executor, copy, transaction handling in different modules.


> Thanks,
> Pavan
>
> --
> Pavan Deolasee
> EnterpriseDB http://www.enterprisedb.com
>



-- 
Best regards,
Andrei Martsinchyk                   mailto:and...@gm...

On Tue, Jan 31, 2012 at 8:32 PM, Andrei Martsinchyk
<and...@gm...> wrote:
> OK, after taking closer look I see the solution is not so simple as I
> initially thought.
> The problem is with other sessions, their view of pgxc_node table should be
> in sync with the pooler. I remember I mentioned this issue long time ago and
> as far as I know it is still here. By executing pgxc_pool_reload() DBA may
> force pooler update from catalog, but other users need to reconnect to have
> their node caches up to date.
> I came to mind that a good solution is a shared memory table where
> coordinator and datanode definitions are stored. An auxiliary process can
> access shared memory without having to read catalog directly; session that
> modify data nodes may update this table immediately, other session may
> access the table instead of having local buffers and therefore node info
> will allways be up to date. Access to shared memory is cheap, comparable to
> cost of access to local memory.
> I am attaching a patch that demonstrate the idea: here is a definition of
> shared memory tables and function that populates tables from catalog info.
> It is just a demo of the approach and won't compile with the current code,
> if you like the idea I can try and complete it.

I think shared memory state is a good compromise. The auxiliary
processes do have access to the shared memory and LWLocks. So we can
synchronize access to the shared state nicely. Also, at the face of
it, it doesn't seem correct for pooler process to have access to the
catalogs etc since its primary responsibility is managing connections
to the remote node and not manipulate anything locally.

BTW, I also feel execRemote.c is currently placed at a wrong place. I
can't see why it should be inside "pool" directory.

Thanks,
Pavan

-- 
Pavan Deolasee
EnterpriseDB http://www.enterprisedb.com