Comparing changes

This set of patches aims to support the detection of update_deleted conflicts, which occur when the apply worker cannot find the target tuple to be updated (e.g., the tuple has been removed by a different origin). To detect this conflict consistently and correctly, we must ensure that tuples deleted by other origins are not prematurely removed by VACUUM before conflict detection. If these tuples are removed too soon, a different conflict might arise and be resolved incorrectly, causing data inconsistency between nodes. To achieve this, we will retain the dead tuples on the subscriber for some period. The concept is that dead tuples are useful for detecting conflicts only during the application of concurrent transactions from remote nodes. After applying and flushing all remote transactions that occurred concurrently with the tuple DELETE, any subsequent UPDATE from a remote node should have a later timestamp. In such cases, it is acceptable to detect an update_missing scenario and convert the UPDATE to an INSERT when applying it. But, for concurrent remote transactions with earlier timestamps than the DELETE, detecting update_deleted is necessary, as the UPDATEs in remote transactions should be ignored if their timestamp is earlier than that of the dead tuples. We assume that the appropriate resolution for update_deleted conflicts, to achieve eventual consistency, is the last-update-win strategy. This means that when detecting the update_deleted conflict, and the remote update has a later timestamp, the resolution would be to convert the UPDATE to an INSERT. Remote updates with earlier timestamps compared to the dead tuples will be disregarded. To implement this, an additional replication slot named pg_conflict_detection will be created on the subscriber side and maintained by the launcher. This slot will be used to retain dead tuples. Each apply worker will maintain its own non-removable transaction ID, while the launcher collects these IDs to determine whether to advance the xmin value of the replication slot. The process of advancing the non-removable transaction ID in the apply worker involves: 1) Call GetOldestActiveTransactionId() to take oldestRunningXid as the candidate xid. 2) Send a message to the walsender requesting the publisher status, which includes the latest WAL write position and information about transactions that are in the commit phase. 3) Wait for the status from the walsender. After receiving the first status, do not proceed if there are concurrent remote transactions that are still in the commit phase. These transactions might have been assigned an earlier commit timestamp but have not yet written the commit WAL record. Continue to request the publisher status until all these transactions have completed. 4) Advance the non-removable transaction ID if the current flush location has reached or surpassed the last received WAL position. These steps are repeated at intervals that are dynamically adjusted based on whether a new transaction ID has been assigned since the last advancement. This mechanism ensures that dead tuples are not removed until all concurrent transactions have been applied. It works for both bidirectional and non-bidirectional replication scenarios. Since the mechanism relies on a single replication slot, it not only assists in retaining dead tuples but also preserves commit timestamps and origin data. These information will be displayed in the additional logs generated for logical replication conflicts. Furthermore, the preserved commit timestamps and origin data are essential for consistently detecting update_origin_differs conflicts. This patch allows each apply worker to maintain the non-removable transaction ID in the shared memory following the steps described above. The actual replication slot management is implemented in the following patches.

This patch enables the logical replication launcher to create and maintain a replication slot named pg_conflict_detection. The launcher periodically collects the oldest_nonremovable_xid from all apply workers. It then computes the minimum transaction ID and advances the xmin value of the replication slot if it precedes the computed value. The interval for updating the slot (nap time) is dynamically adjusted based on the activity of the apply workers. The launcher waits for a certain period before performing the next update, with the duration varying depending on whether the xmin value of the replication slot was updated during the last cycle.

This patch adds a subscription option allowing users to specify whether information on the subscriber, which is useful for detecting update_deleted conflicts, should be retained. The default setting is false. If set to true, the detection of update_deleted will be enabled, and an additional replication slot named pg_conflict_detection will be created on the subscriber to prevent conflict information from being removed. Note that if multiple subscriptions on one node enable this option, only one replication slot will be created. The logical launcher will create and maintain a replication slot named pg_conflict_detection only if any local subscription has the retain_conflict_info option enabled. Enabling retain_conflict_info is prohibited if the publisher is currently in recovery mode (operating as a standby server). Bump catalog version

This commit introduces a GUC option max_conflict_retention_duration, designed to prevent excessive accumulation of dead tuples when subscription with retain_conflict_info enabled is present and the the apply worker cannot catch up with the publisher's workload. If the time spent advancing non-removable transaction ID surpasses the max_conflict_retention_duration threshold, the apply worker would stop retaining information for conflict detection. The replication slot pg_conflict_detection will be invalidated if all apply workers associated with the subscription, where retain_conflict_info is enabled, confirm that the retention duration exceeded the max_conflict_retention_duration. In this patch, a replication slot will not be automatically re-created if it becomes invalidated. Users can disable retain_conflict_info and re-enable it after confirming that the replication slot has been dropped. An upcoming patch will include support for automatic slot recreation once at least one apply worker confirms that the retention duration is within the max_conflict_retention_duration limit. To monitor worker's conflict retention status, this patch also introduces a new column 'retain_conflict_info' in the pg_stat_subscription view. This column indicates whether the apply worker is effectively retaining conflict information. The value is set to true only if retain_conflict_info is enabled for the associated subscription, and the retention duration for conflict detection by the apply worker has not exceeded max_conflict_retention_duration.

…uced The patch allows the launcher to drop and re-create the invalidated slot, if at least one apply worker has confirmed that the retention duration is now within the max_conflict_retention_duration.

This patch supports detecting update_deleted conflicts during update operations. If the target row cannot be found when applying update operations, we perform an additional scan of the table using snapshotAny. This scan aims to locate the most recently deleted row that matches the old column values from the remote update operation and has not yet been removed by VACUUM. If any such tuples are found, we report the update_deleted conflict along with the origin and transaction information that deleted the tuple.

…ication This branch was automatically generated by a robot using patches from an email thread registered at: https://commitfest.postgresql.org/patch/5378 The branch will be overwritten each time a new patch version is posted to the thread, and also periodically to check for bitrot caused by changes on the master branch. Patch(es): https://www.postgresql.org/message-id/TYAPR01MB572428A7A4A875C069567B8E94852@TYAPR01MB5724.jpnprd01.prod.outlook.com Author(s): Zhijie Hou

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comparing changes

Open a pull request

Commits on Apr 30, 2025

This comparison is taking too long to generate.