Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: postgresql-cfbot/postgresql
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: cf/5585~1
Choose a base ref
...
head repository: postgresql-cfbot/postgresql
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: cf/5585
Choose a head ref
  • 4 commits
  • 10 files changed
  • 2 contributors

Commits on Apr 28, 2025

  1. Filter transactions that need not be published. 

    This patch set aims to filter transactions at the decode stage rather than
streaming time, which allows the system to skip processing transactions that do
not contain tables included in the logical replication walsender's publication
list. As a result, this can prevent the serialization of unnecessary records to
disk during non-streaming mode, especially in large transactions, and also
reduce memory and CPU usage in streaming mode when many changes can be filtered
out.

A hash table of relation file locators is implemented to cache whether a
relation is filterable or not. This ensures that we only need to retrieve
relation and publication information from the catalog when a cache entry is
invalid, avoiding the overhead of starting a transaction for each record.

Filtering is temporarily suspended for a sequence of changes (set to 100
changes) when an unfilterable change is encountered. This strategy minimizes
the overhead of hash searching for every record, which is beneficial when the
majority of tables in an instance are published and thus unfilterable. The
threshold of 100 was determined to be the optimal balance based on performance
tests.

Additionally, transactions containing WAL records (INTERNAL_SNAPSHOT,
COMMAND_ID, or INVALIDATION) that modify the historical snapshot
constructed during logical decoding are deemed unfilterable. This
is necessary because constructing a correct historical snapshot
for searching publication information requires processing these WAL record.

Note that this patch filters changes only for system catalog relations,
non-logically logged relations, or temporary heaps and sequences. A subsequent
patch will introduce a new output plugin in pgoutput, which will further filter
changes for relations not included in publications.
    @itsajin
    itsajin authored and Commitfest Bot committed Apr 28, 2025
    Configuration menu
    Copy the full SHA
    d0dd4ac View commit details
    Browse the repository at this point in the history

  2. Introduce an output plugin callback to filter changes 

    This new output plugin callback provides an option to logical decoding plugins to filter out
changes early. The primary purpose of the callback is to conserve memory and processing cycles
by excluding changes that are not required by output plugins.
    @itsajin
    itsajin authored and Commitfest Bot committed Apr 28, 2025
    Configuration menu
    Copy the full SHA
    5a9bd47 View commit details
    Browse the repository at this point in the history

  3. Tests for filtering unpublished changes

    @itsajin
    itsajin authored and Commitfest Bot committed Apr 28, 2025
    Configuration menu
    Copy the full SHA
    957926f View commit details
    Browse the repository at this point in the history

  4. [CF 5585] v17 - Filter irrelevant change before reassemble transactio… 

    …ns during logical decoding

This branch was automatically generated by a robot using patches from an
email thread registered at:

https://commitfest.postgresql.org/patch/5585

The branch will be overwritten each time a new patch version is posted to
the thread, and also periodically to check for bitrot caused by changes
on the master branch.

Patch(es): https://www.postgresql.org/message-id/CAFPTHDaSn_-LuaE6v6s9+9HuvqpsJLbg-jz9FokZWekXydU39A@mail.gmail.com
Author(s): Jie Li, Ajin Cherian, Zhijie Hou
    Commitfest Bot committed Apr 28, 2025
    Configuration menu
    Copy the full SHA
    2753768 View commit details
    Browse the repository at this point in the history
Loading